2023-11-28T17:24:12.121 INFO:root:teuthology version: 0.0.1.dev209+g5ae09a5 2023-11-28T17:24:12.121 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_vshankar@teuthology --name vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi --archive /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398 --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} -- /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398/orig.config.yaml 2023-11-28T17:24:12.170 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-28T17:24:12.260 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398 branch: wip-vshankar-testing-20231127.102654 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} email: vshankar@redhat.com first_in_suite: false job_id: '7469398' kernel: flavor: default kdb: true sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20231127.102654 ceph: cephfs: max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: false mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: false mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - overall HEALTH_ - \(MON_DOWN\) - SLOW_OPS - slow request sha1: 47c1c547372ed5bff7984647dc200bac0e019caf ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 47c1c547372ed5bff7984647dc200bac0e019caf thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20231127.102654 sha1: 47c1c547372ed5bff7984647dc200bac0e019caf owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 1098 sha1: 47c1c547372ed5bff7984647dc200bac0e019caf sleep_before_teardown: 0 subset: 854/1024 suite: fs suite_branch: wip-vshankar-testing-20231127.102654 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 47c1c547372ed5bff7984647dc200bac0e019caf syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+iCcxfSf82GEbV3qrLN2zk3B72kftTGkorTEp9DLx11Cnbjrg3wMKm4Bv4ZWsqV9EvbYu0IFCdKCD0aBeVm94= smithi113.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdQNw65LjyaXDZutgOR30XOHDfvOBDbuZ+sVTLSEfsOXxgHHCpJpOhUpLvdBhD8YpfEX13hi6Gj6cLSu9FU5IE= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - mon_thrash: check_mds_failover: true revive_delay: 20 thrash_delay: 10 - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 5ae09a5cc7064ca2bf994df19f3006b79794789a timestamp: 2023-11-28_06:23:42 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-11-28T17:24:12.261 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa; will attempt to use it 2023-11-28T17:24:12.261 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks 2023-11-28T17:24:12.261 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-28T17:24:12.263 INFO:teuthology.task.internal:Checking packages... 2023-11-28T17:24:12.295 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '47c1c547372ed5bff7984647dc200bac0e019caf' 2023-11-28T17:24:12.295 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-28T17:24:12.295 INFO:teuthology.packaging:ref: None 2023-11-28T17:24:12.295 INFO:teuthology.packaging:tag: None 2023-11-28T17:24:12.295 INFO:teuthology.packaging:branch: wip-vshankar-testing-20231127.102654 2023-11-28T17:24:12.295 INFO:teuthology.packaging:sha1: 47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T17:24:12.296 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20231127.102654 2023-11-28T17:24:12.452 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7531.g47c1c547 2023-11-28T17:24:12.452 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-28T17:24:12.458 INFO:teuthology.task.internal:no buildpackages task found 2023-11-28T17:24:12.458 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-28T17:24:12.553 INFO:teuthology.task.internal:Saving configuration 2023-11-28T17:24:12.572 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-28T17:24:12.583 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T17:24:12.612 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 17:18:35.879422', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+iCcxfSf82GEbV3qrLN2zk3B72kftTGkorTEp9DLx11Cnbjrg3wMKm4Bv4ZWsqV9EvbYu0IFCdKCD0aBeVm94='} 2023-11-28T17:24:12.635 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 17:18:35.877664', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdQNw65LjyaXDZutgOR30XOHDfvOBDbuZ+sVTLSEfsOXxgHHCpJpOhUpLvdBhD8YpfEX13hi6Gj6cLSu9FU5IE='} 2023-11-28T17:24:12.635 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-28T17:24:12.641 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-11-28T17:24:12.641 INFO:teuthology.task.internal:roles: ubuntu@smithi113.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-11-28T17:24:12.641 INFO:teuthology.run_tasks:Running task console_log... 2023-11-28T17:24:12.755 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8583f88a60>, signals=[15]) 2023-11-28T17:24:12.755 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-28T17:24:12.761 INFO:teuthology.task.internal:Opening connections... 2023-11-28T17:24:12.761 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2023-11-28T17:24:12.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T17:24:12.887 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi113.front.sepia.ceph.com 2023-11-28T17:24:12.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T17:24:12.972 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-28T17:24:12.978 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2023-11-28T17:24:12.997 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2023-11-28T17:24:12.997 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2023-11-28T17:24:13.053 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2023-11-28T17:24:13.054 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2023-11-28T17:24:13.054 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2023-11-28T17:24:13.054 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2023-11-28T17:24:13.054 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2023-11-28T17:24:13.054 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2023-11-28T17:24:13.054 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-28T17:24:13.054 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2023-11-28T17:24:13.055 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2023-11-28T17:24:13.055 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-28T17:24:13.055 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2023-11-28T17:24:13.055 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-28T17:24:13.055 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-28T17:24:13.055 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-28T17:24:13.056 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2023-11-28T17:24:13.083 DEBUG:teuthology.orchestra.run.smithi113:> uname -m 2023-11-28T17:24:13.101 INFO:teuthology.orchestra.run.smithi113.stdout:x86_64 2023-11-28T17:24:13.101 DEBUG:teuthology.orchestra.run.smithi113:> cat /etc/os-release 2023-11-28T17:24:13.157 INFO:teuthology.orchestra.run.smithi113.stdout:NAME="CentOS Stream" 2023-11-28T17:24:13.158 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION="9" 2023-11-28T17:24:13.158 INFO:teuthology.orchestra.run.smithi113.stdout:ID="centos" 2023-11-28T17:24:13.158 INFO:teuthology.orchestra.run.smithi113.stdout:ID_LIKE="rhel fedora" 2023-11-28T17:24:13.158 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION_ID="9" 2023-11-28T17:24:13.158 INFO:teuthology.orchestra.run.smithi113.stdout:PLATFORM_ID="platform:el9" 2023-11-28T17:24:13.158 INFO:teuthology.orchestra.run.smithi113.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-28T17:24:13.158 INFO:teuthology.orchestra.run.smithi113.stdout:ANSI_COLOR="0;31" 2023-11-28T17:24:13.158 INFO:teuthology.orchestra.run.smithi113.stdout:LOGO="fedora-logo-icon" 2023-11-28T17:24:13.158 INFO:teuthology.orchestra.run.smithi113.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-28T17:24:13.159 INFO:teuthology.orchestra.run.smithi113.stdout:HOME_URL="https://centos.org/" 2023-11-28T17:24:13.159 INFO:teuthology.orchestra.run.smithi113.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-28T17:24:13.159 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-28T17:24:13.159 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-28T17:24:13.159 INFO:teuthology.lock.ops:Updating smithi113.front.sepia.ceph.com on lock server 2023-11-28T17:24:13.182 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-28T17:24:13.192 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-28T17:24:13.217 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-28T17:24:13.217 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2023-11-28T17:24:13.220 DEBUG:teuthology.orchestra.run.smithi113:> test '!' -e /home/ubuntu/cephtest 2023-11-28T17:24:13.236 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-28T17:24:13.241 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-28T17:24:13.241 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2023-11-28T17:24:13.276 DEBUG:teuthology.orchestra.run.smithi113:> test -z $(ls -A /var/lib/ceph) 2023-11-28T17:24:13.383 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-28T17:24:13.440 INFO:teuthology.run_tasks:Running task kernel... 2023-11-28T17:24:13.458 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'} 2023-11-28T17:24:13.458 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}}, timeout 300 2023-11-28T17:24:13.459 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-28T17:24:13.460 DEBUG:teuthology.orchestra.run.smithi113:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-28T17:24:13.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:24:13.476 DEBUG:teuthology.task.kernel:sha1 for mon.a is 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T17:24:13.477 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "8247bbb5b470f7e2a1f3eefcc16e24cf1338f606"... 2023-11-28T17:24:13.477 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2023-11-28T17:24:13.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:24:13.478 DEBUG:teuthology.task.kernel:sha1 for mon.b is 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T17:24:13.478 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "8247bbb5b470f7e2a1f3eefcc16e24cf1338f606"... 2023-11-28T17:24:13.478 DEBUG:teuthology.orchestra.run.smithi113:> uname -r 2023-11-28T17:24:13.533 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-361.el9.x86_64 2023-11-28T17:24:13.533 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T17:24:13.534 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-11-28T17:24:13.534 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T17:24:13.538 INFO:teuthology.orchestra.run.smithi113.stdout:5.14.0-361.el9.x86_64 2023-11-28T17:24:13.539 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T17:24:13.539 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-11-28T17:24:13.539 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T17:24:13.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum upgrade -y linux-firmware 2023-11-28T17:24:13.688 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum upgrade -y linux-firmware 2023-11-28T17:24:15.983 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 9.4 MB/s | 6.4 MB 00:00 2023-11-28T17:24:16.024 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS-9 - AppStream 9.0 MB/s | 6.4 MB 00:00 2023-11-28T17:24:17.675 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2023-11-28T17:24:17.726 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2023-11-28T17:24:19.350 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - BaseOS 6.3 MB/s | 7.9 MB 00:01 2023-11-28T17:24:20.982 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 7.9 MB 00:02 2023-11-28T17:24:21.459 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - AppStream 18 MB/s | 18 MB 00:00 2023-11-28T17:24:24.000 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 18 MB 00:01 2023-11-28T17:24:25.693 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 5.8 MB 00:00 2023-11-28T17:24:27.160 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 15 kB 00:00 2023-11-28T17:24:28.453 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2023-11-28T17:24:28.892 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 5.8 MB 00:01 2023-11-28T17:24:30.627 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 15 kB 00:00 2023-11-28T17:24:32.080 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2023-11-28T17:24:33.336 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2023-11-28T17:24:35.766 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2023-11-28T17:24:35.779 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2023-11-28T17:24:35.779 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2023-11-28T17:24:35.779 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2023-11-28T17:24:35.779 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrading: 2023-11-28T17:24:35.779 INFO:teuthology.orchestra.run.smithi113.stdout: linux-firmware noarch 20231030-141.el9 CentOS-BaseOS 328 M 2023-11-28T17:24:35.779 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:24:35.779 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2023-11-28T17:24:35.779 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2023-11-28T17:24:35.780 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrade 1 Package 2023-11-28T17:24:35.780 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:24:35.780 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 328 M 2023-11-28T17:24:35.780 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2023-11-28T17:24:36.820 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2023-11-28T17:24:39.314 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2023-11-28T17:24:39.329 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2023-11-28T17:24:39.329 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2023-11-28T17:24:39.329 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2023-11-28T17:24:39.329 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2023-11-28T17:24:39.329 INFO:teuthology.orchestra.run.smithi098.stdout: linux-firmware noarch 20231030-141.el9 CentOS-BaseOS 328 M 2023-11-28T17:24:39.329 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:24:39.329 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2023-11-28T17:24:39.329 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2023-11-28T17:24:39.329 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 1 Package 2023-11-28T17:24:39.330 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:24:39.330 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 328 M 2023-11-28T17:24:39.330 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2023-11-28T17:24:42.503 INFO:teuthology.orchestra.run.smithi113.stdout:linux-firmware-20231030-141.el9.noarch.rpm 49 MB/s | 328 MB 00:06 2023-11-28T17:24:42.504 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2023-11-28T17:24:42.504 INFO:teuthology.orchestra.run.smithi113.stdout:Total 49 MB/s | 328 MB 00:06 2023-11-28T17:24:42.505 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2023-11-28T17:24:42.591 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2023-11-28T17:24:42.591 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2023-11-28T17:24:44.709 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2023-11-28T17:24:44.710 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2023-11-28T17:24:47.462 INFO:teuthology.orchestra.run.smithi098.stdout:linux-firmware-20231030-141.el9.noarch.rpm 40 MB/s | 328 MB 00:08 2023-11-28T17:24:47.463 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2023-11-28T17:24:47.463 INFO:teuthology.orchestra.run.smithi098.stdout:Total 40 MB/s | 328 MB 00:08 2023-11-28T17:24:47.464 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2023-11-28T17:24:47.551 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2023-11-28T17:24:47.551 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2023-11-28T17:24:49.682 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2023-11-28T17:24:49.682 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2023-11-28T17:24:50.615 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2023-11-28T17:24:51.145 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-28T17:24:51.224 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T17:24:52.233 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T17:24:52.233 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-28T17:24:52.623 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T17:24:52.623 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:24:52.624 INFO:teuthology.orchestra.run.smithi113.stdout:Upgraded: 2023-11-28T17:24:52.624 INFO:teuthology.orchestra.run.smithi113.stdout: linux-firmware-20231030-141.el9.noarch 2023-11-28T17:24:52.624 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:24:52.624 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2023-11-28T17:24:52.917 INFO:teuthology.task.kernel:Downloading kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on mon.b... 2023-11-28T17:24:52.918 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T17:24:53.073 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ 2023-11-28T17:24:53.074 DEBUG:teuthology.orchestra.run.smithi113:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ --input-file=- 2023-11-28T17:24:53.076 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi113.front.sepia.ceph.com to complete... 2023-11-28T17:24:55.618 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2023-11-28T17:24:56.091 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-28T17:24:56.178 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T17:24:57.157 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T17:24:57.157 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-28T17:24:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T17:24:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:24:57.574 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2023-11-28T17:24:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: linux-firmware-20231030-141.el9.noarch 2023-11-28T17:24:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:24:57.575 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2023-11-28T17:24:57.865 INFO:teuthology.task.kernel:Downloading kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on mon.a... 2023-11-28T17:24:57.865 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T17:24:58.077 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ 2023-11-28T17:24:58.077 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ --input-file=- 2023-11-28T17:24:58.080 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi098.front.sepia.ceph.com to complete... 2023-11-28T17:25:06.139 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28 17:25:06 URL:https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm [320493731/320493731] -> "/tmp/kernel.x86_64.rpm" [1] 2023-11-28T17:25:06.140 INFO:teuthology.orchestra.run.smithi113.stderr:FINISHED --2023-11-28 17:25:06-- 2023-11-28T17:25:06.140 INFO:teuthology.orchestra.run.smithi113.stderr:Total wall clock time: 13s 2023-11-28T17:25:06.140 INFO:teuthology.orchestra.run.smithi113.stderr:Downloaded: 1 files, 306M in 13s (24.0 MB/s) 2023-11-28T17:25:06.141 INFO:teuthology.task.kernel:Installing kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on mon.b... 2023-11-28T17:25:06.141 DEBUG:teuthology.orchestra.run.smithi113:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-11-28T17:25:07.990 INFO:teuthology.orchestra.run.smithi113.stdout:Verifying... ######################################## 2023-11-28T17:25:08.039 INFO:teuthology.orchestra.run.smithi113.stdout:Preparing... ######################################## 2023-11-28T17:25:08.117 INFO:teuthology.orchestra.run.smithi113.stdout:Updating / installing... 2023-11-28T17:25:15.184 INFO:teuthology.orchestra.run.smithi113.stdout:kernel-6.6.0_g8247bbb5b470-1 ######################################## 2023-11-28T17:25:30.803 INFO:teuthology.orchestra.run.smithi113.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-11-28T17:25:31.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28 17:25:31 URL:https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm [320493731/320493731] -> "/tmp/kernel.x86_64.rpm" [1] 2023-11-28T17:25:31.808 INFO:teuthology.orchestra.run.smithi098.stderr:FINISHED --2023-11-28 17:25:31-- 2023-11-28T17:25:31.808 INFO:teuthology.orchestra.run.smithi098.stderr:Total wall clock time: 34s 2023-11-28T17:25:31.808 INFO:teuthology.orchestra.run.smithi098.stderr:Downloaded: 1 files, 306M in 33s (9.24 MB/s) 2023-11-28T17:25:31.809 INFO:teuthology.task.kernel:Installing kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on mon.a... 2023-11-28T17:25:31.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-11-28T17:25:33.663 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying... ######################################## 2023-11-28T17:25:34.167 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing... ######################################## 2023-11-28T17:25:34.248 INFO:teuthology.orchestra.run.smithi098.stdout:Updating / installing... 2023-11-28T17:25:41.278 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-6.6.0_g8247bbb5b470-1 ######################################## 2023-11-28T17:25:52.146 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi113.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-11-28T17:25:52.147 DEBUG:teuthology.orchestra.run.smithi113:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-11-28T17:25:52.224 INFO:teuthology.orchestra.run.smithi113.stdout:/boot/System.map-6.6.0-g8247bbb5b470 2023-11-28T17:25:52.224 INFO:teuthology.orchestra.run.smithi113.stdout:/boot/config-6.6.0-g8247bbb5b470 2023-11-28T17:25:52.224 INFO:teuthology.orchestra.run.smithi113.stdout:/boot/vmlinuz-6.6.0-g8247bbb5b470 2023-11-28T17:25:52.224 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470 2023-11-28T17:25:52.225 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel 2023-11-28T17:25:52.225 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch 2023-11-28T17:25:52.225 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86 2023-11-28T17:25:52.225 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto 2023-11-28T17:25:52.225 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-11-28T17:25:52.225 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-11-28T17:25:52.225 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-11-28T17:25:52.226 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-11-28T17:25:52.226 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-11-28T17:25:52.226 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-11-28T17:25:52.226 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-11-28T17:25:52.226 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32c-intel.ko 2023-11-28T17:25:52.226 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-11-28T17:25:52.227 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-11-28T17:25:52.227 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-11-28T17:25:52.227 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-11-28T17:25:52.227 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx2.ko 2023-11-28T17:25:52.227 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-11-28T17:25:52.228 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-11-28T17:25:52.228 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-11-28T17:25:52.228 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-11-28T17:25:52.228 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-11-28T17:25:52.228 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-11-28T17:25:52.228 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-11-28T17:25:52.229 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel 2023-11-28T17:25:52.229 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu 2023-11-28T17:25:52.229 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce 2023-11-28T17:25:52.229 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-11-28T17:25:52.229 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm 2023-11-28T17:25:52.229 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-amd.ko 2023-11-28T17:25:52.230 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-intel.ko 2023-11-28T17:25:52.230 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm.ko 2023-11-28T17:25:52.230 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto 2023-11-28T17:25:52.230 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ansi_cprng.ko 2023-11-28T17:25:52.230 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/anubis.ko 2023-11-28T17:25:52.230 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/arc4.ko 2023-11-28T17:25:52.230 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx 2023-11-28T17:25:52.231 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_memcpy.ko 2023-11-28T17:25:52.231 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_pq.ko 2023-11-28T17:25:52.231 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_raid6_recov.ko 2023-11-28T17:25:52.231 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_tx.ko 2023-11-28T17:25:52.231 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_xor.ko 2023-11-28T17:25:52.231 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/raid6test.ko 2023-11-28T17:25:52.232 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authenc.ko 2023-11-28T17:25:52.232 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authencesn.ko 2023-11-28T17:25:52.232 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blake2b_generic.ko 2023-11-28T17:25:52.232 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_common.ko 2023-11-28T17:25:52.232 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_generic.ko 2023-11-28T17:25:52.232 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/camellia_generic.ko 2023-11-28T17:25:52.233 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast5_generic.ko 2023-11-28T17:25:52.233 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast6_generic.ko 2023-11-28T17:25:52.233 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast_common.ko 2023-11-28T17:25:52.233 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ccm.ko 2023-11-28T17:25:52.233 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cmac.ko 2023-11-28T17:25:52.233 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crc32_generic.ko 2023-11-28T17:25:52.234 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_engine.ko 2023-11-28T17:25:52.234 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_user.ko 2023-11-28T17:25:52.234 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cts.ko 2023-11-28T17:25:52.234 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/des_generic.ko 2023-11-28T17:25:52.234 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/dh_generic.ko 2023-11-28T17:25:52.234 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/echainiv.ko 2023-11-28T17:25:52.234 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/essiv.ko 2023-11-28T17:25:52.235 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/fcrypt.ko 2023-11-28T17:25:52.235 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/khazad.ko 2023-11-28T17:25:52.235 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4.ko 2023-11-28T17:25:52.235 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4hc.ko 2023-11-28T17:25:52.235 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/md4.ko 2023-11-28T17:25:52.235 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/michael_mic.ko 2023-11-28T17:25:52.235 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcbc.ko 2023-11-28T17:25:52.236 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcrypt.ko 2023-11-28T17:25:52.236 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/rmd160.ko 2023-11-28T17:25:52.236 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/seed.ko 2023-11-28T17:25:52.236 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/serpent_generic.ko 2023-11-28T17:25:52.236 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tcrypt.ko 2023-11-28T17:25:52.236 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tea.ko 2023-11-28T17:25:52.237 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_common.ko 2023-11-28T17:25:52.237 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_generic.ko 2023-11-28T17:25:52.237 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/vmac.ko 2023-11-28T17:25:52.237 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/wp512.ko 2023-11-28T17:25:52.237 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xcbc.ko 2023-11-28T17:25:52.238 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xor.ko 2023-11-28T17:25:52.238 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xxhash_generic.ko 2023-11-28T17:25:52.238 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers 2023-11-28T17:25:52.238 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi 2023-11-28T17:25:52.238 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_ipmi.ko 2023-11-28T17:25:52.238 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_pad.ko 2023-11-28T17:25:52.238 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/custom_method.ko 2023-11-28T17:25:52.238 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/ec_sys.ko 2023-11-28T17:25:52.238 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/platform_profile.ko 2023-11-28T17:25:52.238 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbs.ko 2023-11-28T17:25:52.238 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbshc.ko 2023-11-28T17:25:52.239 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/video.ko 2023-11-28T17:25:52.239 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata 2023-11-28T17:25:52.239 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/acard-ahci.ko 2023-11-28T17:25:52.239 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ahci_platform.ko 2023-11-28T17:25:52.239 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ata_generic.ko 2023-11-28T17:25:52.239 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/libahci_platform.ko 2023-11-28T17:25:52.239 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_acpi.ko 2023-11-28T17:25:52.239 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ali.ko 2023-11-28T17:25:52.239 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_amd.ko 2023-11-28T17:25:52.239 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_artop.ko 2023-11-28T17:25:52.239 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atiixp.ko 2023-11-28T17:25:52.240 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atp867x.ko 2023-11-28T17:25:52.240 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd640.ko 2023-11-28T17:25:52.240 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd64x.ko 2023-11-28T17:25:52.240 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cypress.ko 2023-11-28T17:25:52.240 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_efar.ko 2023-11-28T17:25:52.240 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt366.ko 2023-11-28T17:25:52.240 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt37x.ko 2023-11-28T17:25:52.240 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x2n.ko 2023-11-28T17:25:52.240 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x3.ko 2023-11-28T17:25:52.240 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it8213.ko 2023-11-28T17:25:52.240 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it821x.ko 2023-11-28T17:25:52.240 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_jmicron.ko 2023-11-28T17:25:52.240 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_marvell.ko 2023-11-28T17:25:52.241 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_mpiix.ko 2023-11-28T17:25:52.241 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_netcell.ko 2023-11-28T17:25:52.241 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ninja32.ko 2023-11-28T17:25:52.241 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87410.ko 2023-11-28T17:25:52.241 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87415.ko 2023-11-28T17:25:52.241 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_oldpiix.ko 2023-11-28T17:25:52.241 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_opti.ko 2023-11-28T17:25:52.241 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_optidma.ko 2023-11-28T17:25:52.241 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pcmcia.ko 2023-11-28T17:25:52.241 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc2027x.ko 2023-11-28T17:25:52.241 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-11-28T17:25:52.241 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_piccolo.ko 2023-11-28T17:25:52.242 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_rdc.ko 2023-11-28T17:25:52.242 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sch.ko 2023-11-28T17:25:52.242 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_serverworks.ko 2023-11-28T17:25:52.242 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sil680.ko 2023-11-28T17:25:52.242 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sis.ko 2023-11-28T17:25:52.242 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sl82c105.ko 2023-11-28T17:25:52.242 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_triflex.ko 2023-11-28T17:25:52.242 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_via.ko 2023-11-28T17:25:52.242 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pdc_adma.ko 2023-11-28T17:25:52.242 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_inic162x.ko 2023-11-28T17:25:52.242 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_mv.ko 2023-11-28T17:25:52.243 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_nv.ko 2023-11-28T17:25:52.243 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_promise.ko 2023-11-28T17:25:52.243 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_qstor.ko 2023-11-28T17:25:52.243 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil.ko 2023-11-28T17:25:52.243 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil24.ko 2023-11-28T17:25:52.243 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sis.ko 2023-11-28T17:25:52.243 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_svw.ko 2023-11-28T17:25:52.243 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sx4.ko 2023-11-28T17:25:52.243 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_uli.ko 2023-11-28T17:25:52.243 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_via.ko 2023-11-28T17:25:52.243 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_vsc.ko 2023-11-28T17:25:52.243 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay 2023-11-28T17:25:52.243 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay/ks0108.ko 2023-11-28T17:25:52.244 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base 2023-11-28T17:25:52.244 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap 2023-11-28T17:25:52.244 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-i2c.ko 2023-11-28T17:25:52.244 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-mmio.ko 2023-11-28T17:25:52.244 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma 2023-11-28T17:25:52.244 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma/bcma.ko 2023-11-28T17:25:52.244 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block 2023-11-28T17:25:52.244 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe 2023-11-28T17:25:52.244 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe/aoe.ko 2023-11-28T17:25:52.244 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/brd.ko 2023-11-28T17:25:52.244 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd 2023-11-28T17:25:52.244 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd/drbd.ko 2023-11-28T17:25:52.245 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/floppy.ko 2023-11-28T17:25:52.245 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/loop.ko 2023-11-28T17:25:52.245 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx 2023-11-28T17:25:52.245 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-11-28T17:25:52.245 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/nbd.ko 2023-11-28T17:25:52.245 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk 2023-11-28T17:25:52.245 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk/null_blk.ko 2023-11-28T17:25:52.245 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/pktcdvd.ko 2023-11-28T17:25:52.245 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/rbd.ko 2023-11-28T17:25:52.245 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/virtio_blk.ko 2023-11-28T17:25:52.245 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback 2023-11-28T17:25:52.246 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-11-28T17:25:52.246 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkfront.ko 2023-11-28T17:25:52.246 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram 2023-11-28T17:25:52.246 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram/zram.ko 2023-11-28T17:25:52.246 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char 2023-11-28T17:25:52.246 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hangcheck-timer.ko 2023-11-28T17:25:52.246 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random 2023-11-28T17:25:52.246 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/amd-rng.ko 2023-11-28T17:25:52.246 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/intel-rng.ko 2023-11-28T17:25:52.246 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-11-28T17:25:52.246 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/via-rng.ko 2023-11-28T17:25:52.246 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/virtio-rng.ko 2023-11-28T17:25:52.247 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi 2023-11-28T17:25:52.247 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-11-28T17:25:52.247 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-11-28T17:25:52.247 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-11-28T17:25:52.247 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_si.ko 2023-11-28T17:25:52.247 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-11-28T17:25:52.247 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-11-28T17:25:52.247 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/lp.ko 2023-11-28T17:25:52.247 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave 2023-11-28T17:25:52.247 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave/mwave.ko 2023-11-28T17:25:52.247 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ppdev.ko 2023-11-28T17:25:52.248 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tlclk.ko 2023-11-28T17:25:52.248 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm 2023-11-28T17:25:52.248 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm.ko 2023-11-28T17:25:52.248 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_atmel.ko 2023-11-28T17:25:52.248 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_infineon.ko 2023-11-28T17:25:52.248 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_nsc.ko 2023-11-28T17:25:52.248 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis.ko 2023-11-28T17:25:52.248 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-11-28T17:25:52.248 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/uv_mmtimer.ko 2023-11-28T17:25:52.248 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/virtio_console.ko 2023-11-28T17:25:52.248 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq 2023-11-28T17:25:52.248 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-11-28T17:25:52.249 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-11-28T17:25:52.249 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/p4-clockmod.ko 2023-11-28T17:25:52.249 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-11-28T17:25:52.249 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/powernow-k8.ko 2023-11-28T17:25:52.249 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/speedstep-lib.ko 2023-11-28T17:25:52.249 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto 2023-11-28T17:25:52.249 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp 2023-11-28T17:25:52.249 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-11-28T17:25:52.249 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp.ko 2023-11-28T17:25:52.249 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel 2023-11-28T17:25:52.249 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat 2023-11-28T17:25:52.249 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common 2023-11-28T17:25:52.249 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-11-28T17:25:52.250 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-11-28T17:25:52.250 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-11-28T17:25:52.250 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-aes.ko 2023-11-28T17:25:52.250 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-sha.ko 2023-11-28T17:25:52.250 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio 2023-11-28T17:25:52.250 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-11-28T17:25:52.250 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca 2023-11-28T17:25:52.250 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca/dca.ko 2023-11-28T17:25:52.250 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq 2023-11-28T17:25:52.250 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-11-28T17:25:52.250 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma 2023-11-28T17:25:52.250 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw 2023-11-28T17:25:52.251 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw/dw_dmac.ko 2023-11-28T17:25:52.251 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat 2023-11-28T17:25:52.251 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat/ioatdma.ko 2023-11-28T17:25:52.251 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac 2023-11-28T17:25:52.251 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/amd64_edac.ko 2023-11-28T17:25:52.251 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/e752x_edac.ko 2023-11-28T17:25:52.251 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/edac_mce_amd.ko 2023-11-28T17:25:52.251 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3000_edac.ko 2023-11-28T17:25:52.251 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3200_edac.ko 2023-11-28T17:25:52.251 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5100_edac.ko 2023-11-28T17:25:52.251 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5400_edac.ko 2023-11-28T17:25:52.251 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7300_edac.ko 2023-11-28T17:25:52.251 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7core_edac.ko 2023-11-28T17:25:52.252 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i82975x_edac.ko 2023-11-28T17:25:52.252 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/ie31200_edac.ko 2023-11-28T17:25:52.252 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/sb_edac.ko 2023-11-28T17:25:52.252 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/x38_edac.ko 2023-11-28T17:25:52.252 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware 2023-11-28T17:25:52.252 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/edd.ko 2023-11-28T17:25:52.252 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/iscsi_ibft.ko 2023-11-28T17:25:52.252 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio 2023-11-28T17:25:52.252 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-ich.ko 2023-11-28T17:25:52.252 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-viperboard.ko 2023-11-28T17:25:52.252 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid 2023-11-28T17:25:52.252 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-appleir.ko 2023-11-28T17:25:52.252 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-aureal.ko 2023-11-28T17:25:52.253 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-axff.ko 2023-11-28T17:25:52.253 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-dr.ko 2023-11-28T17:25:52.253 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elecom.ko 2023-11-28T17:25:52.253 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elo.ko 2023-11-28T17:25:52.253 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-emsff.ko 2023-11-28T17:25:52.253 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gaff.ko 2023-11-28T17:25:52.253 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gt683r.ko 2023-11-28T17:25:52.253 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gyration.ko 2023-11-28T17:25:52.253 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-kbd.ko 2023-11-28T17:25:52.253 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-mouse.ko 2023-11-28T17:25:52.253 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtekff.ko 2023-11-28T17:25:52.253 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-hyperv.ko 2023-11-28T17:25:52.253 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-icade.ko 2023-11-28T17:25:52.254 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-keytouch.ko 2023-11-28T17:25:52.254 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-kye.ko 2023-11-28T17:25:52.254 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lcpower.ko 2023-11-28T17:25:52.254 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-led.ko 2023-11-28T17:25:52.254 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lenovo.ko 2023-11-28T17:25:52.254 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-dj.ko 2023-11-28T17:25:52.254 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-11-28T17:25:52.254 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-multitouch.ko 2023-11-28T17:25:52.254 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-ortek.ko 2023-11-28T17:25:52.254 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-penmount.ko 2023-11-28T17:25:52.254 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-petalynx.ko 2023-11-28T17:25:52.254 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-picolcd.ko 2023-11-28T17:25:52.255 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-pl.ko 2023-11-28T17:25:52.255 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-plantronics.ko 2023-11-28T17:25:52.255 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-primax.ko 2023-11-28T17:25:52.255 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-rmi.ko 2023-11-28T17:25:52.255 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-arvo.ko 2023-11-28T17:25:52.255 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-common.ko 2023-11-28T17:25:52.255 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-isku.ko 2023-11-28T17:25:52.255 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kone.ko 2023-11-28T17:25:52.255 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-11-28T17:25:52.255 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-konepure.ko 2023-11-28T17:25:52.255 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-11-28T17:25:52.255 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-lua.ko 2023-11-28T17:25:52.256 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-pyra.ko 2023-11-28T17:25:52.256 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-ryos.ko 2023-11-28T17:25:52.256 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-savu.ko 2023-11-28T17:25:52.256 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat.ko 2023-11-28T17:25:52.256 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-saitek.ko 2023-11-28T17:25:52.256 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-samsung.ko 2023-11-28T17:25:52.256 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sensor-hub.ko 2023-11-28T17:25:52.256 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sjoy.ko 2023-11-28T17:25:52.256 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sony.ko 2023-11-28T17:25:52.256 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-speedlink.ko 2023-11-28T17:25:52.256 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-steelseries.ko 2023-11-28T17:25:52.256 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sunplus.ko 2023-11-28T17:25:52.256 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-thrustmaster.ko 2023-11-28T17:25:52.257 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tivo.ko 2023-11-28T17:25:52.257 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tmff.ko 2023-11-28T17:25:52.257 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-topseed.ko 2023-11-28T17:25:52.257 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-twinhan.ko 2023-11-28T17:25:52.257 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-uclogic.ko 2023-11-28T17:25:52.257 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-waltop.ko 2023-11-28T17:25:52.257 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-wiimote.ko 2023-11-28T17:25:52.257 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-xinmo.ko 2023-11-28T17:25:52.257 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zpff.ko 2023-11-28T17:25:52.257 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zydacron.ko 2023-11-28T17:25:52.258 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/uhid.ko 2023-11-28T17:25:52.258 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/wacom.ko 2023-11-28T17:25:52.258 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv 2023-11-28T17:25:52.258 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_balloon.ko 2023-11-28T17:25:52.258 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_utils.ko 2023-11-28T17:25:52.258 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_vmbus.ko 2023-11-28T17:25:52.258 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon 2023-11-28T17:25:52.258 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru.ko 2023-11-28T17:25:52.259 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru3.ko 2023-11-28T17:25:52.259 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/acpi_power_meter.ko 2023-11-28T17:25:52.259 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7414.ko 2023-11-28T17:25:52.259 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7418.ko 2023-11-28T17:25:52.259 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adc128d818.ko 2023-11-28T17:25:52.259 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1025.ko 2023-11-28T17:25:52.259 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1026.ko 2023-11-28T17:25:52.259 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1029.ko 2023-11-28T17:25:52.259 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1031.ko 2023-11-28T17:25:52.259 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm9240.ko 2023-11-28T17:25:52.259 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ads7828.ko 2023-11-28T17:25:52.259 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7410.ko 2023-11-28T17:25:52.259 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7411.ko 2023-11-28T17:25:52.259 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7462.ko 2023-11-28T17:25:52.260 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7470.ko 2023-11-28T17:25:52.260 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7475.ko 2023-11-28T17:25:52.260 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7x10.ko 2023-11-28T17:25:52.260 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/amc6821.ko 2023-11-28T17:25:52.260 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/applesmc.ko 2023-11-28T17:25:52.260 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asb100.ko 2023-11-28T17:25:52.260 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asc7621.ko 2023-11-28T17:25:52.260 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asus_atk0110.ko 2023-11-28T17:25:52.260 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/atxp1.ko 2023-11-28T17:25:52.260 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/coretemp.ko 2023-11-28T17:25:52.260 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-11-28T17:25:52.260 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dme1737.ko 2023-11-28T17:25:52.260 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds1621.ko 2023-11-28T17:25:52.261 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds620.ko 2023-11-28T17:25:52.261 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc1403.ko 2023-11-28T17:25:52.261 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc6w201.ko 2023-11-28T17:25:52.261 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71805f.ko 2023-11-28T17:25:52.261 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71882fg.ko 2023-11-28T17:25:52.261 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f75375s.ko 2023-11-28T17:25:52.261 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fam15h_power.ko 2023-11-28T17:25:52.261 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fschmd.ko 2023-11-28T17:25:52.261 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g760a.ko 2023-11-28T17:25:52.261 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g762.ko 2023-11-28T17:25:52.261 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl518sm.ko 2023-11-28T17:25:52.261 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl520sm.ko 2023-11-28T17:25:52.262 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/hwmon-vid.ko 2023-11-28T17:25:52.262 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5500_temp.ko 2023-11-28T17:25:52.262 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5k_amb.ko 2023-11-28T17:25:52.262 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmaem.ko 2023-11-28T17:25:52.262 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmpex.ko 2023-11-28T17:25:52.262 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina209.ko 2023-11-28T17:25:52.262 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina2xx.ko 2023-11-28T17:25:52.262 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/it87.ko 2023-11-28T17:25:52.262 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k10temp.ko 2023-11-28T17:25:52.262 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k8temp.ko 2023-11-28T17:25:52.262 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lineage-pem.ko 2023-11-28T17:25:52.263 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm63.ko 2023-11-28T17:25:52.263 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm73.ko 2023-11-28T17:25:52.263 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm75.ko 2023-11-28T17:25:52.263 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm77.ko 2023-11-28T17:25:52.263 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm78.ko 2023-11-28T17:25:52.263 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm80.ko 2023-11-28T17:25:52.263 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm83.ko 2023-11-28T17:25:52.263 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm85.ko 2023-11-28T17:25:52.263 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm87.ko 2023-11-28T17:25:52.263 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm90.ko 2023-11-28T17:25:52.263 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm92.ko 2023-11-28T17:25:52.264 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm93.ko 2023-11-28T17:25:52.264 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95234.ko 2023-11-28T17:25:52.264 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95241.ko 2023-11-28T17:25:52.264 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95245.ko 2023-11-28T17:25:52.264 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc2945.ko 2023-11-28T17:25:52.264 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4151.ko 2023-11-28T17:25:52.264 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4215.ko 2023-11-28T17:25:52.264 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4222.ko 2023-11-28T17:25:52.264 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4245.ko 2023-11-28T17:25:52.264 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4260.ko 2023-11-28T17:25:52.264 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4261.ko 2023-11-28T17:25:52.265 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max16065.ko 2023-11-28T17:25:52.265 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1619.ko 2023-11-28T17:25:52.265 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1668.ko 2023-11-28T17:25:52.265 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max197.ko 2023-11-28T17:25:52.265 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6639.ko 2023-11-28T17:25:52.265 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6650.ko 2023-11-28T17:25:52.265 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6697.ko 2023-11-28T17:25:52.265 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/mcp3021.ko 2023-11-28T17:25:52.265 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6683.ko 2023-11-28T17:25:52.265 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775-core.ko 2023-11-28T17:25:52.265 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775.ko 2023-11-28T17:25:52.265 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct7802.ko 2023-11-28T17:25:52.265 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87360.ko 2023-11-28T17:25:52.266 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87427.ko 2023-11-28T17:25:52.266 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pcf8591.ko 2023-11-28T17:25:52.266 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus 2023-11-28T17:25:52.266 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-11-28T17:25:52.266 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-11-28T17:25:52.266 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-11-28T17:25:52.266 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max16064.ko 2023-11-28T17:25:52.266 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max34440.ko 2023-11-28T17:25:52.266 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max8688.ko 2023-11-28T17:25:52.266 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-11-28T17:25:52.266 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-11-28T17:25:52.267 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-11-28T17:25:52.267 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-11-28T17:25:52.267 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-11-28T17:25:52.267 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-11-28T17:25:52.267 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/powr1220.ko 2023-11-28T17:25:52.267 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5627.ko 2023-11-28T17:25:52.267 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5636.ko 2023-11-28T17:25:52.267 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch56xx-common.ko 2023-11-28T17:25:52.267 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht15.ko 2023-11-28T17:25:52.267 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht21.ko 2023-11-28T17:25:52.267 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/shtc1.ko 2023-11-28T17:25:52.268 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sis5595.ko 2023-11-28T17:25:52.268 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47b397.ko 2023-11-28T17:25:52.268 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m1.ko 2023-11-28T17:25:52.268 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m192.ko 2023-11-28T17:25:52.268 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/thmc50.ko 2023-11-28T17:25:52.268 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp102.ko 2023-11-28T17:25:52.268 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp103.ko 2023-11-28T17:25:52.268 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp401.ko 2023-11-28T17:25:52.268 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp421.ko 2023-11-28T17:25:52.268 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via-cputemp.ko 2023-11-28T17:25:52.268 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via686a.ko 2023-11-28T17:25:52.268 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt1211.ko 2023-11-28T17:25:52.268 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt8231.ko 2023-11-28T17:25:52.269 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627ehf.ko 2023-11-28T17:25:52.269 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627hf.ko 2023-11-28T17:25:52.269 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83781d.ko 2023-11-28T17:25:52.269 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83791d.ko 2023-11-28T17:25:52.269 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83792d.ko 2023-11-28T17:25:52.269 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83793.ko 2023-11-28T17:25:52.269 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83795.ko 2023-11-28T17:25:52.269 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l785ts.ko 2023-11-28T17:25:52.269 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l786ng.ko 2023-11-28T17:25:52.269 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c 2023-11-28T17:25:52.269 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos 2023-11-28T17:25:52.269 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-11-28T17:25:52.269 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-11-28T17:25:52.270 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses 2023-11-28T17:25:52.270 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-11-28T17:25:52.270 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-11-28T17:25:52.270 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-11-28T17:25:52.270 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-11-28T17:25:52.270 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-11-28T17:25:52.270 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-11-28T17:25:52.270 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-11-28T17:25:52.270 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-11-28T17:25:52.270 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-i801.ko 2023-11-28T17:25:52.270 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-isch.ko 2023-11-28T17:25:52.270 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-11-28T17:25:52.270 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-11-28T17:25:52.271 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-11-28T17:25:52.271 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-parport.ko 2023-11-28T17:25:52.271 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-11-28T17:25:52.271 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-11-28T17:25:52.271 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-11-28T17:25:52.271 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-11-28T17:25:52.271 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-11-28T17:25:52.271 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-11-28T17:25:52.271 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-via.ko 2023-11-28T17:25:52.271 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-11-28T17:25:52.271 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-11-28T17:25:52.272 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-dev.ko 2023-11-28T17:25:52.272 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-mux.ko 2023-11-28T17:25:52.272 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-11-28T17:25:52.272 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-smbus.ko 2023-11-28T17:25:52.272 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-stub.ko 2023-11-28T17:25:52.272 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband 2023-11-28T17:25:52.272 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core 2023-11-28T17:25:52.272 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_cm.ko 2023-11-28T17:25:52.272 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_core.ko 2023-11-28T17:25:52.272 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_umad.ko 2023-11-28T17:25:52.272 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-11-28T17:25:52.272 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/iw_cm.ko 2023-11-28T17:25:52.272 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_cm.ko 2023-11-28T17:25:52.273 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-11-28T17:25:52.273 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw 2023-11-28T17:25:52.273 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4 2023-11-28T17:25:52.273 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-11-28T17:25:52.273 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4 2023-11-28T17:25:52.273 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-11-28T17:25:52.273 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5 2023-11-28T17:25:52.273 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-11-28T17:25:52.273 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca 2023-11-28T17:25:52.273 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-11-28T17:25:52.273 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp 2023-11-28T17:25:52.273 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib 2023-11-28T17:25:52.273 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-11-28T17:25:52.274 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser 2023-11-28T17:25:52.274 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-11-28T17:25:52.274 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert 2023-11-28T17:25:52.274 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-11-28T17:25:52.274 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp 2023-11-28T17:25:52.274 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-11-28T17:25:52.274 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt 2023-11-28T17:25:52.274 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-11-28T17:25:52.274 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input 2023-11-28T17:25:52.274 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport 2023-11-28T17:25:52.274 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-11-28T17:25:52.275 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/fm801-gp.ko 2023-11-28T17:25:52.275 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/gameport.ko 2023-11-28T17:25:52.275 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/joydev.ko 2023-11-28T17:25:52.275 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard 2023-11-28T17:25:52.275 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/gpio_keys.ko 2023-11-28T17:25:52.275 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse 2023-11-28T17:25:52.275 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/appletouch.ko 2023-11-28T17:25:52.275 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/bcm5974.ko 2023-11-28T17:25:52.275 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/cyapatp.ko 2023-11-28T17:25:52.275 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/elan_i2c.ko 2023-11-28T17:25:52.275 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/sermouse.ko 2023-11-28T17:25:52.275 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-11-28T17:25:52.275 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_usb.ko 2023-11-28T17:25:52.276 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/vsxxxaa.ko 2023-11-28T17:25:52.276 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4 2023-11-28T17:25:52.276 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4/rmi_core.ko 2023-11-28T17:25:52.276 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio 2023-11-28T17:25:52.276 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/altera_ps2.ko 2023-11-28T17:25:52.276 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/arc_ps2.ko 2023-11-28T17:25:52.276 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-11-28T17:25:52.276 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/serio_raw.ko 2023-11-28T17:25:52.276 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/sparse-keymap.ko 2023-11-28T17:25:52.276 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu 2023-11-28T17:25:52.276 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd 2023-11-28T17:25:52.276 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd/iommu_v2.ko 2023-11-28T17:25:52.276 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds 2023-11-28T17:25:52.277 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-blinkm.ko 2023-11-28T17:25:52.277 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm3530.ko 2023-11-28T17:25:52.277 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lp3944.ko 2023-11-28T17:25:52.277 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-ss4200.ko 2023-11-28T17:25:52.277 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger 2023-11-28T17:25:52.277 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-11-28T17:25:52.277 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-11-28T17:25:52.277 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-11-28T17:25:52.277 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-11-28T17:25:52.277 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-11-28T17:25:52.277 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-11-28T17:25:52.277 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-11-28T17:25:52.278 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-11-28T17:25:52.278 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md 2023-11-28T17:25:52.278 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache 2023-11-28T17:25:52.278 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache/bcache.ko 2023-11-28T17:25:52.278 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-bio-prison.ko 2023-11-28T17:25:52.278 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache-smq.ko 2023-11-28T17:25:52.278 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache.ko 2023-11-28T17:25:52.278 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-crypt.ko 2023-11-28T17:25:52.278 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-delay.ko 2023-11-28T17:25:52.278 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-flakey.ko 2023-11-28T17:25:52.278 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-log-userspace.ko 2023-11-28T17:25:52.279 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-multipath.ko 2023-11-28T17:25:52.279 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-queue-length.ko 2023-11-28T17:25:52.279 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-raid.ko 2023-11-28T17:25:52.279 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-round-robin.ko 2023-11-28T17:25:52.279 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-service-time.ko 2023-11-28T17:25:52.279 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-switch.ko 2023-11-28T17:25:52.279 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-thin-pool.ko 2023-11-28T17:25:52.279 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-verity.ko 2023-11-28T17:25:52.279 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/faulty.ko 2023-11-28T17:25:52.279 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/linear.ko 2023-11-28T17:25:52.279 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/multipath.ko 2023-11-28T17:25:52.279 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data 2023-11-28T17:25:52.279 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-11-28T17:25:52.279 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid0.ko 2023-11-28T17:25:52.280 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid1.ko 2023-11-28T17:25:52.280 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid10.ko 2023-11-28T17:25:52.280 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid456.ko 2023-11-28T17:25:52.280 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message 2023-11-28T17:25:52.280 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion 2023-11-28T17:25:52.280 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptbase.ko 2023-11-28T17:25:52.280 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptctl.ko 2023-11-28T17:25:52.280 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptfc.ko 2023-11-28T17:25:52.280 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptlan.ko 2023-11-28T17:25:52.280 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptsas.ko 2023-11-28T17:25:52.280 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptscsih.ko 2023-11-28T17:25:52.280 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptspi.ko 2023-11-28T17:25:52.280 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd 2023-11-28T17:25:52.281 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_ich.ko 2023-11-28T17:25:52.281 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_sch.ko 2023-11-28T17:25:52.281 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/mfd-core.ko 2023-11-28T17:25:52.281 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/sm501.ko 2023-11-28T17:25:52.281 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/viperboard.ko 2023-11-28T17:25:52.281 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/vx855.ko 2023-11-28T17:25:52.281 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/wl1273-core.ko 2023-11-28T17:25:52.281 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc 2023-11-28T17:25:52.281 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl 2023-11-28T17:25:52.281 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-11-28T17:25:52.281 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds9802als.ko 2023-11-28T17:25:52.281 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds990x.ko 2023-11-28T17:25:52.281 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/bh1770glc.ko 2023-11-28T17:25:52.281 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710 2023-11-28T17:25:52.282 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710/cb710.ko 2023-11-28T17:25:52.282 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom 2023-11-28T17:25:52.282 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/at24.ko 2023-11-28T17:25:52.282 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom.ko 2023-11-28T17:25:52.282 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-11-28T17:25:52.282 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/max6875.ko 2023-11-28T17:25:52.282 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/enclosure.ko 2023-11-28T17:25:52.282 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/hpilo.ko 2023-11-28T17:25:52.282 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29003.ko 2023-11-28T17:25:52.282 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29020.ko 2023-11-28T17:25:52.282 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d 2023-11-28T17:25:52.283 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-11-28T17:25:52.283 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-11-28T17:25:52.283 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei 2023-11-28T17:25:52.283 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-me.ko 2023-11-28T17:25:52.283 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-txe.ko 2023-11-28T17:25:52.283 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei.ko 2023-11-28T17:25:52.283 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru 2023-11-28T17:25:52.283 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru/gru.ko 2023-11-28T17:25:52.283 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp 2023-11-28T17:25:52.283 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xp.ko 2023-11-28T17:25:52.283 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpc.ko 2023-11-28T17:25:52.283 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-11-28T17:25:52.283 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_7xx1.ko 2023-11-28T17:25:52.283 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_core.ko 2023-11-28T17:25:52.284 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tsl2550.ko 2023-11-28T17:25:52.284 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_balloon.ko 2023-11-28T17:25:52.284 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci 2023-11-28T17:25:52.284 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-11-28T17:25:52.284 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net 2023-11-28T17:25:52.284 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding 2023-11-28T17:25:52.284 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding/bonding.ko 2023-11-28T17:25:52.284 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa 2023-11-28T17:25:52.284 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53 2023-11-28T17:25:52.284 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53/b53_common.ko 2023-11-28T17:25:52.284 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/bcm-sf2.ko 2023-11-28T17:25:52.284 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6060.ko 2023-11-28T17:25:52.285 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx 2023-11-28T17:25:52.285 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-11-28T17:25:52.285 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dummy.ko 2023-11-28T17:25:52.285 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/eql.ko 2023-11-28T17:25:52.285 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet 2023-11-28T17:25:52.285 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com 2023-11-28T17:25:52.285 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-11-28T17:25:52.285 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-11-28T17:25:52.285 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-11-28T17:25:52.285 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-11-28T17:25:52.285 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390 2023-11-28T17:25:52.285 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/8390.ko 2023-11-28T17:25:52.285 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-11-28T17:25:52.286 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-11-28T17:25:52.286 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-11-28T17:25:52.286 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec 2023-11-28T17:25:52.286 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-11-28T17:25:52.286 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere 2023-11-28T17:25:52.286 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere/et131x.ko 2023-11-28T17:25:52.286 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon 2023-11-28T17:25:52.286 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-11-28T17:25:52.286 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera 2023-11-28T17:25:52.286 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-11-28T17:25:52.286 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd 2023-11-28T17:25:52.286 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-11-28T17:25:52.287 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-11-28T17:25:52.287 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-11-28T17:25:52.287 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros 2023-11-28T17:25:52.287 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx 2023-11-28T17:25:52.287 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-11-28T17:25:52.287 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c 2023-11-28T17:25:52.287 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-11-28T17:25:52.287 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e 2023-11-28T17:25:52.287 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-11-28T17:25:52.287 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx 2023-11-28T17:25:52.287 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-11-28T17:25:52.287 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-11-28T17:25:52.288 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom 2023-11-28T17:25:52.288 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-11-28T17:25:52.288 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-11-28T17:25:52.288 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-11-28T17:25:52.288 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-11-28T17:25:52.288 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-11-28T17:25:52.288 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet 2023-11-28T17:25:52.288 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-11-28T17:25:52.288 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-11-28T17:25:52.288 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade 2023-11-28T17:25:52.288 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna 2023-11-28T17:25:52.288 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-11-28T17:25:52.289 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio 2023-11-28T17:25:52.289 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb 2023-11-28T17:25:52.289 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-11-28T17:25:52.289 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-11-28T17:25:52.289 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-11-28T17:25:52.289 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-11-28T17:25:52.289 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-11-28T17:25:52.289 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-11-28T17:25:52.289 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-11-28T17:25:52.289 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-11-28T17:25:52.289 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-11-28T17:25:52.289 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco 2023-11-28T17:25:52.290 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic 2023-11-28T17:25:52.290 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-11-28T17:25:52.290 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec 2023-11-28T17:25:52.290 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip 2023-11-28T17:25:52.290 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-11-28T17:25:52.290 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-11-28T17:25:52.290 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-11-28T17:25:52.290 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-11-28T17:25:52.290 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-11-28T17:25:52.290 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-11-28T17:25:52.290 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink 2023-11-28T17:25:52.290 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-11-28T17:25:52.291 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-11-28T17:25:52.291 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dnet.ko 2023-11-28T17:25:52.291 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex 2023-11-28T17:25:52.291 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet 2023-11-28T17:25:52.291 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-11-28T17:25:52.291 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ethoc.ko 2023-11-28T17:25:52.291 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/fealnx.ko 2023-11-28T17:25:52.291 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel 2023-11-28T17:25:52.291 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e100.ko 2023-11-28T17:25:52.291 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000 2023-11-28T17:25:52.291 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-11-28T17:25:52.291 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e 2023-11-28T17:25:52.292 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-11-28T17:25:52.292 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k 2023-11-28T17:25:52.292 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-11-28T17:25:52.292 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e 2023-11-28T17:25:52.292 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-11-28T17:25:52.292 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf 2023-11-28T17:25:52.292 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-11-28T17:25:52.292 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb 2023-11-28T17:25:52.292 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-11-28T17:25:52.292 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf 2023-11-28T17:25:52.292 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-11-28T17:25:52.292 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe 2023-11-28T17:25:52.293 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-11-28T17:25:52.293 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf 2023-11-28T17:25:52.293 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-11-28T17:25:52.293 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/jme.ko 2023-11-28T17:25:52.293 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell 2023-11-28T17:25:52.293 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-11-28T17:25:52.293 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/skge.ko 2023-11-28T17:25:52.293 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-11-28T17:25:52.293 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox 2023-11-28T17:25:52.293 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4 2023-11-28T17:25:52.293 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-11-28T17:25:52.293 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-11-28T17:25:52.293 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5 2023-11-28T17:25:52.294 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-11-28T17:25:52.294 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-11-28T17:25:52.294 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel 2023-11-28T17:25:52.294 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-11-28T17:25:52.294 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom 2023-11-28T17:25:52.294 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge 2023-11-28T17:25:52.294 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-11-28T17:25:52.294 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi 2023-11-28T17:25:52.294 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-11-28T17:25:52.294 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-11-28T17:25:52.295 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion 2023-11-28T17:25:52.295 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-11-28T17:25:52.295 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia 2023-11-28T17:25:52.295 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-11-28T17:25:52.295 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines 2023-11-28T17:25:52.295 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-11-28T17:25:52.295 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-11-28T17:25:52.295 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic 2023-11-28T17:25:52.295 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen 2023-11-28T17:25:52.295 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-11-28T17:25:52.295 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-11-28T17:25:52.295 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-11-28T17:25:52.295 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-11-28T17:25:52.296 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc 2023-11-28T17:25:52.296 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-11-28T17:25:52.296 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek 2023-11-28T17:25:52.296 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-11-28T17:25:52.296 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-11-28T17:25:52.296 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/atp.ko 2023-11-28T17:25:52.296 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-11-28T17:25:52.296 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker 2023-11-28T17:25:52.296 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-11-28T17:25:52.296 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc 2023-11-28T17:25:52.296 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-11-28T17:25:52.296 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan 2023-11-28T17:25:52.296 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-11-28T17:25:52.297 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis 2023-11-28T17:25:52.297 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis190.ko 2023-11-28T17:25:52.297 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis900.ko 2023-11-28T17:25:52.297 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc 2023-11-28T17:25:52.297 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-11-28T17:25:52.297 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-11-28T17:25:52.297 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-11-28T17:25:52.297 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-11-28T17:25:52.297 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro 2023-11-28T17:25:52.297 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac 2023-11-28T17:25:52.297 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-11-28T17:25:52.297 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-11-28T17:25:52.298 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun 2023-11-28T17:25:52.298 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/cassini.ko 2023-11-28T17:25:52.298 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/niu.ko 2023-11-28T17:25:52.298 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sungem.ko 2023-11-28T17:25:52.298 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-11-28T17:25:52.298 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti 2023-11-28T17:25:52.298 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-11-28T17:25:52.298 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti 2023-11-28T17:25:52.298 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti/tlan.ko 2023-11-28T17:25:52.298 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via 2023-11-28T17:25:52.299 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-11-28T17:25:52.299 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-11-28T17:25:52.299 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet 2023-11-28T17:25:52.299 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-11-28T17:25:52.299 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-11-28T17:25:52.299 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom 2023-11-28T17:25:52.299 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-11-28T17:25:52.299 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/geneve.ko 2023-11-28T17:25:52.299 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv 2023-11-28T17:25:52.299 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-11-28T17:25:52.299 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154 2023-11-28T17:25:52.299 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154/fakelb.ko 2023-11-28T17:25:52.299 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ifb.ko 2023-11-28T17:25:52.300 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan 2023-11-28T17:25:52.300 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan/ipvlan.ko 2023-11-28T17:25:52.300 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvlan.ko 2023-11-28T17:25:52.300 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvtap.ko 2023-11-28T17:25:52.300 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio 2023-11-28T17:25:52.300 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio.ko 2023-11-28T17:25:52.300 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-11-28T17:25:52.300 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-11-28T17:25:52.300 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-regmap.ko 2023-11-28T17:25:52.300 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mii.ko 2023-11-28T17:25:52.300 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/net_failover.ko 2023-11-28T17:25:52.301 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/netconsole.ko 2023-11-28T17:25:52.301 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/nlmon.ko 2023-11-28T17:25:52.301 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs 2023-11-28T17:25:52.301 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs-lynx.ko 2023-11-28T17:25:52.301 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-11-28T17:25:52.301 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy 2023-11-28T17:25:52.301 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/amd.ko 2023-11-28T17:25:52.301 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-11-28T17:25:52.301 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-11-28T17:25:52.301 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm7xxx.ko 2023-11-28T17:25:52.301 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm87xx.ko 2023-11-28T17:25:52.301 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/broadcom.ko 2023-11-28T17:25:52.301 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/cicada.ko 2023-11-28T17:25:52.302 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/davicom.ko 2023-11-28T17:25:52.302 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/dp83640.ko 2023-11-28T17:25:52.302 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/et1011c.ko 2023-11-28T17:25:52.302 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/icplus.ko 2023-11-28T17:25:52.302 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/lxt.ko 2023-11-28T17:25:52.302 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/marvell.ko 2023-11-28T17:25:52.302 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/micrel.ko 2023-11-28T17:25:52.302 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/national.ko 2023-11-28T17:25:52.302 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/phylink.ko 2023-11-28T17:25:52.302 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/qsemi.ko 2023-11-28T17:25:52.302 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/realtek.ko 2023-11-28T17:25:52.302 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/smsc.ko 2023-11-28T17:25:52.303 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/ste10Xp.ko 2023-11-28T17:25:52.303 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/vitesse.ko 2023-11-28T17:25:52.303 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp 2023-11-28T17:25:52.303 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/bsd_comp.ko 2023-11-28T17:25:52.303 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_async.ko 2023-11-28T17:25:52.303 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_deflate.ko 2023-11-28T17:25:52.303 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_generic.ko 2023-11-28T17:25:52.303 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_mppe.ko 2023-11-28T17:25:52.303 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_synctty.ko 2023-11-28T17:25:52.303 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppoe.ko 2023-11-28T17:25:52.303 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppox.ko 2023-11-28T17:25:52.303 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pptp.ko 2023-11-28T17:25:52.303 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip 2023-11-28T17:25:52.304 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slhc.ko 2023-11-28T17:25:52.304 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slip.ko 2023-11-28T17:25:52.304 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/sungem_phy.ko 2023-11-28T17:25:52.304 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tap.ko 2023-11-28T17:25:52.304 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team 2023-11-28T17:25:52.304 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team.ko 2023-11-28T17:25:52.304 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_activebackup.ko 2023-11-28T17:25:52.304 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_broadcast.ko 2023-11-28T17:25:52.304 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-11-28T17:25:52.304 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_random.ko 2023-11-28T17:25:52.304 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-11-28T17:25:52.304 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tun.ko 2023-11-28T17:25:52.304 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/veth.ko 2023-11-28T17:25:52.305 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/virtio_net.ko 2023-11-28T17:25:52.305 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3 2023-11-28T17:25:52.305 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-11-28T17:25:52.305 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan 2023-11-28T17:25:52.305 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan/vxlan.ko 2023-11-28T17:25:52.305 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback 2023-11-28T17:25:52.305 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback/xen-netback.ko 2023-11-28T17:25:52.305 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netfront.ko 2023-11-28T17:25:52.305 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb 2023-11-28T17:25:52.305 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb/ntb.ko 2023-11-28T17:25:52.305 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme 2023-11-28T17:25:52.305 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host 2023-11-28T17:25:52.305 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme-core.ko 2023-11-28T17:25:52.306 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme.ko 2023-11-28T17:25:52.306 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport 2023-11-28T17:25:52.306 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport.ko 2023-11-28T17:25:52.306 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_cs.ko 2023-11-28T17:25:52.306 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_pc.ko 2023-11-28T17:25:52.306 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_serial.ko 2023-11-28T17:25:52.306 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci 2023-11-28T17:25:52.306 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug 2023-11-28T17:25:52.307 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-11-28T17:25:52.307 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie 2023-11-28T17:25:52.307 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie/aer_inject.ko 2023-11-28T17:25:52.307 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/xen-pcifront.ko 2023-11-28T17:25:52.307 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia 2023-11-28T17:25:52.307 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/i82092.ko 2023-11-28T17:25:52.307 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pd6729.ko 2023-11-28T17:25:52.307 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/yenta_socket.ko 2023-11-28T17:25:52.307 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl 2023-11-28T17:25:52.307 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel 2023-11-28T17:25:52.308 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-11-28T17:25:52.308 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform 2023-11-28T17:25:52.308 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86 2023-11-28T17:25:52.308 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acer-wmi.ko 2023-11-28T17:25:52.308 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acerhdf.ko 2023-11-28T17:25:52.308 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/apple-gmux.ko 2023-11-28T17:25:52.308 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-laptop.ko 2023-11-28T17:25:52.308 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-11-28T17:25:52.308 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-wmi.ko 2023-11-28T17:25:52.308 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/classmate-laptop.ko 2023-11-28T17:25:52.308 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-11-28T17:25:52.308 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-11-28T17:25:52.308 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-11-28T17:25:52.309 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-11-28T17:25:52.309 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/hdaps.ko 2023-11-28T17:25:52.309 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel 2023-11-28T17:25:52.309 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-11-28T17:25:52.309 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel_ips.ko 2023-11-28T17:25:52.309 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/msi-wmi.ko 2023-11-28T17:25:52.309 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/mxm-wmi.ko 2023-11-28T17:25:52.309 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-11-28T17:25:52.309 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-laptop.ko 2023-11-28T17:25:52.310 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-q10.ko 2023-11-28T17:25:52.310 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/topstar-laptop.ko 2023-11-28T17:25:52.310 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-11-28T17:25:52.310 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_haps.ko 2023-11-28T17:25:52.310 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi-bmof.ko 2023-11-28T17:25:52.310 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi.ko 2023-11-28T17:25:52.310 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap 2023-11-28T17:25:52.310 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_common.ko 2023-11-28T17:25:52.310 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_msr.ko 2023-11-28T17:25:52.310 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps 2023-11-28T17:25:52.310 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients 2023-11-28T17:25:52.310 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-gpio.ko 2023-11-28T17:25:52.311 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-ldisc.ko 2023-11-28T17:25:52.311 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps_parport.ko 2023-11-28T17:25:52.311 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/pps_core.ko 2023-11-28T17:25:52.311 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp 2023-11-28T17:25:52.311 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp.ko 2023-11-28T17:25:52.311 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp_kvm.ko 2023-11-28T17:25:52.311 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc 2023-11-28T17:25:52.311 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-bq32k.ko 2023-11-28T17:25:52.311 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1286.ko 2023-11-28T17:25:52.311 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1307.ko 2023-11-28T17:25:52.311 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1374.ko 2023-11-28T17:25:52.311 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1511.ko 2023-11-28T17:25:52.312 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1553.ko 2023-11-28T17:25:52.312 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1672.ko 2023-11-28T17:25:52.312 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1742.ko 2023-11-28T17:25:52.312 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds2404.ko 2023-11-28T17:25:52.312 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds3232.ko 2023-11-28T17:25:52.312 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-em3027.ko 2023-11-28T17:25:52.312 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-fm3130.ko 2023-11-28T17:25:52.312 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl12022.ko 2023-11-28T17:25:52.312 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl1208.ko 2023-11-28T17:25:52.312 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m41t80.ko 2023-11-28T17:25:52.312 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t35.ko 2023-11-28T17:25:52.312 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t59.ko 2023-11-28T17:25:52.313 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max6900.ko 2023-11-28T17:25:52.313 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-msm6242.ko 2023-11-28T17:25:52.313 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf2127.ko 2023-11-28T17:25:52.313 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf85063.ko 2023-11-28T17:25:52.313 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8523.ko 2023-11-28T17:25:52.313 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8563.ko 2023-11-28T17:25:52.313 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8583.ko 2023-11-28T17:25:52.313 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rp5c01.ko 2023-11-28T17:25:52.313 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rs5c372.ko 2023-11-28T17:25:52.313 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-11-28T17:25:52.313 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8025.ko 2023-11-28T17:25:52.313 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8581.ko 2023-11-28T17:25:52.313 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-11-28T17:25:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-x1205.ko 2023-11-28T17:25:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi 2023-11-28T17:25:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-9xxx.ko 2023-11-28T17:25:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-sas.ko 2023-11-28T17:25:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-xxxx.ko 2023-11-28T17:25:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/BusLogic.ko 2023-11-28T17:25:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/a100u2w.ko 2023-11-28T17:25:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid 2023-11-28T17:25:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid/aacraid.ko 2023-11-28T17:25:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/advansys.ko 2023-11-28T17:25:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx 2023-11-28T17:25:52.314 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-11-28T17:25:52.315 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-11-28T17:25:52.315 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/am53c974.ko 2023-11-28T17:25:52.315 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr 2023-11-28T17:25:52.315 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-11-28T17:25:52.315 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/atp870u.ko 2023-11-28T17:25:52.315 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi 2023-11-28T17:25:52.315 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-11-28T17:25:52.315 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa 2023-11-28T17:25:52.315 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa/bfa.ko 2023-11-28T17:25:52.315 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc 2023-11-28T17:25:52.315 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-11-28T17:25:52.315 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i 2023-11-28T17:25:52.315 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-11-28T17:25:52.316 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ch.ko 2023-11-28T17:25:52.316 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor 2023-11-28T17:25:52.316 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor/csiostor.ko 2023-11-28T17:25:52.316 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi 2023-11-28T17:25:52.316 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i 2023-11-28T17:25:52.316 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-11-28T17:25:52.316 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i 2023-11-28T17:25:52.316 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-11-28T17:25:52.316 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-11-28T17:25:52.316 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dc395x.ko 2023-11-28T17:25:52.316 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler 2023-11-28T17:25:52.317 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-11-28T17:25:52.317 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-11-28T17:25:52.317 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-11-28T17:25:52.317 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-11-28T17:25:52.317 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dmx3191d.ko 2023-11-28T17:25:52.317 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r 2023-11-28T17:25:52.317 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r/esas2r.ko 2023-11-28T17:25:52.317 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esp_scsi.ko 2023-11-28T17:25:52.317 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe 2023-11-28T17:25:52.317 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/fcoe.ko 2023-11-28T17:25:52.317 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-11-28T17:25:52.317 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic 2023-11-28T17:25:52.318 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic/fnic.ko 2023-11-28T17:25:52.318 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hpsa.ko 2023-11-28T17:25:52.318 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hptiop.ko 2023-11-28T17:25:52.318 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hv_storvsc.ko 2023-11-28T17:25:52.318 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/initio.ko 2023-11-28T17:25:52.318 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ipr.ko 2023-11-28T17:25:52.318 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ips.ko 2023-11-28T17:25:52.318 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci 2023-11-28T17:25:52.318 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci/isci.ko 2023-11-28T17:25:52.319 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-11-28T17:25:52.319 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_tcp.ko 2023-11-28T17:25:52.319 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc 2023-11-28T17:25:52.319 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc/libfc.ko 2023-11-28T17:25:52.319 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi.ko 2023-11-28T17:25:52.319 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi_tcp.ko 2023-11-28T17:25:52.319 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas 2023-11-28T17:25:52.319 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas/libsas.ko 2023-11-28T17:25:52.319 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc 2023-11-28T17:25:52.319 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc/lpfc.ko 2023-11-28T17:25:52.319 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid 2023-11-28T17:25:52.319 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid.ko 2023-11-28T17:25:52.320 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-11-28T17:25:52.320 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-11-28T17:25:52.320 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-11-28T17:25:52.320 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas 2023-11-28T17:25:52.320 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-11-28T17:25:52.320 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas 2023-11-28T17:25:52.320 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas/mvsas.ko 2023-11-28T17:25:52.320 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvumi.ko 2023-11-28T17:25:52.320 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001 2023-11-28T17:25:52.320 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-11-28T17:25:52.320 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pmcraid.ko 2023-11-28T17:25:52.320 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla1280.ko 2023-11-28T17:25:52.321 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx 2023-11-28T17:25:52.321 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-11-28T17:25:52.321 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-11-28T17:25:52.321 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx 2023-11-28T17:25:52.321 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-11-28T17:25:52.321 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/raid_class.ko 2023-11-28T17:25:52.321 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_debug.ko 2023-11-28T17:25:52.321 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_fc.ko 2023-11-28T17:25:52.321 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-11-28T17:25:52.321 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_sas.ko 2023-11-28T17:25:52.321 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_spi.ko 2023-11-28T17:25:52.321 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_srp.ko 2023-11-28T17:25:52.322 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ses.ko 2023-11-28T17:25:52.322 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/st.ko 2023-11-28T17:25:52.322 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/stex.ko 2023-11-28T17:25:52.322 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2 2023-11-28T17:25:52.322 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-11-28T17:25:52.322 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/virtio_scsi.ko 2023-11-28T17:25:52.322 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/vmw_pvscsi.ko 2023-11-28T17:25:52.322 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/wd719x.ko 2023-11-28T17:25:52.322 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb 2023-11-28T17:25:52.322 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb/ssb.ko 2023-11-28T17:25:52.323 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target 2023-11-28T17:25:52.323 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi 2023-11-28T17:25:52.323 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-11-28T17:25:52.323 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback 2023-11-28T17:25:52.323 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback/tcm_loop.ko 2023-11-28T17:25:52.323 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_file.ko 2023-11-28T17:25:52.323 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_iblock.ko 2023-11-28T17:25:52.323 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_mod.ko 2023-11-28T17:25:52.323 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_pscsi.ko 2023-11-28T17:25:52.323 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_user.ko 2023-11-28T17:25:52.324 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc 2023-11-28T17:25:52.324 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-11-28T17:25:52.324 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal 2023-11-28T17:25:52.324 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel 2023-11-28T17:25:52.324 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal 2023-11-28T17:25:52.324 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-11-28T17:25:52.324 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-11-28T17:25:52.324 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-11-28T17:25:52.324 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-11-28T17:25:52.324 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-11-28T17:25:52.324 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-11-28T17:25:52.325 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-11-28T17:25:52.325 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-11-28T17:25:52.325 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-11-28T17:25:52.325 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-11-28T17:25:52.325 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-11-28T17:25:52.325 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-11-28T17:25:52.325 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-11-28T17:25:52.325 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-11-28T17:25:52.325 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-11-28T17:25:52.325 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty 2023-11-28T17:25:52.325 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless 2023-11-28T17:25:52.326 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-11-28T17:25:52.326 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_gsm.ko 2023-11-28T17:25:52.326 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_hdlc.ko 2023-11-28T17:25:52.326 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/nozomi.ko 2023-11-28T17:25:52.326 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial 2023-11-28T17:25:52.326 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250 2023-11-28T17:25:52.326 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-11-28T17:25:52.326 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/arc_uart.ko 2023-11-28T17:25:52.326 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm 2023-11-28T17:25:52.326 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm/jsm.ko 2023-11-28T17:25:52.326 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs 2023-11-28T17:25:52.327 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core 2023-11-28T17:25:52.327 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core/ufshcd-core.ko 2023-11-28T17:25:52.327 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host 2023-11-28T17:25:52.327 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-11-28T17:25:52.327 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio 2023-11-28T17:25:52.327 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio.ko 2023-11-28T17:25:52.327 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_aec.ko 2023-11-28T17:25:52.327 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_cif.ko 2023-11-28T17:25:52.327 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_pci_generic.ko 2023-11-28T17:25:52.327 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_sercos3.ko 2023-11-28T17:25:52.327 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb 2023-11-28T17:25:52.328 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class 2023-11-28T17:25:52.328 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-acm.ko 2023-11-28T17:25:52.328 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-wdm.ko 2023-11-28T17:25:52.328 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usblp.ko 2023-11-28T17:25:52.328 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usbtmc.ko 2023-11-28T17:25:52.328 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host 2023-11-28T17:25:52.328 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/sl811-hcd.ko 2023-11-28T17:25:52.328 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image 2023-11-28T17:25:52.328 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/mdc800.ko 2023-11-28T17:25:52.328 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/microtek.ko 2023-11-28T17:25:52.328 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc 2023-11-28T17:25:52.328 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/adutux.ko 2023-11-28T17:25:52.329 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/appledisplay.ko 2023-11-28T17:25:52.329 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi26.ko 2023-11-28T17:25:52.329 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi62.ko 2023-11-28T17:25:52.329 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ezusb.ko 2023-11-28T17:25:52.329 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/idmouse.ko 2023-11-28T17:25:52.329 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/iowarrior.ko 2023-11-28T17:25:52.329 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/isight_firmware.ko 2023-11-28T17:25:52.329 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ldusb.ko 2023-11-28T17:25:52.329 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/legousbtower.ko 2023-11-28T17:25:52.329 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga 2023-11-28T17:25:52.330 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-11-28T17:25:52.330 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/trancevibrator.ko 2023-11-28T17:25:52.330 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usb3503.ko 2023-11-28T17:25:52.330 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usblcd.ko 2023-11-28T17:25:52.330 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usbsevseg.ko 2023-11-28T17:25:52.330 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/uss720.ko 2023-11-28T17:25:52.330 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/yurex.ko 2023-11-28T17:25:52.330 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy 2023-11-28T17:25:52.330 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-generic.ko 2023-11-28T17:25:52.330 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial 2023-11-28T17:25:52.330 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/aircable.ko 2023-11-28T17:25:52.331 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ark3116.ko 2023-11-28T17:25:52.331 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/belkin_sa.ko 2023-11-28T17:25:52.331 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ch341.ko 2023-11-28T17:25:52.331 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cp210x.ko 2023-11-28T17:25:52.331 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cyberjack.ko 2023-11-28T17:25:52.331 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cypress_m8.ko 2023-11-28T17:25:52.331 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/digi_acceleport.ko 2023-11-28T17:25:52.331 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/empeg.ko 2023-11-28T17:25:52.333 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ftdi_sio.ko 2023-11-28T17:25:52.333 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/garmin_gps.ko 2023-11-28T17:25:52.333 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_edgeport.ko 2023-11-28T17:25:52.333 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_ti.ko 2023-11-28T17:25:52.333 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipaq.ko 2023-11-28T17:25:52.333 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipw.ko 2023-11-28T17:25:52.333 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ir-usb.ko 2023-11-28T17:25:52.333 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-11-28T17:25:52.333 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan.ko 2023-11-28T17:25:52.333 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan_pda.ko 2023-11-28T17:25:52.333 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kl5kusb105.ko 2023-11-28T17:25:52.334 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kobil_sct.ko 2023-11-28T17:25:52.334 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mct_u232.ko 2023-11-28T17:25:52.334 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7720.ko 2023-11-28T17:25:52.334 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7840.ko 2023-11-28T17:25:52.334 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/navman.ko 2023-11-28T17:25:52.334 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/omninet.ko 2023-11-28T17:25:52.334 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/opticon.ko 2023-11-28T17:25:52.334 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/option.ko 2023-11-28T17:25:52.334 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/oti6858.ko 2023-11-28T17:25:52.334 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/pl2303.ko 2023-11-28T17:25:52.334 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcaux.ko 2023-11-28T17:25:52.335 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcserial.ko 2023-11-28T17:25:52.335 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/quatech2.ko 2023-11-28T17:25:52.335 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/safe_serial.ko 2023-11-28T17:25:52.335 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/sierra.ko 2023-11-28T17:25:52.335 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/spcp8x5.ko 2023-11-28T17:25:52.335 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ssu100.ko 2023-11-28T17:25:52.335 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/symbolserial.ko 2023-11-28T17:25:52.335 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-11-28T17:25:52.335 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-11-28T17:25:52.335 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_debug.ko 2023-11-28T17:25:52.335 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_wwan.ko 2023-11-28T17:25:52.336 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/visor.ko 2023-11-28T17:25:52.336 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/whiteheat.ko 2023-11-28T17:25:52.336 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/xsens_mt.ko 2023-11-28T17:25:52.336 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage 2023-11-28T17:25:52.336 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/uas.ko 2023-11-28T17:25:52.336 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-alauda.ko 2023-11-28T17:25:52.336 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-cypress.ko 2023-11-28T17:25:52.336 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-datafab.ko 2023-11-28T17:25:52.336 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-11-28T17:25:52.336 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-freecom.ko 2023-11-28T17:25:52.336 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-isd200.ko 2023-11-28T17:25:52.336 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-11-28T17:25:52.337 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-karma.ko 2023-11-28T17:25:52.337 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-onetouch.ko 2023-11-28T17:25:52.337 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-realtek.ko 2023-11-28T17:25:52.337 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr09.ko 2023-11-28T17:25:52.337 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr55.ko 2023-11-28T17:25:52.337 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-usbat.ko 2023-11-28T17:25:52.337 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/usb-storage.ko 2023-11-28T17:25:52.337 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip 2023-11-28T17:25:52.337 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-core.ko 2023-11-28T17:25:52.337 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-host.ko 2023-11-28T17:25:52.337 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-11-28T17:25:52.337 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio 2023-11-28T17:25:52.337 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci 2023-11-28T17:25:52.338 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-11-28T17:25:52.338 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci.ko 2023-11-28T17:25:52.338 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio.ko 2023-11-28T17:25:52.338 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-11-28T17:25:52.338 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost 2023-11-28T17:25:52.338 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost.ko 2023-11-28T17:25:52.338 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_iotlb.ko 2023-11-28T17:25:52.338 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_net.ko 2023-11-28T17:25:52.338 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_scsi.ko 2023-11-28T17:25:52.338 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video 2023-11-28T17:25:52.338 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight 2023-11-28T17:25:52.338 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/apple_bl.ko 2023-11-28T17:25:52.339 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lcd.ko 2023-11-28T17:25:52.339 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/platform_lcd.ko 2023-11-28T17:25:52.339 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio 2023-11-28T17:25:52.339 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio.ko 2023-11-28T17:25:52.339 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_balloon.ko 2023-11-28T17:25:52.339 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_mmio.ko 2023-11-28T17:25:52.339 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci.ko 2023-11-28T17:25:52.339 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-11-28T17:25:52.339 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-11-28T17:25:52.339 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_ring.ko 2023-11-28T17:25:52.339 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1 2023-11-28T17:25:52.339 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters 2023-11-28T17:25:52.339 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2482.ko 2023-11-28T17:25:52.340 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2490.ko 2023-11-28T17:25:52.340 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves 2023-11-28T17:25:52.340 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-11-28T17:25:52.340 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-11-28T17:25:52.340 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-11-28T17:25:52.340 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-11-28T17:25:52.340 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-11-28T17:25:52.340 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-11-28T17:25:52.340 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-11-28T17:25:52.340 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-11-28T17:25:52.340 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-11-28T17:25:52.341 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_smem.ko 2023-11-28T17:25:52.341 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_therm.ko 2023-11-28T17:25:52.341 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/wire.ko 2023-11-28T17:25:52.341 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog 2023-11-28T17:25:52.341 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim1535_wdt.ko 2023-11-28T17:25:52.341 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim7101_wdt.ko 2023-11-28T17:25:52.341 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/f71808e_wdt.ko 2023-11-28T17:25:52.341 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/hpwdt.ko 2023-11-28T17:25:52.341 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/i6300esb.ko 2023-11-28T17:25:52.341 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-11-28T17:25:52.341 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_wdt.ko 2023-11-28T17:25:52.341 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ib700wdt.ko 2023-11-28T17:25:52.342 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ibmasr.ko 2023-11-28T17:25:52.342 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-11-28T17:25:52.342 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it8712f_wdt.ko 2023-11-28T17:25:52.342 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it87_wdt.ko 2023-11-28T17:25:52.342 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/machzwd.ko 2023-11-28T17:25:52.342 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/nv_tco.ko 2023-11-28T17:25:52.342 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_pci.ko 2023-11-28T17:25:52.342 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_usb.ko 2023-11-28T17:25:52.342 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-11-28T17:25:52.342 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sch311x_wdt.ko 2023-11-28T17:25:52.342 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/softdog.ko 2023-11-28T17:25:52.342 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sp5100_tco.ko 2023-11-28T17:25:52.343 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/via_wdt.ko 2023-11-28T17:25:52.343 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-11-28T17:25:52.343 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83877f_wdt.ko 2023-11-28T17:25:52.343 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83977f_wdt.ko 2023-11-28T17:25:52.343 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wdt_pci.ko 2023-11-28T17:25:52.343 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/xen_wdt.ko 2023-11-28T17:25:52.343 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen 2023-11-28T17:25:52.343 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-acpi-processor.ko 2023-11-28T17:25:52.343 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-evtchn.ko 2023-11-28T17:25:52.344 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntalloc.ko 2023-11-28T17:25:52.344 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntdev.ko 2023-11-28T17:25:52.344 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback 2023-11-28T17:25:52.344 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-11-28T17:25:52.344 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-privcmd.ko 2023-11-28T17:25:52.344 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs 2023-11-28T17:25:52.344 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs/xenfs.ko 2023-11-28T17:25:52.344 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs 2023-11-28T17:25:52.344 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p 2023-11-28T17:25:52.344 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p/9p.ko 2023-11-28T17:25:52.344 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs 2023-11-28T17:25:52.344 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs/affs.ko 2023-11-28T17:25:52.345 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs 2023-11-28T17:25:52.345 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs/befs.ko 2023-11-28T17:25:52.345 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/binfmt_misc.ko 2023-11-28T17:25:52.345 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs 2023-11-28T17:25:52.345 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs/btrfs.ko 2023-11-28T17:25:52.345 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles 2023-11-28T17:25:52.345 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles/cachefiles.ko 2023-11-28T17:25:52.345 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph 2023-11-28T17:25:52.345 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph/ceph.ko 2023-11-28T17:25:52.345 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda 2023-11-28T17:25:52.345 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda/coda.ko 2023-11-28T17:25:52.345 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs 2023-11-28T17:25:52.345 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs/cramfs.ko 2023-11-28T17:25:52.346 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm 2023-11-28T17:25:52.346 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm/dlm.ko 2023-11-28T17:25:52.346 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs 2023-11-28T17:25:52.346 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs/ecryptfs.ko 2023-11-28T17:25:52.346 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs 2023-11-28T17:25:52.346 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs/f2fs.ko 2023-11-28T17:25:52.346 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat 2023-11-28T17:25:52.346 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/fat.ko 2023-11-28T17:25:52.346 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/msdos.ko 2023-11-28T17:25:52.346 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/vfat.ko 2023-11-28T17:25:52.346 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache 2023-11-28T17:25:52.346 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache/fscache.ko 2023-11-28T17:25:52.347 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse 2023-11-28T17:25:52.347 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/cuse.ko 2023-11-28T17:25:52.347 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/fuse.ko 2023-11-28T17:25:52.347 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2 2023-11-28T17:25:52.347 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2/gfs2.ko 2023-11-28T17:25:52.347 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs 2023-11-28T17:25:52.347 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs/hfs.ko 2023-11-28T17:25:52.347 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus 2023-11-28T17:25:52.347 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus/hfsplus.ko 2023-11-28T17:25:52.347 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs 2023-11-28T17:25:52.347 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs/isofs.ko 2023-11-28T17:25:52.347 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs 2023-11-28T17:25:52.347 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs/jfs.ko 2023-11-28T17:25:52.348 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd 2023-11-28T17:25:52.348 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd/lockd.ko 2023-11-28T17:25:52.348 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix 2023-11-28T17:25:52.348 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix/minix.ko 2023-11-28T17:25:52.348 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs 2023-11-28T17:25:52.348 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs/netfs.ko 2023-11-28T17:25:52.348 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs 2023-11-28T17:25:52.348 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout 2023-11-28T17:25:52.348 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-11-28T17:25:52.348 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout 2023-11-28T17:25:52.348 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-11-28T17:25:52.348 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout 2023-11-28T17:25:52.349 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-11-28T17:25:52.349 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfs.ko 2023-11-28T17:25:52.349 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv3.ko 2023-11-28T17:25:52.349 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv4.ko 2023-11-28T17:25:52.349 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common 2023-11-28T17:25:52.349 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/grace.ko 2023-11-28T17:25:52.349 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/nfs_acl.ko 2023-11-28T17:25:52.349 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd 2023-11-28T17:25:52.349 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd/nfsd.ko 2023-11-28T17:25:52.349 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2 2023-11-28T17:25:52.349 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2/nilfs2.ko 2023-11-28T17:25:52.349 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls 2023-11-28T17:25:52.350 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-celtic.ko 2023-11-28T17:25:52.350 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-centeuro.ko 2023-11-28T17:25:52.350 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-croatian.ko 2023-11-28T17:25:52.350 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-cyrillic.ko 2023-11-28T17:25:52.350 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-gaelic.ko 2023-11-28T17:25:52.350 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-greek.ko 2023-11-28T17:25:52.350 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-iceland.ko 2023-11-28T17:25:52.350 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-inuit.ko 2023-11-28T17:25:52.350 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-roman.ko 2023-11-28T17:25:52.350 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-romanian.ko 2023-11-28T17:25:52.350 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-turkish.ko 2023-11-28T17:25:52.350 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1250.ko 2023-11-28T17:25:52.351 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1251.ko 2023-11-28T17:25:52.351 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1255.ko 2023-11-28T17:25:52.351 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp737.ko 2023-11-28T17:25:52.351 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp775.ko 2023-11-28T17:25:52.351 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp850.ko 2023-11-28T17:25:52.351 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp852.ko 2023-11-28T17:25:52.351 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp855.ko 2023-11-28T17:25:52.351 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp857.ko 2023-11-28T17:25:52.351 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp860.ko 2023-11-28T17:25:52.351 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp861.ko 2023-11-28T17:25:52.351 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp862.ko 2023-11-28T17:25:52.351 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp863.ko 2023-11-28T17:25:52.352 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp864.ko 2023-11-28T17:25:52.352 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp865.ko 2023-11-28T17:25:52.352 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp866.ko 2023-11-28T17:25:52.352 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp869.ko 2023-11-28T17:25:52.352 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp874.ko 2023-11-28T17:25:52.352 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp932.ko 2023-11-28T17:25:52.352 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp936.ko 2023-11-28T17:25:52.352 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp949.ko 2023-11-28T17:25:52.352 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp950.ko 2023-11-28T17:25:52.352 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_euc-jp.ko 2023-11-28T17:25:52.352 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-1.ko 2023-11-28T17:25:52.352 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-13.ko 2023-11-28T17:25:52.352 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-14.ko 2023-11-28T17:25:52.353 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-15.ko 2023-11-28T17:25:52.353 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-2.ko 2023-11-28T17:25:52.353 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-3.ko 2023-11-28T17:25:52.353 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-4.ko 2023-11-28T17:25:52.353 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-5.ko 2023-11-28T17:25:52.353 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-6.ko 2023-11-28T17:25:52.353 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-7.ko 2023-11-28T17:25:52.353 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-9.ko 2023-11-28T17:25:52.353 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-r.ko 2023-11-28T17:25:52.353 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-ru.ko 2023-11-28T17:25:52.353 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-u.ko 2023-11-28T17:25:52.353 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_ucs2_utils.ko 2023-11-28T17:25:52.353 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_utf8.ko 2023-11-28T17:25:52.354 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2 2023-11-28T17:25:52.354 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster 2023-11-28T17:25:52.354 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-11-28T17:25:52.354 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm 2023-11-28T17:25:52.354 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-11-28T17:25:52.354 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs 2023-11-28T17:25:52.354 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-11-28T17:25:52.354 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2.ko 2023-11-28T17:25:52.354 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-11-28T17:25:52.354 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-11-28T17:25:52.354 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-11-28T17:25:52.355 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs 2023-11-28T17:25:52.355 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs/overlay.ko 2023-11-28T17:25:52.355 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore 2023-11-28T17:25:52.355 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore/ramoops.ko 2023-11-28T17:25:52.355 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs 2023-11-28T17:25:52.355 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs/reiserfs.ko 2023-11-28T17:25:52.355 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs 2023-11-28T17:25:52.355 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs/romfs.ko 2023-11-28T17:25:52.355 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb 2023-11-28T17:25:52.355 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client 2023-11-28T17:25:52.355 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client/cifs.ko 2023-11-28T17:25:52.355 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common 2023-11-28T17:25:52.356 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_arc4.ko 2023-11-28T17:25:52.356 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_md4.ko 2023-11-28T17:25:52.356 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs 2023-11-28T17:25:52.356 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs/squashfs.ko 2023-11-28T17:25:52.356 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv 2023-11-28T17:25:52.356 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv/sysv.ko 2023-11-28T17:25:52.356 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf 2023-11-28T17:25:52.356 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf/udf.ko 2023-11-28T17:25:52.356 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs 2023-11-28T17:25:52.356 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs/ufs.ko 2023-11-28T17:25:52.356 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs 2023-11-28T17:25:52.356 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs/xfs.ko 2023-11-28T17:25:52.356 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel 2023-11-28T17:25:52.357 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu 2023-11-28T17:25:52.357 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu/rcutorture.ko 2023-11-28T17:25:52.357 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/torture.ko 2023-11-28T17:25:52.357 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace 2023-11-28T17:25:52.357 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-11-28T17:25:52.357 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib 2023-11-28T17:25:52.357 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/asn1_encoder.ko 2023-11-28T17:25:52.357 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc-itu-t.ko 2023-11-28T17:25:52.357 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc8.ko 2023-11-28T17:25:52.357 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto 2023-11-28T17:25:52.357 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libarc4.ko 2023-11-28T17:25:52.357 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libdes.ko 2023-11-28T17:25:52.358 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/libcrc32c.ko 2023-11-28T17:25:52.358 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lru_cache.ko 2023-11-28T17:25:52.358 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4 2023-11-28T17:25:52.358 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4_compress.ko 2023-11-28T17:25:52.358 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4hc_compress.ko 2023-11-28T17:25:52.358 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math 2023-11-28T17:25:52.358 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math/cordic.ko 2023-11-28T17:25:52.358 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6 2023-11-28T17:25:52.358 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6/raid6_pq.ko 2023-11-28T17:25:52.358 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon 2023-11-28T17:25:52.358 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon/reed_solomon.ko 2023-11-28T17:25:52.358 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_bm.ko 2023-11-28T17:25:52.359 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_fsm.ko 2023-11-28T17:25:52.359 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_kmp.ko 2023-11-28T17:25:52.359 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd 2023-11-28T17:25:52.359 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd/zstd_compress.ko 2023-11-28T17:25:52.359 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm 2023-11-28T17:25:52.359 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm/hwpoison-inject.ko 2023-11-28T17:25:52.359 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net 2023-11-28T17:25:52.359 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan 2023-11-28T17:25:52.359 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/6lowpan.ko 2023-11-28T17:25:52.359 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_dest.ko 2023-11-28T17:25:52.359 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_fragment.ko 2023-11-28T17:25:52.359 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_hop.ko 2023-11-28T17:25:52.360 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_ipv6.ko 2023-11-28T17:25:52.360 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_mobility.ko 2023-11-28T17:25:52.360 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_routing.ko 2023-11-28T17:25:52.360 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_udp.ko 2023-11-28T17:25:52.360 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802 2023-11-28T17:25:52.360 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/garp.ko 2023-11-28T17:25:52.360 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/mrp.ko 2023-11-28T17:25:52.360 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/p8022.ko 2023-11-28T17:25:52.360 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/psnap.ko 2023-11-28T17:25:52.360 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/stp.ko 2023-11-28T17:25:52.360 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q 2023-11-28T17:25:52.360 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q/8021q.ko 2023-11-28T17:25:52.361 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p 2023-11-28T17:25:52.361 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet.ko 2023-11-28T17:25:52.361 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_fd.ko 2023-11-28T17:25:52.361 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_rdma.ko 2023-11-28T17:25:52.361 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_virtio.ko 2023-11-28T17:25:52.361 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv 2023-11-28T17:25:52.361 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv/batman-adv.ko 2023-11-28T17:25:52.361 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge 2023-11-28T17:25:52.361 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/br_netfilter.ko 2023-11-28T17:25:52.361 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/bridge.ko 2023-11-28T17:25:52.361 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter 2023-11-28T17:25:52.362 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-11-28T17:25:52.362 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_among.ko 2023-11-28T17:25:52.362 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arp.ko 2023-11-28T17:25:52.362 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-11-28T17:25:52.362 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-11-28T17:25:52.362 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip.ko 2023-11-28T17:25:52.362 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-11-28T17:25:52.362 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_limit.ko 2023-11-28T17:25:52.362 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_log.ko 2023-11-28T17:25:52.362 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark.ko 2023-11-28T17:25:52.362 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-11-28T17:25:52.362 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-11-28T17:25:52.363 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-11-28T17:25:52.363 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-11-28T17:25:52.363 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_snat.ko 2023-11-28T17:25:52.363 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_stp.ko 2023-11-28T17:25:52.363 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-11-28T17:25:52.363 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-11-28T17:25:52.363 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-11-28T17:25:52.363 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-11-28T17:25:52.363 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtables.ko 2023-11-28T17:25:52.363 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-11-28T17:25:52.363 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-11-28T17:25:52.363 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-11-28T17:25:52.363 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph 2023-11-28T17:25:52.364 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph/libceph.ko 2023-11-28T17:25:52.364 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core 2023-11-28T17:25:52.364 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/failover.ko 2023-11-28T17:25:52.364 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/pktgen.ko 2023-11-28T17:25:52.364 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp 2023-11-28T17:25:52.364 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp.ko 2023-11-28T17:25:52.364 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_diag.ko 2023-11-28T17:25:52.364 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv4.ko 2023-11-28T17:25:52.364 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv6.ko 2023-11-28T17:25:52.364 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver 2023-11-28T17:25:52.364 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver/dns_resolver.ko 2023-11-28T17:25:52.364 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa 2023-11-28T17:25:52.365 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/dsa_core.ko 2023-11-28T17:25:52.365 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_brcm.ko 2023-11-28T17:25:52.365 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_dsa.ko 2023-11-28T17:25:52.365 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_none.ko 2023-11-28T17:25:52.365 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_trailer.ko 2023-11-28T17:25:52.365 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154 2023-11-28T17:25:52.365 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan 2023-11-28T17:25:52.365 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-11-28T17:25:52.365 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154.ko 2023-11-28T17:25:52.365 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154_socket.ko 2023-11-28T17:25:52.365 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4 2023-11-28T17:25:52.365 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ah4.ko 2023-11-28T17:25:52.366 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/esp4.ko 2023-11-28T17:25:52.366 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/fou.ko 2023-11-28T17:25:52.366 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/gre.ko 2023-11-28T17:25:52.366 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/inet_diag.ko 2023-11-28T17:25:52.366 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_gre.ko 2023-11-28T17:25:52.366 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_tunnel.ko 2023-11-28T17:25:52.366 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_vti.ko 2023-11-28T17:25:52.366 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipcomp.ko 2023-11-28T17:25:52.366 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipip.ko 2023-11-28T17:25:52.366 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter 2023-11-28T17:25:52.366 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arp_tables.ko 2023-11-28T17:25:52.367 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-11-28T17:25:52.367 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-11-28T17:25:52.367 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-11-28T17:25:52.367 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-11-28T17:25:52.367 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-11-28T17:25:52.367 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-11-28T17:25:52.367 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-11-28T17:25:52.367 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-11-28T17:25:52.367 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-11-28T17:25:52.367 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_security.ko 2023-11-28T17:25:52.367 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-11-28T17:25:52.367 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-11-28T17:25:52.368 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-11-28T17:25:52.368 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-11-28T17:25:52.368 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-11-28T17:25:52.368 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-11-28T17:25:52.368 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-11-28T17:25:52.368 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-11-28T17:25:52.368 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-11-28T17:25:52.368 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-11-28T17:25:52.368 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_bic.ko 2023-11-28T17:25:52.368 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_dctcp.ko 2023-11-28T17:25:52.368 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_diag.ko 2023-11-28T17:25:52.368 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_highspeed.ko 2023-11-28T17:25:52.368 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_htcp.ko 2023-11-28T17:25:52.369 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_hybla.ko 2023-11-28T17:25:52.369 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_illinois.ko 2023-11-28T17:25:52.369 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_lp.ko 2023-11-28T17:25:52.369 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_scalable.ko 2023-11-28T17:25:52.369 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_vegas.ko 2023-11-28T17:25:52.369 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_veno.ko 2023-11-28T17:25:52.369 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_westwood.ko 2023-11-28T17:25:52.369 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_yeah.ko 2023-11-28T17:25:52.369 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tunnel4.ko 2023-11-28T17:25:52.369 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_diag.ko 2023-11-28T17:25:52.369 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_tunnel.ko 2023-11-28T17:25:52.369 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/xfrm4_tunnel.ko 2023-11-28T17:25:52.370 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6 2023-11-28T17:25:52.370 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ah6.ko 2023-11-28T17:25:52.370 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/esp6.ko 2023-11-28T17:25:52.370 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/fou6.ko 2023-11-28T17:25:52.370 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_tunnel.ko 2023-11-28T17:25:52.370 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-11-28T17:25:52.370 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_vti.ko 2023-11-28T17:25:52.370 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ipcomp6.ko 2023-11-28T17:25:52.370 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter 2023-11-28T17:25:52.370 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-11-28T17:25:52.370 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-11-28T17:25:52.370 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-11-28T17:25:52.371 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-11-28T17:25:52.371 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-11-28T17:25:52.371 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-11-28T17:25:52.371 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-11-28T17:25:52.371 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-11-28T17:25:52.371 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-11-28T17:25:52.371 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-11-28T17:25:52.371 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-11-28T17:25:52.371 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-11-28T17:25:52.371 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-11-28T17:25:52.371 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-11-28T17:25:52.372 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-11-28T17:25:52.372 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-11-28T17:25:52.372 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-11-28T17:25:52.372 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-11-28T17:25:52.372 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-11-28T17:25:52.372 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-11-28T17:25:52.372 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-11-28T17:25:52.372 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-11-28T17:25:52.372 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-11-28T17:25:52.372 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-11-28T17:25:52.372 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/sit.ko 2023-11-28T17:25:52.372 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/tunnel6.ko 2023-11-28T17:25:52.373 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/xfrm6_tunnel.ko 2023-11-28T17:25:52.373 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key 2023-11-28T17:25:52.373 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key/af_key.ko 2023-11-28T17:25:52.373 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp 2023-11-28T17:25:52.373 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_core.ko 2023-11-28T17:25:52.373 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_debugfs.ko 2023-11-28T17:25:52.373 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_eth.ko 2023-11-28T17:25:52.373 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip.ko 2023-11-28T17:25:52.373 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip6.ko 2023-11-28T17:25:52.373 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_netlink.ko 2023-11-28T17:25:52.373 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ppp.ko 2023-11-28T17:25:52.373 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc 2023-11-28T17:25:52.374 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc/llc.ko 2023-11-28T17:25:52.374 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154 2023-11-28T17:25:52.374 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154/mac802154.ko 2023-11-28T17:25:52.374 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls 2023-11-28T17:25:52.374 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls/mpls_gso.ko 2023-11-28T17:25:52.374 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter 2023-11-28T17:25:52.374 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset 2023-11-28T17:25:52.374 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set.ko 2023-11-28T17:25:52.374 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-11-28T17:25:52.374 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-11-28T17:25:52.374 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-11-28T17:25:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-11-28T17:25:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-11-28T17:25:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-11-28T17:25:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-11-28T17:25:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-11-28T17:25:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-11-28T17:25:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-11-28T17:25:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-11-28T17:25:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-11-28T17:25:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-11-28T17:25:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-11-28T17:25:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-11-28T17:25:52.375 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs 2023-11-28T17:25:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs.ko 2023-11-28T17:25:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-11-28T17:25:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-11-28T17:25:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-11-28T17:25:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-11-28T17:25:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-11-28T17:25:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-11-28T17:25:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-11-28T17:25:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-11-28T17:25:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-11-28T17:25:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-11-28T17:25:52.376 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-11-28T17:25:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-11-28T17:25:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-11-28T17:25:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conncount.ko 2023-11-28T17:25:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack.ko 2023-11-28T17:25:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-11-28T17:25:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-11-28T17:25:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-11-28T17:25:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_h323.ko 2023-11-28T17:25:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_irc.ko 2023-11-28T17:25:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-11-28T17:25:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-11-28T17:25:52.377 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-11-28T17:25:52.378 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sane.ko 2023-11-28T17:25:52.378 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sip.ko 2023-11-28T17:25:52.378 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-11-28T17:25:52.378 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-11-28T17:25:52.378 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_dup_netdev.ko 2023-11-28T17:25:52.378 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table.ko 2023-11-28T17:25:52.378 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table_inet.ko 2023-11-28T17:25:52.378 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_log_syslog.ko 2023-11-28T17:25:52.378 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat.ko 2023-11-28T17:25:52.378 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_amanda.ko 2023-11-28T17:25:52.378 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_ftp.ko 2023-11-28T17:25:52.378 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_irc.ko 2023-11-28T17:25:52.379 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_sip.ko 2023-11-28T17:25:52.379 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_tftp.ko 2023-11-28T17:25:52.379 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_synproxy_core.ko 2023-11-28T17:25:52.379 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_tables.ko 2023-11-28T17:25:52.379 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink.ko 2023-11-28T17:25:52.379 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_acct.ko 2023-11-28T17:25:52.379 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_log.ko 2023-11-28T17:25:52.379 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_osf.ko 2023-11-28T17:25:52.379 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_queue.ko 2023-11-28T17:25:52.379 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_chain_nat.ko 2023-11-28T17:25:52.379 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_compat.ko 2023-11-28T17:25:52.379 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_ct.ko 2023-11-28T17:25:52.379 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_dup_netdev.ko 2023-11-28T17:25:52.380 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fib.ko 2023-11-28T17:25:52.380 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fwd_netdev.ko 2023-11-28T17:25:52.380 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_hash.ko 2023-11-28T17:25:52.380 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_limit.ko 2023-11-28T17:25:52.380 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_log.ko 2023-11-28T17:25:52.380 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_masq.ko 2023-11-28T17:25:52.380 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_nat.ko 2023-11-28T17:25:52.380 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_numgen.ko 2023-11-28T17:25:52.380 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_queue.ko 2023-11-28T17:25:52.380 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_quota.ko 2023-11-28T17:25:52.380 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_redir.ko 2023-11-28T17:25:52.380 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject.ko 2023-11-28T17:25:52.380 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject_inet.ko 2023-11-28T17:25:52.381 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_socket.ko 2023-11-28T17:25:52.381 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_tunnel.ko 2023-11-28T17:25:52.381 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_AUDIT.ko 2023-11-28T17:25:52.381 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CHECKSUM.ko 2023-11-28T17:25:52.381 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CLASSIFY.ko 2023-11-28T17:25:52.381 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-11-28T17:25:52.381 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CT.ko 2023-11-28T17:25:52.381 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_DSCP.ko 2023-11-28T17:25:52.381 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HL.ko 2023-11-28T17:25:52.381 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HMARK.ko 2023-11-28T17:25:52.381 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_IDLETIMER.ko 2023-11-28T17:25:52.381 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LED.ko 2023-11-28T17:25:52.381 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LOG.ko 2023-11-28T17:25:52.382 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_MASQUERADE.ko 2023-11-28T17:25:52.382 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NETMAP.ko 2023-11-28T17:25:52.382 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFLOG.ko 2023-11-28T17:25:52.382 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFQUEUE.ko 2023-11-28T17:25:52.382 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_RATEEST.ko 2023-11-28T17:25:52.382 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_REDIRECT.ko 2023-11-28T17:25:52.382 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_SECMARK.ko 2023-11-28T17:25:52.382 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPMSS.ko 2023-11-28T17:25:52.382 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-11-28T17:25:52.382 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TEE.ko 2023-11-28T17:25:52.382 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TPROXY.ko 2023-11-28T17:25:52.382 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TRACE.ko 2023-11-28T17:25:52.383 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_addrtype.ko 2023-11-28T17:25:52.383 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_bpf.ko 2023-11-28T17:25:52.383 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cgroup.ko 2023-11-28T17:25:52.383 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cluster.ko 2023-11-28T17:25:52.383 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_comment.ko 2023-11-28T17:25:52.383 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connbytes.ko 2023-11-28T17:25:52.383 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlabel.ko 2023-11-28T17:25:52.383 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlimit.ko 2023-11-28T17:25:52.383 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connmark.ko 2023-11-28T17:25:52.383 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_conntrack.ko 2023-11-28T17:25:52.383 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cpu.ko 2023-11-28T17:25:52.383 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dccp.ko 2023-11-28T17:25:52.384 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_devgroup.ko 2023-11-28T17:25:52.384 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dscp.ko 2023-11-28T17:25:52.384 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ecn.ko 2023-11-28T17:25:52.384 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_esp.ko 2023-11-28T17:25:52.384 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hashlimit.ko 2023-11-28T17:25:52.384 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_helper.ko 2023-11-28T17:25:52.384 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hl.ko 2023-11-28T17:25:52.384 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipcomp.ko 2023-11-28T17:25:52.384 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_iprange.ko 2023-11-28T17:25:52.384 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipvs.ko 2023-11-28T17:25:52.384 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_l2tp.ko 2023-11-28T17:25:52.384 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_length.ko 2023-11-28T17:25:52.385 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_limit.ko 2023-11-28T17:25:52.385 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mac.ko 2023-11-28T17:25:52.385 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mark.ko 2023-11-28T17:25:52.385 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_multiport.ko 2023-11-28T17:25:52.385 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nat.ko 2023-11-28T17:25:52.385 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nfacct.ko 2023-11-28T17:25:52.385 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_osf.ko 2023-11-28T17:25:52.385 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_owner.ko 2023-11-28T17:25:52.385 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_physdev.ko 2023-11-28T17:25:52.385 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_pkttype.ko 2023-11-28T17:25:52.385 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_policy.ko 2023-11-28T17:25:52.385 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_quota.ko 2023-11-28T17:25:52.385 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_rateest.ko 2023-11-28T17:25:52.386 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_realm.ko 2023-11-28T17:25:52.386 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_recent.ko 2023-11-28T17:25:52.386 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_sctp.ko 2023-11-28T17:25:52.386 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_set.ko 2023-11-28T17:25:52.386 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_socket.ko 2023-11-28T17:25:52.386 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_state.ko 2023-11-28T17:25:52.386 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_statistic.ko 2023-11-28T17:25:52.386 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_string.ko 2023-11-28T17:25:52.386 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_tcpmss.ko 2023-11-28T17:25:52.386 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_time.ko 2023-11-28T17:25:52.387 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_u32.ko 2023-11-28T17:25:52.387 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink 2023-11-28T17:25:52.387 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink/netlink_diag.ko 2023-11-28T17:25:52.387 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh 2023-11-28T17:25:52.387 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh/nsh.ko 2023-11-28T17:25:52.387 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch 2023-11-28T17:25:52.387 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/openvswitch.ko 2023-11-28T17:25:52.387 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-geneve.ko 2023-11-28T17:25:52.387 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-gre.ko 2023-11-28T17:25:52.387 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-vxlan.ko 2023-11-28T17:25:52.387 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet 2023-11-28T17:25:52.387 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet/af_packet_diag.ko 2023-11-28T17:25:52.388 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds 2023-11-28T17:25:52.388 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds.ko 2023-11-28T17:25:52.388 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_rdma.ko 2023-11-28T17:25:52.388 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_tcp.ko 2023-11-28T17:25:52.388 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched 2023-11-28T17:25:52.388 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_csum.ko 2023-11-28T17:25:52.388 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_gact.ko 2023-11-28T17:25:52.388 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_ipt.ko 2023-11-28T17:25:52.388 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_mirred.ko 2023-11-28T17:25:52.388 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_nat.ko 2023-11-28T17:25:52.388 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_pedit.ko 2023-11-28T17:25:52.388 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_police.ko 2023-11-28T17:25:52.388 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_simple.ko 2023-11-28T17:25:52.389 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_skbedit.ko 2023-11-28T17:25:52.389 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_vlan.ko 2023-11-28T17:25:52.389 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_basic.ko 2023-11-28T17:25:52.389 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_bpf.ko 2023-11-28T17:25:52.389 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_flow.ko 2023-11-28T17:25:52.389 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_fw.ko 2023-11-28T17:25:52.389 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_route.ko 2023-11-28T17:25:52.389 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_u32.ko 2023-11-28T17:25:52.389 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_cmp.ko 2023-11-28T17:25:52.389 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_ipset.ko 2023-11-28T17:25:52.389 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_meta.ko 2023-11-28T17:25:52.389 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_nbyte.ko 2023-11-28T17:25:52.390 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_text.ko 2023-11-28T17:25:52.390 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_u32.ko 2023-11-28T17:25:52.390 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_choke.ko 2023-11-28T17:25:52.390 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_codel.ko 2023-11-28T17:25:52.390 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_drr.ko 2023-11-28T17:25:52.390 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_fq.ko 2023-11-28T17:25:52.390 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_gred.ko 2023-11-28T17:25:52.390 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hfsc.ko 2023-11-28T17:25:52.390 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hhf.ko 2023-11-28T17:25:52.390 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_htb.ko 2023-11-28T17:25:52.390 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_ingress.ko 2023-11-28T17:25:52.390 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio.ko 2023-11-28T17:25:52.391 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio_lib.ko 2023-11-28T17:25:52.391 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_multiq.ko 2023-11-28T17:25:52.391 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_netem.ko 2023-11-28T17:25:52.391 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_pie.ko 2023-11-28T17:25:52.391 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_plug.ko 2023-11-28T17:25:52.391 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_prio.ko 2023-11-28T17:25:52.391 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_qfq.ko 2023-11-28T17:25:52.391 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_red.ko 2023-11-28T17:25:52.391 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfb.ko 2023-11-28T17:25:52.391 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfq.ko 2023-11-28T17:25:52.391 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_tbf.ko 2023-11-28T17:25:52.391 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_teql.ko 2023-11-28T17:25:52.391 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp 2023-11-28T17:25:52.392 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp.ko 2023-11-28T17:25:52.392 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp_diag.ko 2023-11-28T17:25:52.392 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc 2023-11-28T17:25:52.392 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss 2023-11-28T17:25:52.392 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-11-28T17:25:52.392 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-11-28T17:25:52.392 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/sunrpc.ko 2023-11-28T17:25:52.392 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma 2023-11-28T17:25:52.392 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-11-28T17:25:52.392 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc 2023-11-28T17:25:52.392 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/diag.ko 2023-11-28T17:25:52.392 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/tipc.ko 2023-11-28T17:25:52.393 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix 2023-11-28T17:25:52.393 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix/unix_diag.ko 2023-11-28T17:25:52.393 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock 2023-11-28T17:25:52.393 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-11-28T17:25:52.393 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-11-28T17:25:52.393 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock.ko 2023-11-28T17:25:52.393 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_diag.ko 2023-11-28T17:25:52.393 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_loopback.ko 2023-11-28T17:25:52.393 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm 2023-11-28T17:25:52.393 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/xfrm_ipcomp.ko 2023-11-28T17:25:52.393 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security 2023-11-28T17:25:52.393 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys 2023-11-28T17:25:52.394 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys 2023-11-28T17:25:52.394 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-11-28T17:25:52.394 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys 2023-11-28T17:25:52.394 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys/trusted.ko 2023-11-28T17:25:52.394 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt 2023-11-28T17:25:52.394 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib 2023-11-28T17:25:52.394 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib/irqbypass.ko 2023-11-28T17:25:52.394 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias 2023-11-28T17:25:52.394 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias.bin 2023-11-28T17:25:52.394 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin 2023-11-28T17:25:52.394 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.alias.bin 2023-11-28T17:25:52.394 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.bin 2023-11-28T17:25:52.394 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.modinfo 2023-11-28T17:25:52.395 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep 2023-11-28T17:25:52.395 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep.bin 2023-11-28T17:25:52.395 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.devname 2023-11-28T17:25:52.395 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.order 2023-11-28T17:25:52.395 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.softdep 2023-11-28T17:25:52.395 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols 2023-11-28T17:25:52.395 INFO:teuthology.orchestra.run.smithi113.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols.bin 2023-11-28T17:25:52.396 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-g8247bbb5b470 2023-11-28T17:25:52.396 DEBUG:teuthology.orchestra.run.smithi113:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-11-28T17:25:52.418 INFO:teuthology.orchestra.run.smithi113.stdout:postinstall scriptlet (using /bin/sh): 2023-11-28T17:25:52.418 INFO:teuthology.orchestra.run.smithi113.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-g8247bbb5b470 -a -r /boot/System.map-6.6.0-g8247bbb5b470 ]; then 2023-11-28T17:25:52.418 INFO:teuthology.orchestra.run.smithi113.stdout:cp /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm 2023-11-28T17:25:52.418 INFO:teuthology.orchestra.run.smithi113.stdout:cp /boot/System.map-6.6.0-g8247bbb5b470 /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T17:25:52.418 INFO:teuthology.orchestra.run.smithi113.stdout:rm -f /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/System.map-6.6.0-g8247bbb5b470 2023-11-28T17:25:52.418 INFO:teuthology.orchestra.run.smithi113.stdout:/sbin/installkernel 6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T17:25:52.418 INFO:teuthology.orchestra.run.smithi113.stdout:rm -f /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T17:25:52.419 INFO:teuthology.orchestra.run.smithi113.stdout:fi 2023-11-28T17:25:52.419 INFO:teuthology.orchestra.run.smithi113.stdout:preuninstall scriptlet (using /bin/sh): 2023-11-28T17:25:52.419 INFO:teuthology.orchestra.run.smithi113.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-11-28T17:25:52.419 INFO:teuthology.orchestra.run.smithi113.stdout:new-kernel-pkg --remove 6.6.0-g8247bbb5b470 --rminitrd --initrdfile=/boot/initramfs-6.6.0-g8247bbb5b470.img 2023-11-28T17:25:52.419 INFO:teuthology.orchestra.run.smithi113.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-11-28T17:25:52.419 INFO:teuthology.orchestra.run.smithi113.stdout:kernel-install remove 6.6.0-g8247bbb5b470 2023-11-28T17:25:52.419 INFO:teuthology.orchestra.run.smithi113.stdout:fi 2023-11-28T17:25:52.419 INFO:teuthology.orchestra.run.smithi113.stdout:postuninstall scriptlet (using /bin/sh): 2023-11-28T17:25:52.419 INFO:teuthology.orchestra.run.smithi113.stdout:if [ -x /sbin/update-bootloader ]; then 2023-11-28T17:25:52.419 INFO:teuthology.orchestra.run.smithi113.stdout:/sbin/update-bootloader --remove 6.6.0-g8247bbb5b470 2023-11-28T17:25:52.419 INFO:teuthology.orchestra.run.smithi113.stdout:fi 2023-11-28T17:25:52.420 DEBUG:teuthology.orchestra.run.smithi113:> sudo rpm -qi grub2-tools 2023-11-28T17:25:52.493 INFO:teuthology.orchestra.run.smithi113.stdout:Name : grub2-tools 2023-11-28T17:25:52.494 INFO:teuthology.orchestra.run.smithi113.stdout:Epoch : 1 2023-11-28T17:25:52.494 INFO:teuthology.orchestra.run.smithi113.stdout:Version : 2.06 2023-11-28T17:25:52.494 INFO:teuthology.orchestra.run.smithi113.stdout:Release : 68.el9 2023-11-28T17:25:52.494 INFO:teuthology.orchestra.run.smithi113.stdout:Architecture: x86_64 2023-11-28T17:25:52.494 INFO:teuthology.orchestra.run.smithi113.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-28T17:25:52.494 INFO:teuthology.orchestra.run.smithi113.stdout:Group : Unspecified 2023-11-28T17:25:52.494 INFO:teuthology.orchestra.run.smithi113.stdout:Size : 8264254 2023-11-28T17:25:52.494 INFO:teuthology.orchestra.run.smithi113.stdout:License : GPLv3+ 2023-11-28T17:25:52.494 INFO:teuthology.orchestra.run.smithi113.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-28T17:25:52.494 INFO:teuthology.orchestra.run.smithi113.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-28T17:25:52.495 INFO:teuthology.orchestra.run.smithi113.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-28T17:25:52.495 INFO:teuthology.orchestra.run.smithi113.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-28T17:25:52.495 INFO:teuthology.orchestra.run.smithi113.stdout:Packager : builder@centos.org 2023-11-28T17:25:52.495 INFO:teuthology.orchestra.run.smithi113.stdout:Vendor : CentOS 2023-11-28T17:25:52.495 INFO:teuthology.orchestra.run.smithi113.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-28T17:25:52.495 INFO:teuthology.orchestra.run.smithi113.stdout:Summary : Support tools for GRUB. 2023-11-28T17:25:52.495 INFO:teuthology.orchestra.run.smithi113.stdout:Description : 2023-11-28T17:25:52.495 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:25:52.495 INFO:teuthology.orchestra.run.smithi113.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-28T17:25:52.495 INFO:teuthology.orchestra.run.smithi113.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-28T17:25:52.495 INFO:teuthology.orchestra.run.smithi113.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-28T17:25:52.495 INFO:teuthology.orchestra.run.smithi113.stdout:hardware devices. 2023-11-28T17:25:52.495 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:25:52.496 INFO:teuthology.orchestra.run.smithi113.stdout:This subpackage provides tools for support of all platforms. 2023-11-28T17:25:52.496 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-28T17:25:52.496 INFO:teuthology.task.kernel:Updating grub on smithi113 to boot 6.6.0-g8247bbb5b470 2023-11-28T17:25:52.497 DEBUG:teuthology.orchestra.run.smithi113:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-28T17:25:53.107 INFO:teuthology.orchestra.run.smithi113.stderr:Generating grub configuration file ... 2023-11-28T17:25:54.475 INFO:teuthology.orchestra.run.smithi113.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-28T17:25:54.530 INFO:teuthology.orchestra.run.smithi113.stderr:done 2023-11-28T17:25:54.533 DEBUG:teuthology.orchestra.run.smithi113:> mktemp 2023-11-28T17:25:54.547 INFO:teuthology.orchestra.run.smithi113.stdout:/tmp/tmp.EHjEYmzAz6 2023-11-28T17:25:54.547 DEBUG:teuthology.orchestra.run.smithi113:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EHjEYmzAz6 2023-11-28T17:25:54.623 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0666 /tmp/tmp.EHjEYmzAz6 2023-11-28T17:25:54.765 DEBUG:teuthology.orchestra.remote:smithi113:/tmp/tmp.EHjEYmzAz6 is 6KB 2023-11-28T17:25:54.785 DEBUG:teuthology.orchestra.run.smithi113:> rm -fr /tmp/tmp.EHjEYmzAz6 2023-11-28T17:25:54.799 DEBUG:teuthology.orchestra.run.smithi113:> sudo /bin/ls /boot/loader/entries || true 2023-11-28T17:25:54.863 INFO:teuthology.orchestra.run.smithi113.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-28T17:25:54.864 INFO:teuthology.orchestra.run.smithi113.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-28T17:25:54.864 INFO:teuthology.orchestra.run.smithi113.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-28T17:25:54.864 INFO:teuthology.orchestra.run.smithi113.stdout:15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470.conf 2023-11-28T17:25:54.866 DEBUG:teuthology.orchestra.run.smithi113:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470 2023-11-28T17:25:55.032 INFO:teuthology.orchestra.run.smithi098.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-11-28T17:25:55.036 DEBUG:teuthology.orchestra.run.smithi113:> sudo shutdown -r now 2023-11-28T17:26:16.191 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-11-28T17:26:16.192 DEBUG:teuthology.orchestra.run.smithi098:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-11-28T17:26:16.257 INFO:teuthology.orchestra.run.smithi098.stdout:/boot/System.map-6.6.0-g8247bbb5b470 2023-11-28T17:26:16.257 INFO:teuthology.orchestra.run.smithi098.stdout:/boot/config-6.6.0-g8247bbb5b470 2023-11-28T17:26:16.257 INFO:teuthology.orchestra.run.smithi098.stdout:/boot/vmlinuz-6.6.0-g8247bbb5b470 2023-11-28T17:26:16.257 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470 2023-11-28T17:26:16.257 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel 2023-11-28T17:26:16.257 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch 2023-11-28T17:26:16.257 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86 2023-11-28T17:26:16.257 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto 2023-11-28T17:26:16.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-11-28T17:26:16.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-11-28T17:26:16.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-11-28T17:26:16.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-11-28T17:26:16.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-11-28T17:26:16.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-11-28T17:26:16.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-11-28T17:26:16.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32c-intel.ko 2023-11-28T17:26:16.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-11-28T17:26:16.258 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-11-28T17:26:16.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-11-28T17:26:16.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-11-28T17:26:16.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx2.ko 2023-11-28T17:26:16.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-11-28T17:26:16.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-11-28T17:26:16.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-11-28T17:26:16.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-11-28T17:26:16.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-11-28T17:26:16.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-11-28T17:26:16.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-11-28T17:26:16.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel 2023-11-28T17:26:16.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu 2023-11-28T17:26:16.259 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce 2023-11-28T17:26:16.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-11-28T17:26:16.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm 2023-11-28T17:26:16.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-amd.ko 2023-11-28T17:26:16.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-intel.ko 2023-11-28T17:26:16.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm.ko 2023-11-28T17:26:16.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto 2023-11-28T17:26:16.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ansi_cprng.ko 2023-11-28T17:26:16.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/anubis.ko 2023-11-28T17:26:16.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/arc4.ko 2023-11-28T17:26:16.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx 2023-11-28T17:26:16.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_memcpy.ko 2023-11-28T17:26:16.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_pq.ko 2023-11-28T17:26:16.260 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_raid6_recov.ko 2023-11-28T17:26:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_tx.ko 2023-11-28T17:26:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_xor.ko 2023-11-28T17:26:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/raid6test.ko 2023-11-28T17:26:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authenc.ko 2023-11-28T17:26:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authencesn.ko 2023-11-28T17:26:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blake2b_generic.ko 2023-11-28T17:26:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_common.ko 2023-11-28T17:26:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_generic.ko 2023-11-28T17:26:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/camellia_generic.ko 2023-11-28T17:26:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast5_generic.ko 2023-11-28T17:26:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast6_generic.ko 2023-11-28T17:26:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast_common.ko 2023-11-28T17:26:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ccm.ko 2023-11-28T17:26:16.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cmac.ko 2023-11-28T17:26:16.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crc32_generic.ko 2023-11-28T17:26:16.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_engine.ko 2023-11-28T17:26:16.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_user.ko 2023-11-28T17:26:16.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cts.ko 2023-11-28T17:26:16.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/des_generic.ko 2023-11-28T17:26:16.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/dh_generic.ko 2023-11-28T17:26:16.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/echainiv.ko 2023-11-28T17:26:16.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/essiv.ko 2023-11-28T17:26:16.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/fcrypt.ko 2023-11-28T17:26:16.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/khazad.ko 2023-11-28T17:26:16.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4.ko 2023-11-28T17:26:16.262 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4hc.ko 2023-11-28T17:26:16.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/md4.ko 2023-11-28T17:26:16.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/michael_mic.ko 2023-11-28T17:26:16.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcbc.ko 2023-11-28T17:26:16.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcrypt.ko 2023-11-28T17:26:16.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/rmd160.ko 2023-11-28T17:26:16.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/seed.ko 2023-11-28T17:26:16.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/serpent_generic.ko 2023-11-28T17:26:16.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tcrypt.ko 2023-11-28T17:26:16.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tea.ko 2023-11-28T17:26:16.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_common.ko 2023-11-28T17:26:16.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_generic.ko 2023-11-28T17:26:16.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/vmac.ko 2023-11-28T17:26:16.263 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/wp512.ko 2023-11-28T17:26:16.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xcbc.ko 2023-11-28T17:26:16.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xor.ko 2023-11-28T17:26:16.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xxhash_generic.ko 2023-11-28T17:26:16.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers 2023-11-28T17:26:16.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi 2023-11-28T17:26:16.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_ipmi.ko 2023-11-28T17:26:16.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_pad.ko 2023-11-28T17:26:16.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/custom_method.ko 2023-11-28T17:26:16.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/ec_sys.ko 2023-11-28T17:26:16.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/platform_profile.ko 2023-11-28T17:26:16.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbs.ko 2023-11-28T17:26:16.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbshc.ko 2023-11-28T17:26:16.264 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/video.ko 2023-11-28T17:26:16.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata 2023-11-28T17:26:16.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/acard-ahci.ko 2023-11-28T17:26:16.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ahci_platform.ko 2023-11-28T17:26:16.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ata_generic.ko 2023-11-28T17:26:16.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/libahci_platform.ko 2023-11-28T17:26:16.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_acpi.ko 2023-11-28T17:26:16.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ali.ko 2023-11-28T17:26:16.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_amd.ko 2023-11-28T17:26:16.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_artop.ko 2023-11-28T17:26:16.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atiixp.ko 2023-11-28T17:26:16.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atp867x.ko 2023-11-28T17:26:16.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd640.ko 2023-11-28T17:26:16.265 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd64x.ko 2023-11-28T17:26:16.266 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cypress.ko 2023-11-28T17:26:16.266 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_efar.ko 2023-11-28T17:26:16.266 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt366.ko 2023-11-28T17:26:16.266 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt37x.ko 2023-11-28T17:26:16.266 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x2n.ko 2023-11-28T17:26:16.266 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x3.ko 2023-11-28T17:26:16.266 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it8213.ko 2023-11-28T17:26:16.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it821x.ko 2023-11-28T17:26:16.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_jmicron.ko 2023-11-28T17:26:16.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_marvell.ko 2023-11-28T17:26:16.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_mpiix.ko 2023-11-28T17:26:16.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_netcell.ko 2023-11-28T17:26:16.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ninja32.ko 2023-11-28T17:26:16.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87410.ko 2023-11-28T17:26:16.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87415.ko 2023-11-28T17:26:16.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_oldpiix.ko 2023-11-28T17:26:16.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_opti.ko 2023-11-28T17:26:16.267 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_optidma.ko 2023-11-28T17:26:16.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pcmcia.ko 2023-11-28T17:26:16.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc2027x.ko 2023-11-28T17:26:16.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-11-28T17:26:16.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_piccolo.ko 2023-11-28T17:26:16.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_rdc.ko 2023-11-28T17:26:16.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sch.ko 2023-11-28T17:26:16.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_serverworks.ko 2023-11-28T17:26:16.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sil680.ko 2023-11-28T17:26:16.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sis.ko 2023-11-28T17:26:16.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sl82c105.ko 2023-11-28T17:26:16.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_triflex.ko 2023-11-28T17:26:16.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_via.ko 2023-11-28T17:26:16.268 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pdc_adma.ko 2023-11-28T17:26:16.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_inic162x.ko 2023-11-28T17:26:16.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_mv.ko 2023-11-28T17:26:16.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_nv.ko 2023-11-28T17:26:16.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_promise.ko 2023-11-28T17:26:16.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_qstor.ko 2023-11-28T17:26:16.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil.ko 2023-11-28T17:26:16.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil24.ko 2023-11-28T17:26:16.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sis.ko 2023-11-28T17:26:16.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_svw.ko 2023-11-28T17:26:16.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sx4.ko 2023-11-28T17:26:16.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_uli.ko 2023-11-28T17:26:16.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_via.ko 2023-11-28T17:26:16.269 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_vsc.ko 2023-11-28T17:26:16.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay 2023-11-28T17:26:16.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay/ks0108.ko 2023-11-28T17:26:16.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base 2023-11-28T17:26:16.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap 2023-11-28T17:26:16.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-i2c.ko 2023-11-28T17:26:16.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-mmio.ko 2023-11-28T17:26:16.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma 2023-11-28T17:26:16.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma/bcma.ko 2023-11-28T17:26:16.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block 2023-11-28T17:26:16.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe 2023-11-28T17:26:16.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe/aoe.ko 2023-11-28T17:26:16.270 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/brd.ko 2023-11-28T17:26:16.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd 2023-11-28T17:26:16.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd/drbd.ko 2023-11-28T17:26:16.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/floppy.ko 2023-11-28T17:26:16.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/loop.ko 2023-11-28T17:26:16.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx 2023-11-28T17:26:16.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-11-28T17:26:16.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/nbd.ko 2023-11-28T17:26:16.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk 2023-11-28T17:26:16.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk/null_blk.ko 2023-11-28T17:26:16.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/pktcdvd.ko 2023-11-28T17:26:16.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/rbd.ko 2023-11-28T17:26:16.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/virtio_blk.ko 2023-11-28T17:26:16.271 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback 2023-11-28T17:26:16.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-11-28T17:26:16.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkfront.ko 2023-11-28T17:26:16.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram 2023-11-28T17:26:16.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram/zram.ko 2023-11-28T17:26:16.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char 2023-11-28T17:26:16.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hangcheck-timer.ko 2023-11-28T17:26:16.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random 2023-11-28T17:26:16.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/amd-rng.ko 2023-11-28T17:26:16.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/intel-rng.ko 2023-11-28T17:26:16.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-11-28T17:26:16.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/via-rng.ko 2023-11-28T17:26:16.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/virtio-rng.ko 2023-11-28T17:26:16.272 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi 2023-11-28T17:26:16.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-11-28T17:26:16.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-11-28T17:26:16.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-11-28T17:26:16.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_si.ko 2023-11-28T17:26:16.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-11-28T17:26:16.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-11-28T17:26:16.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/lp.ko 2023-11-28T17:26:16.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave 2023-11-28T17:26:16.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave/mwave.ko 2023-11-28T17:26:16.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ppdev.ko 2023-11-28T17:26:16.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tlclk.ko 2023-11-28T17:26:16.273 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm 2023-11-28T17:26:16.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm.ko 2023-11-28T17:26:16.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_atmel.ko 2023-11-28T17:26:16.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_infineon.ko 2023-11-28T17:26:16.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_nsc.ko 2023-11-28T17:26:16.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis.ko 2023-11-28T17:26:16.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-11-28T17:26:16.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/uv_mmtimer.ko 2023-11-28T17:26:16.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/virtio_console.ko 2023-11-28T17:26:16.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq 2023-11-28T17:26:16.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-11-28T17:26:16.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-11-28T17:26:16.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/p4-clockmod.ko 2023-11-28T17:26:16.274 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-11-28T17:26:16.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/powernow-k8.ko 2023-11-28T17:26:16.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/speedstep-lib.ko 2023-11-28T17:26:16.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto 2023-11-28T17:26:16.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp 2023-11-28T17:26:16.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-11-28T17:26:16.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp.ko 2023-11-28T17:26:16.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel 2023-11-28T17:26:16.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat 2023-11-28T17:26:16.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common 2023-11-28T17:26:16.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-11-28T17:26:16.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-11-28T17:26:16.275 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-11-28T17:26:16.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-aes.ko 2023-11-28T17:26:16.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-sha.ko 2023-11-28T17:26:16.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio 2023-11-28T17:26:16.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-11-28T17:26:16.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca 2023-11-28T17:26:16.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca/dca.ko 2023-11-28T17:26:16.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq 2023-11-28T17:26:16.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-11-28T17:26:16.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma 2023-11-28T17:26:16.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw 2023-11-28T17:26:16.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw/dw_dmac.ko 2023-11-28T17:26:16.276 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat 2023-11-28T17:26:16.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat/ioatdma.ko 2023-11-28T17:26:16.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac 2023-11-28T17:26:16.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/amd64_edac.ko 2023-11-28T17:26:16.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/e752x_edac.ko 2023-11-28T17:26:16.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/edac_mce_amd.ko 2023-11-28T17:26:16.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3000_edac.ko 2023-11-28T17:26:16.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3200_edac.ko 2023-11-28T17:26:16.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5100_edac.ko 2023-11-28T17:26:16.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5400_edac.ko 2023-11-28T17:26:16.277 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7300_edac.ko 2023-11-28T17:26:16.278 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7core_edac.ko 2023-11-28T17:26:16.278 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i82975x_edac.ko 2023-11-28T17:26:16.278 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/ie31200_edac.ko 2023-11-28T17:26:16.278 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/sb_edac.ko 2023-11-28T17:26:16.278 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/x38_edac.ko 2023-11-28T17:26:16.278 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware 2023-11-28T17:26:16.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/edd.ko 2023-11-28T17:26:16.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/iscsi_ibft.ko 2023-11-28T17:26:16.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio 2023-11-28T17:26:16.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-ich.ko 2023-11-28T17:26:16.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-viperboard.ko 2023-11-28T17:26:16.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid 2023-11-28T17:26:16.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-appleir.ko 2023-11-28T17:26:16.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-aureal.ko 2023-11-28T17:26:16.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-axff.ko 2023-11-28T17:26:16.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-dr.ko 2023-11-28T17:26:16.279 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elecom.ko 2023-11-28T17:26:16.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elo.ko 2023-11-28T17:26:16.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-emsff.ko 2023-11-28T17:26:16.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gaff.ko 2023-11-28T17:26:16.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gt683r.ko 2023-11-28T17:26:16.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gyration.ko 2023-11-28T17:26:16.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-kbd.ko 2023-11-28T17:26:16.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-mouse.ko 2023-11-28T17:26:16.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtekff.ko 2023-11-28T17:26:16.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-hyperv.ko 2023-11-28T17:26:16.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-icade.ko 2023-11-28T17:26:16.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-keytouch.ko 2023-11-28T17:26:16.280 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-kye.ko 2023-11-28T17:26:16.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lcpower.ko 2023-11-28T17:26:16.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-led.ko 2023-11-28T17:26:16.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lenovo.ko 2023-11-28T17:26:16.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-dj.ko 2023-11-28T17:26:16.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-11-28T17:26:16.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-multitouch.ko 2023-11-28T17:26:16.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-ortek.ko 2023-11-28T17:26:16.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-penmount.ko 2023-11-28T17:26:16.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-petalynx.ko 2023-11-28T17:26:16.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-picolcd.ko 2023-11-28T17:26:16.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-pl.ko 2023-11-28T17:26:16.281 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-plantronics.ko 2023-11-28T17:26:16.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-primax.ko 2023-11-28T17:26:16.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-rmi.ko 2023-11-28T17:26:16.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-arvo.ko 2023-11-28T17:26:16.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-common.ko 2023-11-28T17:26:16.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-isku.ko 2023-11-28T17:26:16.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kone.ko 2023-11-28T17:26:16.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-11-28T17:26:16.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-konepure.ko 2023-11-28T17:26:16.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-11-28T17:26:16.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-lua.ko 2023-11-28T17:26:16.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-pyra.ko 2023-11-28T17:26:16.282 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-ryos.ko 2023-11-28T17:26:16.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-savu.ko 2023-11-28T17:26:16.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat.ko 2023-11-28T17:26:16.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-saitek.ko 2023-11-28T17:26:16.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-samsung.ko 2023-11-28T17:26:16.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sensor-hub.ko 2023-11-28T17:26:16.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sjoy.ko 2023-11-28T17:26:16.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sony.ko 2023-11-28T17:26:16.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-speedlink.ko 2023-11-28T17:26:16.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-steelseries.ko 2023-11-28T17:26:16.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sunplus.ko 2023-11-28T17:26:16.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-thrustmaster.ko 2023-11-28T17:26:16.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tivo.ko 2023-11-28T17:26:16.283 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tmff.ko 2023-11-28T17:26:16.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-topseed.ko 2023-11-28T17:26:16.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-twinhan.ko 2023-11-28T17:26:16.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-uclogic.ko 2023-11-28T17:26:16.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-waltop.ko 2023-11-28T17:26:16.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-wiimote.ko 2023-11-28T17:26:16.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-xinmo.ko 2023-11-28T17:26:16.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zpff.ko 2023-11-28T17:26:16.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zydacron.ko 2023-11-28T17:26:16.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/uhid.ko 2023-11-28T17:26:16.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/wacom.ko 2023-11-28T17:26:16.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv 2023-11-28T17:26:16.284 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_balloon.ko 2023-11-28T17:26:16.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_utils.ko 2023-11-28T17:26:16.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_vmbus.ko 2023-11-28T17:26:16.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon 2023-11-28T17:26:16.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru.ko 2023-11-28T17:26:16.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru3.ko 2023-11-28T17:26:16.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/acpi_power_meter.ko 2023-11-28T17:26:16.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7414.ko 2023-11-28T17:26:16.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7418.ko 2023-11-28T17:26:16.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adc128d818.ko 2023-11-28T17:26:16.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1025.ko 2023-11-28T17:26:16.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1026.ko 2023-11-28T17:26:16.285 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1029.ko 2023-11-28T17:26:16.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1031.ko 2023-11-28T17:26:16.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm9240.ko 2023-11-28T17:26:16.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ads7828.ko 2023-11-28T17:26:16.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7410.ko 2023-11-28T17:26:16.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7411.ko 2023-11-28T17:26:16.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7462.ko 2023-11-28T17:26:16.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7470.ko 2023-11-28T17:26:16.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7475.ko 2023-11-28T17:26:16.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7x10.ko 2023-11-28T17:26:16.286 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/amc6821.ko 2023-11-28T17:26:16.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/applesmc.ko 2023-11-28T17:26:16.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asb100.ko 2023-11-28T17:26:16.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asc7621.ko 2023-11-28T17:26:16.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asus_atk0110.ko 2023-11-28T17:26:16.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/atxp1.ko 2023-11-28T17:26:16.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/coretemp.ko 2023-11-28T17:26:16.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-11-28T17:26:16.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dme1737.ko 2023-11-28T17:26:16.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds1621.ko 2023-11-28T17:26:16.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds620.ko 2023-11-28T17:26:16.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc1403.ko 2023-11-28T17:26:16.287 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc6w201.ko 2023-11-28T17:26:16.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71805f.ko 2023-11-28T17:26:16.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71882fg.ko 2023-11-28T17:26:16.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f75375s.ko 2023-11-28T17:26:16.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fam15h_power.ko 2023-11-28T17:26:16.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fschmd.ko 2023-11-28T17:26:16.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g760a.ko 2023-11-28T17:26:16.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g762.ko 2023-11-28T17:26:16.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl518sm.ko 2023-11-28T17:26:16.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl520sm.ko 2023-11-28T17:26:16.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/hwmon-vid.ko 2023-11-28T17:26:16.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5500_temp.ko 2023-11-28T17:26:16.288 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5k_amb.ko 2023-11-28T17:26:16.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmaem.ko 2023-11-28T17:26:16.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmpex.ko 2023-11-28T17:26:16.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina209.ko 2023-11-28T17:26:16.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina2xx.ko 2023-11-28T17:26:16.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/it87.ko 2023-11-28T17:26:16.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k10temp.ko 2023-11-28T17:26:16.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k8temp.ko 2023-11-28T17:26:16.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lineage-pem.ko 2023-11-28T17:26:16.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm63.ko 2023-11-28T17:26:16.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm73.ko 2023-11-28T17:26:16.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm75.ko 2023-11-28T17:26:16.289 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm77.ko 2023-11-28T17:26:16.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm78.ko 2023-11-28T17:26:16.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm80.ko 2023-11-28T17:26:16.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm83.ko 2023-11-28T17:26:16.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm85.ko 2023-11-28T17:26:16.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm87.ko 2023-11-28T17:26:16.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm90.ko 2023-11-28T17:26:16.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm92.ko 2023-11-28T17:26:16.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm93.ko 2023-11-28T17:26:16.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95234.ko 2023-11-28T17:26:16.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95241.ko 2023-11-28T17:26:16.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95245.ko 2023-11-28T17:26:16.290 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc2945.ko 2023-11-28T17:26:16.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4151.ko 2023-11-28T17:26:16.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4215.ko 2023-11-28T17:26:16.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4222.ko 2023-11-28T17:26:16.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4245.ko 2023-11-28T17:26:16.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4260.ko 2023-11-28T17:26:16.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4261.ko 2023-11-28T17:26:16.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max16065.ko 2023-11-28T17:26:16.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1619.ko 2023-11-28T17:26:16.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1668.ko 2023-11-28T17:26:16.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max197.ko 2023-11-28T17:26:16.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6639.ko 2023-11-28T17:26:16.291 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6650.ko 2023-11-28T17:26:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6697.ko 2023-11-28T17:26:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/mcp3021.ko 2023-11-28T17:26:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6683.ko 2023-11-28T17:26:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775-core.ko 2023-11-28T17:26:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775.ko 2023-11-28T17:26:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct7802.ko 2023-11-28T17:26:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87360.ko 2023-11-28T17:26:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87427.ko 2023-11-28T17:26:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pcf8591.ko 2023-11-28T17:26:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus 2023-11-28T17:26:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-11-28T17:26:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-11-28T17:26:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-11-28T17:26:16.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max16064.ko 2023-11-28T17:26:16.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max34440.ko 2023-11-28T17:26:16.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max8688.ko 2023-11-28T17:26:16.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-11-28T17:26:16.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-11-28T17:26:16.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-11-28T17:26:16.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-11-28T17:26:16.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-11-28T17:26:16.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-11-28T17:26:16.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/powr1220.ko 2023-11-28T17:26:16.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5627.ko 2023-11-28T17:26:16.293 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5636.ko 2023-11-28T17:26:16.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch56xx-common.ko 2023-11-28T17:26:16.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht15.ko 2023-11-28T17:26:16.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht21.ko 2023-11-28T17:26:16.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/shtc1.ko 2023-11-28T17:26:16.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sis5595.ko 2023-11-28T17:26:16.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47b397.ko 2023-11-28T17:26:16.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m1.ko 2023-11-28T17:26:16.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m192.ko 2023-11-28T17:26:16.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/thmc50.ko 2023-11-28T17:26:16.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp102.ko 2023-11-28T17:26:16.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp103.ko 2023-11-28T17:26:16.294 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp401.ko 2023-11-28T17:26:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp421.ko 2023-11-28T17:26:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via-cputemp.ko 2023-11-28T17:26:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via686a.ko 2023-11-28T17:26:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt1211.ko 2023-11-28T17:26:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt8231.ko 2023-11-28T17:26:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627ehf.ko 2023-11-28T17:26:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627hf.ko 2023-11-28T17:26:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83781d.ko 2023-11-28T17:26:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83791d.ko 2023-11-28T17:26:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83792d.ko 2023-11-28T17:26:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83793.ko 2023-11-28T17:26:16.295 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83795.ko 2023-11-28T17:26:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l785ts.ko 2023-11-28T17:26:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l786ng.ko 2023-11-28T17:26:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c 2023-11-28T17:26:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos 2023-11-28T17:26:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-11-28T17:26:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-11-28T17:26:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses 2023-11-28T17:26:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-11-28T17:26:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-11-28T17:26:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-11-28T17:26:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-11-28T17:26:16.296 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-11-28T17:26:16.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-11-28T17:26:16.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-11-28T17:26:16.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-11-28T17:26:16.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-i801.ko 2023-11-28T17:26:16.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-isch.ko 2023-11-28T17:26:16.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-11-28T17:26:16.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-11-28T17:26:16.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-11-28T17:26:16.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-parport.ko 2023-11-28T17:26:16.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-11-28T17:26:16.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-11-28T17:26:16.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-11-28T17:26:16.297 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-11-28T17:26:16.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-11-28T17:26:16.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-11-28T17:26:16.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-via.ko 2023-11-28T17:26:16.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-11-28T17:26:16.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-11-28T17:26:16.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-dev.ko 2023-11-28T17:26:16.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-mux.ko 2023-11-28T17:26:16.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-11-28T17:26:16.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-smbus.ko 2023-11-28T17:26:16.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-stub.ko 2023-11-28T17:26:16.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband 2023-11-28T17:26:16.298 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core 2023-11-28T17:26:16.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_cm.ko 2023-11-28T17:26:16.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_core.ko 2023-11-28T17:26:16.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_umad.ko 2023-11-28T17:26:16.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-11-28T17:26:16.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/iw_cm.ko 2023-11-28T17:26:16.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_cm.ko 2023-11-28T17:26:16.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-11-28T17:26:16.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw 2023-11-28T17:26:16.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4 2023-11-28T17:26:16.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-11-28T17:26:16.299 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4 2023-11-28T17:26:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-11-28T17:26:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5 2023-11-28T17:26:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-11-28T17:26:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca 2023-11-28T17:26:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-11-28T17:26:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp 2023-11-28T17:26:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib 2023-11-28T17:26:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-11-28T17:26:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser 2023-11-28T17:26:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-11-28T17:26:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert 2023-11-28T17:26:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-11-28T17:26:16.300 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp 2023-11-28T17:26:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-11-28T17:26:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt 2023-11-28T17:26:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-11-28T17:26:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input 2023-11-28T17:26:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport 2023-11-28T17:26:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-11-28T17:26:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/fm801-gp.ko 2023-11-28T17:26:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/gameport.ko 2023-11-28T17:26:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/joydev.ko 2023-11-28T17:26:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard 2023-11-28T17:26:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/gpio_keys.ko 2023-11-28T17:26:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse 2023-11-28T17:26:16.301 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/appletouch.ko 2023-11-28T17:26:16.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/bcm5974.ko 2023-11-28T17:26:16.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/cyapatp.ko 2023-11-28T17:26:16.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/elan_i2c.ko 2023-11-28T17:26:16.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/sermouse.ko 2023-11-28T17:26:16.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-11-28T17:26:16.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_usb.ko 2023-11-28T17:26:16.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/vsxxxaa.ko 2023-11-28T17:26:16.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4 2023-11-28T17:26:16.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4/rmi_core.ko 2023-11-28T17:26:16.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio 2023-11-28T17:26:16.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/altera_ps2.ko 2023-11-28T17:26:16.302 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/arc_ps2.ko 2023-11-28T17:26:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-11-28T17:26:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/serio_raw.ko 2023-11-28T17:26:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/sparse-keymap.ko 2023-11-28T17:26:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu 2023-11-28T17:26:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd 2023-11-28T17:26:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd/iommu_v2.ko 2023-11-28T17:26:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds 2023-11-28T17:26:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-blinkm.ko 2023-11-28T17:26:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm3530.ko 2023-11-28T17:26:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lp3944.ko 2023-11-28T17:26:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-ss4200.ko 2023-11-28T17:26:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger 2023-11-28T17:26:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-11-28T17:26:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-11-28T17:26:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-11-28T17:26:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-11-28T17:26:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-11-28T17:26:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-11-28T17:26:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-11-28T17:26:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-11-28T17:26:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md 2023-11-28T17:26:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache 2023-11-28T17:26:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache/bcache.ko 2023-11-28T17:26:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-bio-prison.ko 2023-11-28T17:26:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache-smq.ko 2023-11-28T17:26:16.304 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache.ko 2023-11-28T17:26:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-crypt.ko 2023-11-28T17:26:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-delay.ko 2023-11-28T17:26:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-flakey.ko 2023-11-28T17:26:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-log-userspace.ko 2023-11-28T17:26:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-multipath.ko 2023-11-28T17:26:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-queue-length.ko 2023-11-28T17:26:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-raid.ko 2023-11-28T17:26:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-round-robin.ko 2023-11-28T17:26:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-service-time.ko 2023-11-28T17:26:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-switch.ko 2023-11-28T17:26:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-thin-pool.ko 2023-11-28T17:26:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-verity.ko 2023-11-28T17:26:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/faulty.ko 2023-11-28T17:26:16.305 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/linear.ko 2023-11-28T17:26:16.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/multipath.ko 2023-11-28T17:26:16.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data 2023-11-28T17:26:16.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-11-28T17:26:16.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid0.ko 2023-11-28T17:26:16.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid1.ko 2023-11-28T17:26:16.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid10.ko 2023-11-28T17:26:16.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid456.ko 2023-11-28T17:26:16.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message 2023-11-28T17:26:16.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion 2023-11-28T17:26:16.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptbase.ko 2023-11-28T17:26:16.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptctl.ko 2023-11-28T17:26:16.306 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptfc.ko 2023-11-28T17:26:16.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptlan.ko 2023-11-28T17:26:16.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptsas.ko 2023-11-28T17:26:16.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptscsih.ko 2023-11-28T17:26:16.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptspi.ko 2023-11-28T17:26:16.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd 2023-11-28T17:26:16.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_ich.ko 2023-11-28T17:26:16.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_sch.ko 2023-11-28T17:26:16.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/mfd-core.ko 2023-11-28T17:26:16.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/sm501.ko 2023-11-28T17:26:16.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/viperboard.ko 2023-11-28T17:26:16.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/vx855.ko 2023-11-28T17:26:16.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/wl1273-core.ko 2023-11-28T17:26:16.307 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc 2023-11-28T17:26:16.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl 2023-11-28T17:26:16.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-11-28T17:26:16.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds9802als.ko 2023-11-28T17:26:16.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds990x.ko 2023-11-28T17:26:16.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/bh1770glc.ko 2023-11-28T17:26:16.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710 2023-11-28T17:26:16.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710/cb710.ko 2023-11-28T17:26:16.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom 2023-11-28T17:26:16.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/at24.ko 2023-11-28T17:26:16.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom.ko 2023-11-28T17:26:16.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-11-28T17:26:16.308 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/max6875.ko 2023-11-28T17:26:16.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/enclosure.ko 2023-11-28T17:26:16.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/hpilo.ko 2023-11-28T17:26:16.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29003.ko 2023-11-28T17:26:16.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29020.ko 2023-11-28T17:26:16.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d 2023-11-28T17:26:16.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-11-28T17:26:16.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-11-28T17:26:16.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei 2023-11-28T17:26:16.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-me.ko 2023-11-28T17:26:16.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-txe.ko 2023-11-28T17:26:16.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei.ko 2023-11-28T17:26:16.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru 2023-11-28T17:26:16.309 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru/gru.ko 2023-11-28T17:26:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp 2023-11-28T17:26:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xp.ko 2023-11-28T17:26:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpc.ko 2023-11-28T17:26:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-11-28T17:26:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_7xx1.ko 2023-11-28T17:26:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_core.ko 2023-11-28T17:26:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tsl2550.ko 2023-11-28T17:26:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_balloon.ko 2023-11-28T17:26:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci 2023-11-28T17:26:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-11-28T17:26:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net 2023-11-28T17:26:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding 2023-11-28T17:26:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding/bonding.ko 2023-11-28T17:26:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa 2023-11-28T17:26:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53 2023-11-28T17:26:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53/b53_common.ko 2023-11-28T17:26:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/bcm-sf2.ko 2023-11-28T17:26:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6060.ko 2023-11-28T17:26:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx 2023-11-28T17:26:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-11-28T17:26:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dummy.ko 2023-11-28T17:26:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/eql.ko 2023-11-28T17:26:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet 2023-11-28T17:26:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com 2023-11-28T17:26:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-11-28T17:26:16.311 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-11-28T17:26:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-11-28T17:26:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-11-28T17:26:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390 2023-11-28T17:26:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/8390.ko 2023-11-28T17:26:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-11-28T17:26:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-11-28T17:26:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-11-28T17:26:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec 2023-11-28T17:26:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-11-28T17:26:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere 2023-11-28T17:26:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere/et131x.ko 2023-11-28T17:26:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon 2023-11-28T17:26:16.312 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-11-28T17:26:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera 2023-11-28T17:26:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-11-28T17:26:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd 2023-11-28T17:26:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-11-28T17:26:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-11-28T17:26:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-11-28T17:26:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros 2023-11-28T17:26:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx 2023-11-28T17:26:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-11-28T17:26:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c 2023-11-28T17:26:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-11-28T17:26:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e 2023-11-28T17:26:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-11-28T17:26:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx 2023-11-28T17:26:16.314 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-11-28T17:26:16.314 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-11-28T17:26:16.314 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom 2023-11-28T17:26:16.314 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-11-28T17:26:16.314 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-11-28T17:26:16.314 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-11-28T17:26:16.314 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-11-28T17:26:16.314 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-11-28T17:26:16.314 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet 2023-11-28T17:26:16.314 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-11-28T17:26:16.314 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-11-28T17:26:16.314 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade 2023-11-28T17:26:16.315 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna 2023-11-28T17:26:16.315 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-11-28T17:26:16.315 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio 2023-11-28T17:26:16.315 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb 2023-11-28T17:26:16.315 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-11-28T17:26:16.315 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-11-28T17:26:16.315 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-11-28T17:26:16.315 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-11-28T17:26:16.315 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-11-28T17:26:16.315 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-11-28T17:26:16.315 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-11-28T17:26:16.315 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-11-28T17:26:16.315 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-11-28T17:26:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco 2023-11-28T17:26:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic 2023-11-28T17:26:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-11-28T17:26:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec 2023-11-28T17:26:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip 2023-11-28T17:26:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-11-28T17:26:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-11-28T17:26:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-11-28T17:26:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-11-28T17:26:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-11-28T17:26:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-11-28T17:26:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink 2023-11-28T17:26:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-11-28T17:26:16.317 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-11-28T17:26:16.317 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dnet.ko 2023-11-28T17:26:16.317 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex 2023-11-28T17:26:16.317 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet 2023-11-28T17:26:16.317 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-11-28T17:26:16.317 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ethoc.ko 2023-11-28T17:26:16.317 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/fealnx.ko 2023-11-28T17:26:16.317 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel 2023-11-28T17:26:16.317 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e100.ko 2023-11-28T17:26:16.317 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000 2023-11-28T17:26:16.317 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-11-28T17:26:16.317 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e 2023-11-28T17:26:16.318 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-11-28T17:26:16.318 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k 2023-11-28T17:26:16.318 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-11-28T17:26:16.318 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e 2023-11-28T17:26:16.318 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-11-28T17:26:16.318 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf 2023-11-28T17:26:16.318 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-11-28T17:26:16.318 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb 2023-11-28T17:26:16.318 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-11-28T17:26:16.318 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf 2023-11-28T17:26:16.318 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-11-28T17:26:16.318 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe 2023-11-28T17:26:16.318 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-11-28T17:26:16.319 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf 2023-11-28T17:26:16.319 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-11-28T17:26:16.319 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/jme.ko 2023-11-28T17:26:16.319 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell 2023-11-28T17:26:16.319 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-11-28T17:26:16.319 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/skge.ko 2023-11-28T17:26:16.319 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-11-28T17:26:16.319 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox 2023-11-28T17:26:16.319 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4 2023-11-28T17:26:16.319 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-11-28T17:26:16.319 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-11-28T17:26:16.321 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5 2023-11-28T17:26:16.321 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-11-28T17:26:16.321 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-11-28T17:26:16.321 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel 2023-11-28T17:26:16.321 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-11-28T17:26:16.321 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom 2023-11-28T17:26:16.321 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge 2023-11-28T17:26:16.321 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-11-28T17:26:16.321 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi 2023-11-28T17:26:16.322 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-11-28T17:26:16.322 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-11-28T17:26:16.322 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion 2023-11-28T17:26:16.322 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-11-28T17:26:16.322 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia 2023-11-28T17:26:16.322 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-11-28T17:26:16.322 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines 2023-11-28T17:26:16.322 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-11-28T17:26:16.322 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-11-28T17:26:16.322 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic 2023-11-28T17:26:16.322 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen 2023-11-28T17:26:16.322 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-11-28T17:26:16.323 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-11-28T17:26:16.323 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-11-28T17:26:16.323 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-11-28T17:26:16.323 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc 2023-11-28T17:26:16.323 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-11-28T17:26:16.323 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek 2023-11-28T17:26:16.323 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-11-28T17:26:16.323 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-11-28T17:26:16.323 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/atp.ko 2023-11-28T17:26:16.323 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-11-28T17:26:16.323 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker 2023-11-28T17:26:16.323 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-11-28T17:26:16.323 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc 2023-11-28T17:26:16.324 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-11-28T17:26:16.324 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan 2023-11-28T17:26:16.324 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-11-28T17:26:16.324 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis 2023-11-28T17:26:16.324 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis190.ko 2023-11-28T17:26:16.324 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis900.ko 2023-11-28T17:26:16.324 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc 2023-11-28T17:26:16.324 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-11-28T17:26:16.324 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-11-28T17:26:16.324 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-11-28T17:26:16.324 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-11-28T17:26:16.324 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro 2023-11-28T17:26:16.324 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac 2023-11-28T17:26:16.325 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-11-28T17:26:16.325 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-11-28T17:26:16.325 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun 2023-11-28T17:26:16.325 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/cassini.ko 2023-11-28T17:26:16.325 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/niu.ko 2023-11-28T17:26:16.325 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sungem.ko 2023-11-28T17:26:16.325 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-11-28T17:26:16.325 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti 2023-11-28T17:26:16.325 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-11-28T17:26:16.325 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti 2023-11-28T17:26:16.325 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti/tlan.ko 2023-11-28T17:26:16.325 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via 2023-11-28T17:26:16.325 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-11-28T17:26:16.326 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-11-28T17:26:16.326 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet 2023-11-28T17:26:16.326 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-11-28T17:26:16.326 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-11-28T17:26:16.326 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom 2023-11-28T17:26:16.326 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-11-28T17:26:16.326 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/geneve.ko 2023-11-28T17:26:16.326 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv 2023-11-28T17:26:16.326 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-11-28T17:26:16.326 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154 2023-11-28T17:26:16.326 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154/fakelb.ko 2023-11-28T17:26:16.326 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ifb.ko 2023-11-28T17:26:16.327 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan 2023-11-28T17:26:16.327 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan/ipvlan.ko 2023-11-28T17:26:16.327 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvlan.ko 2023-11-28T17:26:16.327 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvtap.ko 2023-11-28T17:26:16.327 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio 2023-11-28T17:26:16.327 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio.ko 2023-11-28T17:26:16.327 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-11-28T17:26:16.327 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-11-28T17:26:16.327 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-regmap.ko 2023-11-28T17:26:16.327 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mii.ko 2023-11-28T17:26:16.327 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/net_failover.ko 2023-11-28T17:26:16.327 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/netconsole.ko 2023-11-28T17:26:16.327 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/nlmon.ko 2023-11-28T17:26:16.328 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs 2023-11-28T17:26:16.328 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs-lynx.ko 2023-11-28T17:26:16.328 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-11-28T17:26:16.328 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy 2023-11-28T17:26:16.328 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/amd.ko 2023-11-28T17:26:16.328 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-11-28T17:26:16.328 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-11-28T17:26:16.328 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm7xxx.ko 2023-11-28T17:26:16.328 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm87xx.ko 2023-11-28T17:26:16.328 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/broadcom.ko 2023-11-28T17:26:16.328 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/cicada.ko 2023-11-28T17:26:16.328 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/davicom.ko 2023-11-28T17:26:16.328 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/dp83640.ko 2023-11-28T17:26:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/et1011c.ko 2023-11-28T17:26:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/icplus.ko 2023-11-28T17:26:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/lxt.ko 2023-11-28T17:26:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/marvell.ko 2023-11-28T17:26:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/micrel.ko 2023-11-28T17:26:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/national.ko 2023-11-28T17:26:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/phylink.ko 2023-11-28T17:26:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/qsemi.ko 2023-11-28T17:26:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/realtek.ko 2023-11-28T17:26:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/smsc.ko 2023-11-28T17:26:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/ste10Xp.ko 2023-11-28T17:26:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/vitesse.ko 2023-11-28T17:26:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp 2023-11-28T17:26:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/bsd_comp.ko 2023-11-28T17:26:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_async.ko 2023-11-28T17:26:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_deflate.ko 2023-11-28T17:26:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_generic.ko 2023-11-28T17:26:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_mppe.ko 2023-11-28T17:26:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_synctty.ko 2023-11-28T17:26:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppoe.ko 2023-11-28T17:26:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppox.ko 2023-11-28T17:26:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pptp.ko 2023-11-28T17:26:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip 2023-11-28T17:26:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slhc.ko 2023-11-28T17:26:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slip.ko 2023-11-28T17:26:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/sungem_phy.ko 2023-11-28T17:26:16.331 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tap.ko 2023-11-28T17:26:16.331 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team 2023-11-28T17:26:16.331 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team.ko 2023-11-28T17:26:16.331 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_activebackup.ko 2023-11-28T17:26:16.331 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_broadcast.ko 2023-11-28T17:26:16.331 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-11-28T17:26:16.331 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_random.ko 2023-11-28T17:26:16.331 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-11-28T17:26:16.331 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tun.ko 2023-11-28T17:26:16.331 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/veth.ko 2023-11-28T17:26:16.331 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/virtio_net.ko 2023-11-28T17:26:16.331 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3 2023-11-28T17:26:16.331 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-11-28T17:26:16.332 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan 2023-11-28T17:26:16.332 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan/vxlan.ko 2023-11-28T17:26:16.332 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback 2023-11-28T17:26:16.332 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback/xen-netback.ko 2023-11-28T17:26:16.332 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netfront.ko 2023-11-28T17:26:16.332 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb 2023-11-28T17:26:16.332 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb/ntb.ko 2023-11-28T17:26:16.332 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme 2023-11-28T17:26:16.332 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host 2023-11-28T17:26:16.332 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme-core.ko 2023-11-28T17:26:16.332 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme.ko 2023-11-28T17:26:16.332 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport 2023-11-28T17:26:16.332 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport.ko 2023-11-28T17:26:16.333 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_cs.ko 2023-11-28T17:26:16.333 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_pc.ko 2023-11-28T17:26:16.333 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_serial.ko 2023-11-28T17:26:16.333 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci 2023-11-28T17:26:16.333 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug 2023-11-28T17:26:16.333 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-11-28T17:26:16.333 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie 2023-11-28T17:26:16.333 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie/aer_inject.ko 2023-11-28T17:26:16.333 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/xen-pcifront.ko 2023-11-28T17:26:16.333 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia 2023-11-28T17:26:16.333 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/i82092.ko 2023-11-28T17:26:16.333 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pd6729.ko 2023-11-28T17:26:16.333 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/yenta_socket.ko 2023-11-28T17:26:16.334 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl 2023-11-28T17:26:16.334 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel 2023-11-28T17:26:16.334 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-11-28T17:26:16.334 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform 2023-11-28T17:26:16.334 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86 2023-11-28T17:26:16.334 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acer-wmi.ko 2023-11-28T17:26:16.334 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acerhdf.ko 2023-11-28T17:26:16.334 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/apple-gmux.ko 2023-11-28T17:26:16.334 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-laptop.ko 2023-11-28T17:26:16.334 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-11-28T17:26:16.334 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-wmi.ko 2023-11-28T17:26:16.334 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/classmate-laptop.ko 2023-11-28T17:26:16.335 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-11-28T17:26:16.335 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-11-28T17:26:16.335 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-11-28T17:26:16.335 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-11-28T17:26:16.335 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/hdaps.ko 2023-11-28T17:26:16.335 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel 2023-11-28T17:26:16.335 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-11-28T17:26:16.335 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel_ips.ko 2023-11-28T17:26:16.335 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/msi-wmi.ko 2023-11-28T17:26:16.335 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/mxm-wmi.ko 2023-11-28T17:26:16.335 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-11-28T17:26:16.335 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-laptop.ko 2023-11-28T17:26:16.335 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-q10.ko 2023-11-28T17:26:16.335 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/topstar-laptop.ko 2023-11-28T17:26:16.336 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-11-28T17:26:16.336 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_haps.ko 2023-11-28T17:26:16.336 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi-bmof.ko 2023-11-28T17:26:16.336 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi.ko 2023-11-28T17:26:16.336 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap 2023-11-28T17:26:16.336 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_common.ko 2023-11-28T17:26:16.336 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_msr.ko 2023-11-28T17:26:16.336 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps 2023-11-28T17:26:16.336 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients 2023-11-28T17:26:16.336 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-gpio.ko 2023-11-28T17:26:16.336 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-ldisc.ko 2023-11-28T17:26:16.336 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps_parport.ko 2023-11-28T17:26:16.337 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/pps_core.ko 2023-11-28T17:26:16.337 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp 2023-11-28T17:26:16.337 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp.ko 2023-11-28T17:26:16.337 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp_kvm.ko 2023-11-28T17:26:16.337 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc 2023-11-28T17:26:16.337 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-bq32k.ko 2023-11-28T17:26:16.337 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1286.ko 2023-11-28T17:26:16.337 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1307.ko 2023-11-28T17:26:16.337 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1374.ko 2023-11-28T17:26:16.337 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1511.ko 2023-11-28T17:26:16.337 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1553.ko 2023-11-28T17:26:16.337 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1672.ko 2023-11-28T17:26:16.337 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1742.ko 2023-11-28T17:26:16.338 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds2404.ko 2023-11-28T17:26:16.338 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds3232.ko 2023-11-28T17:26:16.338 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-em3027.ko 2023-11-28T17:26:16.338 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-fm3130.ko 2023-11-28T17:26:16.338 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl12022.ko 2023-11-28T17:26:16.338 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl1208.ko 2023-11-28T17:26:16.338 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m41t80.ko 2023-11-28T17:26:16.338 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t35.ko 2023-11-28T17:26:16.338 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t59.ko 2023-11-28T17:26:16.338 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max6900.ko 2023-11-28T17:26:16.338 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-msm6242.ko 2023-11-28T17:26:16.338 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf2127.ko 2023-11-28T17:26:16.338 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf85063.ko 2023-11-28T17:26:16.339 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8523.ko 2023-11-28T17:26:16.339 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8563.ko 2023-11-28T17:26:16.339 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8583.ko 2023-11-28T17:26:16.339 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rp5c01.ko 2023-11-28T17:26:16.339 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rs5c372.ko 2023-11-28T17:26:16.339 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-11-28T17:26:16.339 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8025.ko 2023-11-28T17:26:16.339 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8581.ko 2023-11-28T17:26:16.339 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-11-28T17:26:16.339 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-x1205.ko 2023-11-28T17:26:16.339 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi 2023-11-28T17:26:16.339 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-9xxx.ko 2023-11-28T17:26:16.339 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-sas.ko 2023-11-28T17:26:16.340 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-xxxx.ko 2023-11-28T17:26:16.340 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/BusLogic.ko 2023-11-28T17:26:16.340 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/a100u2w.ko 2023-11-28T17:26:16.340 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid 2023-11-28T17:26:16.340 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid/aacraid.ko 2023-11-28T17:26:16.340 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/advansys.ko 2023-11-28T17:26:16.340 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx 2023-11-28T17:26:16.340 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-11-28T17:26:16.340 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-11-28T17:26:16.340 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/am53c974.ko 2023-11-28T17:26:16.340 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr 2023-11-28T17:26:16.340 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-11-28T17:26:16.340 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/atp870u.ko 2023-11-28T17:26:16.340 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi 2023-11-28T17:26:16.341 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-11-28T17:26:16.341 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa 2023-11-28T17:26:16.341 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa/bfa.ko 2023-11-28T17:26:16.341 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc 2023-11-28T17:26:16.341 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-11-28T17:26:16.341 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i 2023-11-28T17:26:16.341 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-11-28T17:26:16.341 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ch.ko 2023-11-28T17:26:16.341 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor 2023-11-28T17:26:16.341 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor/csiostor.ko 2023-11-28T17:26:16.341 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi 2023-11-28T17:26:16.341 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i 2023-11-28T17:26:16.342 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-11-28T17:26:16.342 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i 2023-11-28T17:26:16.342 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-11-28T17:26:16.342 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-11-28T17:26:16.342 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dc395x.ko 2023-11-28T17:26:16.342 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler 2023-11-28T17:26:16.342 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-11-28T17:26:16.342 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-11-28T17:26:16.342 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-11-28T17:26:16.342 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-11-28T17:26:16.342 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dmx3191d.ko 2023-11-28T17:26:16.342 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r 2023-11-28T17:26:16.343 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r/esas2r.ko 2023-11-28T17:26:16.343 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esp_scsi.ko 2023-11-28T17:26:16.343 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe 2023-11-28T17:26:16.343 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/fcoe.ko 2023-11-28T17:26:16.343 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-11-28T17:26:16.343 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic 2023-11-28T17:26:16.343 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic/fnic.ko 2023-11-28T17:26:16.343 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hpsa.ko 2023-11-28T17:26:16.343 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hptiop.ko 2023-11-28T17:26:16.343 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hv_storvsc.ko 2023-11-28T17:26:16.343 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/initio.ko 2023-11-28T17:26:16.343 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ipr.ko 2023-11-28T17:26:16.343 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ips.ko 2023-11-28T17:26:16.344 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci 2023-11-28T17:26:16.344 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci/isci.ko 2023-11-28T17:26:16.344 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-11-28T17:26:16.344 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_tcp.ko 2023-11-28T17:26:16.344 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc 2023-11-28T17:26:16.344 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc/libfc.ko 2023-11-28T17:26:16.344 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi.ko 2023-11-28T17:26:16.344 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi_tcp.ko 2023-11-28T17:26:16.344 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas 2023-11-28T17:26:16.344 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas/libsas.ko 2023-11-28T17:26:16.344 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc 2023-11-28T17:26:16.344 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc/lpfc.ko 2023-11-28T17:26:16.344 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid 2023-11-28T17:26:16.345 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid.ko 2023-11-28T17:26:16.345 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-11-28T17:26:16.345 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-11-28T17:26:16.345 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-11-28T17:26:16.345 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas 2023-11-28T17:26:16.345 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-11-28T17:26:16.345 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas 2023-11-28T17:26:16.345 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas/mvsas.ko 2023-11-28T17:26:16.345 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvumi.ko 2023-11-28T17:26:16.345 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001 2023-11-28T17:26:16.345 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-11-28T17:26:16.345 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pmcraid.ko 2023-11-28T17:26:16.345 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla1280.ko 2023-11-28T17:26:16.346 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx 2023-11-28T17:26:16.346 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-11-28T17:26:16.346 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-11-28T17:26:16.346 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx 2023-11-28T17:26:16.346 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-11-28T17:26:16.346 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/raid_class.ko 2023-11-28T17:26:16.346 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_debug.ko 2023-11-28T17:26:16.346 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_fc.ko 2023-11-28T17:26:16.346 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-11-28T17:26:16.346 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_sas.ko 2023-11-28T17:26:16.346 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_spi.ko 2023-11-28T17:26:16.347 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_srp.ko 2023-11-28T17:26:16.347 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ses.ko 2023-11-28T17:26:16.347 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/st.ko 2023-11-28T17:26:16.347 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/stex.ko 2023-11-28T17:26:16.347 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2 2023-11-28T17:26:16.347 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-11-28T17:26:16.347 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/virtio_scsi.ko 2023-11-28T17:26:16.347 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/vmw_pvscsi.ko 2023-11-28T17:26:16.347 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/wd719x.ko 2023-11-28T17:26:16.347 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb 2023-11-28T17:26:16.347 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb/ssb.ko 2023-11-28T17:26:16.347 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target 2023-11-28T17:26:16.347 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi 2023-11-28T17:26:16.348 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-11-28T17:26:16.348 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback 2023-11-28T17:26:16.348 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback/tcm_loop.ko 2023-11-28T17:26:16.348 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_file.ko 2023-11-28T17:26:16.348 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_iblock.ko 2023-11-28T17:26:16.348 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_mod.ko 2023-11-28T17:26:16.348 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_pscsi.ko 2023-11-28T17:26:16.348 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_user.ko 2023-11-28T17:26:16.348 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc 2023-11-28T17:26:16.348 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-11-28T17:26:16.348 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal 2023-11-28T17:26:16.348 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel 2023-11-28T17:26:16.349 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal 2023-11-28T17:26:16.349 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-11-28T17:26:16.349 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-11-28T17:26:16.349 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-11-28T17:26:16.349 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-11-28T17:26:16.349 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-11-28T17:26:16.349 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-11-28T17:26:16.349 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-11-28T17:26:16.349 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-11-28T17:26:16.349 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-11-28T17:26:16.349 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-11-28T17:26:16.349 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-11-28T17:26:16.349 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-11-28T17:26:16.350 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-11-28T17:26:16.350 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-11-28T17:26:16.350 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-11-28T17:26:16.350 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty 2023-11-28T17:26:16.350 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless 2023-11-28T17:26:16.350 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-11-28T17:26:16.350 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_gsm.ko 2023-11-28T17:26:16.350 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_hdlc.ko 2023-11-28T17:26:16.350 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/nozomi.ko 2023-11-28T17:26:16.350 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial 2023-11-28T17:26:16.350 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250 2023-11-28T17:26:16.350 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-11-28T17:26:16.351 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/arc_uart.ko 2023-11-28T17:26:16.351 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm 2023-11-28T17:26:16.351 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm/jsm.ko 2023-11-28T17:26:16.351 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs 2023-11-28T17:26:16.351 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core 2023-11-28T17:26:16.351 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core/ufshcd-core.ko 2023-11-28T17:26:16.351 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host 2023-11-28T17:26:16.351 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-11-28T17:26:16.351 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio 2023-11-28T17:26:16.351 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio.ko 2023-11-28T17:26:16.351 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_aec.ko 2023-11-28T17:26:16.351 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_cif.ko 2023-11-28T17:26:16.351 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_pci_generic.ko 2023-11-28T17:26:16.352 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_sercos3.ko 2023-11-28T17:26:16.352 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb 2023-11-28T17:26:16.352 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class 2023-11-28T17:26:16.352 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-acm.ko 2023-11-28T17:26:16.352 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-wdm.ko 2023-11-28T17:26:16.352 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usblp.ko 2023-11-28T17:26:16.352 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usbtmc.ko 2023-11-28T17:26:16.352 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host 2023-11-28T17:26:16.352 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/sl811-hcd.ko 2023-11-28T17:26:16.352 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image 2023-11-28T17:26:16.352 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/mdc800.ko 2023-11-28T17:26:16.352 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/microtek.ko 2023-11-28T17:26:16.353 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc 2023-11-28T17:26:16.353 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/adutux.ko 2023-11-28T17:26:16.353 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/appledisplay.ko 2023-11-28T17:26:16.353 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi26.ko 2023-11-28T17:26:16.353 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi62.ko 2023-11-28T17:26:16.353 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ezusb.ko 2023-11-28T17:26:16.353 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/idmouse.ko 2023-11-28T17:26:16.353 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/iowarrior.ko 2023-11-28T17:26:16.353 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/isight_firmware.ko 2023-11-28T17:26:16.353 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ldusb.ko 2023-11-28T17:26:16.353 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/legousbtower.ko 2023-11-28T17:26:16.353 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga 2023-11-28T17:26:16.353 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-11-28T17:26:16.354 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/trancevibrator.ko 2023-11-28T17:26:16.354 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usb3503.ko 2023-11-28T17:26:16.354 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usblcd.ko 2023-11-28T17:26:16.354 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usbsevseg.ko 2023-11-28T17:26:16.354 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/uss720.ko 2023-11-28T17:26:16.354 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/yurex.ko 2023-11-28T17:26:16.354 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy 2023-11-28T17:26:16.354 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-generic.ko 2023-11-28T17:26:16.354 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial 2023-11-28T17:26:16.354 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/aircable.ko 2023-11-28T17:26:16.354 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ark3116.ko 2023-11-28T17:26:16.354 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/belkin_sa.ko 2023-11-28T17:26:16.355 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ch341.ko 2023-11-28T17:26:16.355 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cp210x.ko 2023-11-28T17:26:16.355 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cyberjack.ko 2023-11-28T17:26:16.355 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cypress_m8.ko 2023-11-28T17:26:16.355 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/digi_acceleport.ko 2023-11-28T17:26:16.355 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/empeg.ko 2023-11-28T17:26:16.355 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ftdi_sio.ko 2023-11-28T17:26:16.355 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/garmin_gps.ko 2023-11-28T17:26:16.355 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_edgeport.ko 2023-11-28T17:26:16.355 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_ti.ko 2023-11-28T17:26:16.355 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipaq.ko 2023-11-28T17:26:16.355 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipw.ko 2023-11-28T17:26:16.356 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ir-usb.ko 2023-11-28T17:26:16.356 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-11-28T17:26:16.356 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan.ko 2023-11-28T17:26:16.356 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan_pda.ko 2023-11-28T17:26:16.356 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kl5kusb105.ko 2023-11-28T17:26:16.356 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kobil_sct.ko 2023-11-28T17:26:16.356 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mct_u232.ko 2023-11-28T17:26:16.356 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7720.ko 2023-11-28T17:26:16.356 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7840.ko 2023-11-28T17:26:16.356 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/navman.ko 2023-11-28T17:26:16.356 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/omninet.ko 2023-11-28T17:26:16.356 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/opticon.ko 2023-11-28T17:26:16.357 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/option.ko 2023-11-28T17:26:16.357 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/oti6858.ko 2023-11-28T17:26:16.357 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/pl2303.ko 2023-11-28T17:26:16.357 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcaux.ko 2023-11-28T17:26:16.357 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcserial.ko 2023-11-28T17:26:16.357 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/quatech2.ko 2023-11-28T17:26:16.357 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/safe_serial.ko 2023-11-28T17:26:16.357 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/sierra.ko 2023-11-28T17:26:16.357 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/spcp8x5.ko 2023-11-28T17:26:16.358 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ssu100.ko 2023-11-28T17:26:16.358 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/symbolserial.ko 2023-11-28T17:26:16.358 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-11-28T17:26:16.358 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-11-28T17:26:16.358 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_debug.ko 2023-11-28T17:26:16.358 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_wwan.ko 2023-11-28T17:26:16.358 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/visor.ko 2023-11-28T17:26:16.358 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/whiteheat.ko 2023-11-28T17:26:16.358 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/xsens_mt.ko 2023-11-28T17:26:16.358 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage 2023-11-28T17:26:16.358 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/uas.ko 2023-11-28T17:26:16.358 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-alauda.ko 2023-11-28T17:26:16.359 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-cypress.ko 2023-11-28T17:26:16.359 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-datafab.ko 2023-11-28T17:26:16.359 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-11-28T17:26:16.359 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-freecom.ko 2023-11-28T17:26:16.359 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-isd200.ko 2023-11-28T17:26:16.359 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-11-28T17:26:16.359 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-karma.ko 2023-11-28T17:26:16.359 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-onetouch.ko 2023-11-28T17:26:16.359 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-realtek.ko 2023-11-28T17:26:16.359 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr09.ko 2023-11-28T17:26:16.359 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr55.ko 2023-11-28T17:26:16.359 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-usbat.ko 2023-11-28T17:26:16.360 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/usb-storage.ko 2023-11-28T17:26:16.360 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip 2023-11-28T17:26:16.360 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-core.ko 2023-11-28T17:26:16.360 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-host.ko 2023-11-28T17:26:16.360 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-11-28T17:26:16.360 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio 2023-11-28T17:26:16.360 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci 2023-11-28T17:26:16.360 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-11-28T17:26:16.360 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci.ko 2023-11-28T17:26:16.360 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio.ko 2023-11-28T17:26:16.360 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-11-28T17:26:16.360 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost 2023-11-28T17:26:16.361 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost.ko 2023-11-28T17:26:16.361 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_iotlb.ko 2023-11-28T17:26:16.361 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_net.ko 2023-11-28T17:26:16.361 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_scsi.ko 2023-11-28T17:26:16.361 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video 2023-11-28T17:26:16.361 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight 2023-11-28T17:26:16.361 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/apple_bl.ko 2023-11-28T17:26:16.361 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lcd.ko 2023-11-28T17:26:16.361 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/platform_lcd.ko 2023-11-28T17:26:16.361 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio 2023-11-28T17:26:16.361 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio.ko 2023-11-28T17:26:16.361 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_balloon.ko 2023-11-28T17:26:16.361 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_mmio.ko 2023-11-28T17:26:16.362 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci.ko 2023-11-28T17:26:16.362 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-11-28T17:26:16.362 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-11-28T17:26:16.362 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_ring.ko 2023-11-28T17:26:16.362 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1 2023-11-28T17:26:16.362 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters 2023-11-28T17:26:16.362 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2482.ko 2023-11-28T17:26:16.362 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2490.ko 2023-11-28T17:26:16.362 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves 2023-11-28T17:26:16.362 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-11-28T17:26:16.362 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-11-28T17:26:16.362 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-11-28T17:26:16.363 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-11-28T17:26:16.363 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-11-28T17:26:16.363 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-11-28T17:26:16.363 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-11-28T17:26:16.363 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-11-28T17:26:16.363 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-11-28T17:26:16.363 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_smem.ko 2023-11-28T17:26:16.363 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_therm.ko 2023-11-28T17:26:16.363 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/wire.ko 2023-11-28T17:26:16.363 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog 2023-11-28T17:26:16.363 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim1535_wdt.ko 2023-11-28T17:26:16.363 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim7101_wdt.ko 2023-11-28T17:26:16.364 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/f71808e_wdt.ko 2023-11-28T17:26:16.364 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/hpwdt.ko 2023-11-28T17:26:16.364 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/i6300esb.ko 2023-11-28T17:26:16.364 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-11-28T17:26:16.364 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_wdt.ko 2023-11-28T17:26:16.364 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ib700wdt.ko 2023-11-28T17:26:16.364 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ibmasr.ko 2023-11-28T17:26:16.364 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-11-28T17:26:16.364 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it8712f_wdt.ko 2023-11-28T17:26:16.364 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it87_wdt.ko 2023-11-28T17:26:16.364 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/machzwd.ko 2023-11-28T17:26:16.364 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/nv_tco.ko 2023-11-28T17:26:16.364 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_pci.ko 2023-11-28T17:26:16.365 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_usb.ko 2023-11-28T17:26:16.365 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-11-28T17:26:16.365 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sch311x_wdt.ko 2023-11-28T17:26:16.365 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/softdog.ko 2023-11-28T17:26:16.365 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sp5100_tco.ko 2023-11-28T17:26:16.365 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/via_wdt.ko 2023-11-28T17:26:16.365 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-11-28T17:26:16.365 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83877f_wdt.ko 2023-11-28T17:26:16.365 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83977f_wdt.ko 2023-11-28T17:26:16.365 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wdt_pci.ko 2023-11-28T17:26:16.365 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/xen_wdt.ko 2023-11-28T17:26:16.365 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen 2023-11-28T17:26:16.366 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-acpi-processor.ko 2023-11-28T17:26:16.366 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-evtchn.ko 2023-11-28T17:26:16.366 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntalloc.ko 2023-11-28T17:26:16.366 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntdev.ko 2023-11-28T17:26:16.366 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback 2023-11-28T17:26:16.366 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-11-28T17:26:16.366 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-privcmd.ko 2023-11-28T17:26:16.366 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs 2023-11-28T17:26:16.366 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs/xenfs.ko 2023-11-28T17:26:16.366 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs 2023-11-28T17:26:16.366 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p 2023-11-28T17:26:16.367 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p/9p.ko 2023-11-28T17:26:16.367 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs 2023-11-28T17:26:16.367 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs/affs.ko 2023-11-28T17:26:16.367 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs 2023-11-28T17:26:16.367 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs/befs.ko 2023-11-28T17:26:16.367 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/binfmt_misc.ko 2023-11-28T17:26:16.367 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs 2023-11-28T17:26:16.367 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs/btrfs.ko 2023-11-28T17:26:16.367 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles 2023-11-28T17:26:16.367 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles/cachefiles.ko 2023-11-28T17:26:16.367 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph 2023-11-28T17:26:16.367 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph/ceph.ko 2023-11-28T17:26:16.367 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda 2023-11-28T17:26:16.368 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda/coda.ko 2023-11-28T17:26:16.368 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs 2023-11-28T17:26:16.368 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs/cramfs.ko 2023-11-28T17:26:16.368 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm 2023-11-28T17:26:16.368 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm/dlm.ko 2023-11-28T17:26:16.368 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs 2023-11-28T17:26:16.368 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs/ecryptfs.ko 2023-11-28T17:26:16.368 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs 2023-11-28T17:26:16.368 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs/f2fs.ko 2023-11-28T17:26:16.368 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat 2023-11-28T17:26:16.368 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/fat.ko 2023-11-28T17:26:16.368 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/msdos.ko 2023-11-28T17:26:16.368 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/vfat.ko 2023-11-28T17:26:16.369 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache 2023-11-28T17:26:16.369 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache/fscache.ko 2023-11-28T17:26:16.369 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse 2023-11-28T17:26:16.369 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/cuse.ko 2023-11-28T17:26:16.369 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/fuse.ko 2023-11-28T17:26:16.369 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2 2023-11-28T17:26:16.369 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2/gfs2.ko 2023-11-28T17:26:16.369 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs 2023-11-28T17:26:16.369 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs/hfs.ko 2023-11-28T17:26:16.369 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus 2023-11-28T17:26:16.369 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus/hfsplus.ko 2023-11-28T17:26:16.369 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs 2023-11-28T17:26:16.369 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs/isofs.ko 2023-11-28T17:26:16.370 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs 2023-11-28T17:26:16.370 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs/jfs.ko 2023-11-28T17:26:16.370 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd 2023-11-28T17:26:16.370 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd/lockd.ko 2023-11-28T17:26:16.370 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix 2023-11-28T17:26:16.370 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix/minix.ko 2023-11-28T17:26:16.370 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs 2023-11-28T17:26:16.370 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs/netfs.ko 2023-11-28T17:26:16.370 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs 2023-11-28T17:26:16.370 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout 2023-11-28T17:26:16.370 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-11-28T17:26:16.370 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout 2023-11-28T17:26:16.371 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-11-28T17:26:16.371 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout 2023-11-28T17:26:16.371 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-11-28T17:26:16.371 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfs.ko 2023-11-28T17:26:16.371 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv3.ko 2023-11-28T17:26:16.371 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv4.ko 2023-11-28T17:26:16.371 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common 2023-11-28T17:26:16.371 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/grace.ko 2023-11-28T17:26:16.371 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/nfs_acl.ko 2023-11-28T17:26:16.371 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd 2023-11-28T17:26:16.371 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd/nfsd.ko 2023-11-28T17:26:16.371 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2 2023-11-28T17:26:16.372 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2/nilfs2.ko 2023-11-28T17:26:16.372 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls 2023-11-28T17:26:16.372 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-celtic.ko 2023-11-28T17:26:16.372 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-centeuro.ko 2023-11-28T17:26:16.372 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-croatian.ko 2023-11-28T17:26:16.372 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-cyrillic.ko 2023-11-28T17:26:16.372 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-gaelic.ko 2023-11-28T17:26:16.372 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-greek.ko 2023-11-28T17:26:16.372 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-iceland.ko 2023-11-28T17:26:16.372 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-inuit.ko 2023-11-28T17:26:16.372 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-roman.ko 2023-11-28T17:26:16.372 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-romanian.ko 2023-11-28T17:26:16.372 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-turkish.ko 2023-11-28T17:26:16.373 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1250.ko 2023-11-28T17:26:16.373 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1251.ko 2023-11-28T17:26:16.373 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1255.ko 2023-11-28T17:26:16.373 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp737.ko 2023-11-28T17:26:16.373 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp775.ko 2023-11-28T17:26:16.373 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp850.ko 2023-11-28T17:26:16.373 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp852.ko 2023-11-28T17:26:16.373 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp855.ko 2023-11-28T17:26:16.373 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp857.ko 2023-11-28T17:26:16.373 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp860.ko 2023-11-28T17:26:16.373 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp861.ko 2023-11-28T17:26:16.373 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp862.ko 2023-11-28T17:26:16.374 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp863.ko 2023-11-28T17:26:16.374 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp864.ko 2023-11-28T17:26:16.374 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp865.ko 2023-11-28T17:26:16.374 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp866.ko 2023-11-28T17:26:16.374 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp869.ko 2023-11-28T17:26:16.374 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp874.ko 2023-11-28T17:26:16.374 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp932.ko 2023-11-28T17:26:16.374 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp936.ko 2023-11-28T17:26:16.374 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp949.ko 2023-11-28T17:26:16.374 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp950.ko 2023-11-28T17:26:16.374 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_euc-jp.ko 2023-11-28T17:26:16.374 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-1.ko 2023-11-28T17:26:16.375 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-13.ko 2023-11-28T17:26:16.375 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-14.ko 2023-11-28T17:26:16.375 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-15.ko 2023-11-28T17:26:16.375 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-2.ko 2023-11-28T17:26:16.375 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-3.ko 2023-11-28T17:26:16.375 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-4.ko 2023-11-28T17:26:16.375 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-5.ko 2023-11-28T17:26:16.375 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-6.ko 2023-11-28T17:26:16.375 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-7.ko 2023-11-28T17:26:16.375 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-9.ko 2023-11-28T17:26:16.375 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-r.ko 2023-11-28T17:26:16.375 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-ru.ko 2023-11-28T17:26:16.375 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-u.ko 2023-11-28T17:26:16.376 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_ucs2_utils.ko 2023-11-28T17:26:16.376 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_utf8.ko 2023-11-28T17:26:16.376 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2 2023-11-28T17:26:16.376 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster 2023-11-28T17:26:16.376 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-11-28T17:26:16.376 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm 2023-11-28T17:26:16.376 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-11-28T17:26:16.376 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs 2023-11-28T17:26:16.376 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-11-28T17:26:16.376 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2.ko 2023-11-28T17:26:16.376 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-11-28T17:26:16.376 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-11-28T17:26:16.377 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-11-28T17:26:16.377 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs 2023-11-28T17:26:16.377 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs/overlay.ko 2023-11-28T17:26:16.377 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore 2023-11-28T17:26:16.377 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore/ramoops.ko 2023-11-28T17:26:16.377 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs 2023-11-28T17:26:16.377 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs/reiserfs.ko 2023-11-28T17:26:16.377 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs 2023-11-28T17:26:16.377 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs/romfs.ko 2023-11-28T17:26:16.377 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb 2023-11-28T17:26:16.377 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client 2023-11-28T17:26:16.377 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client/cifs.ko 2023-11-28T17:26:16.377 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common 2023-11-28T17:26:16.378 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_arc4.ko 2023-11-28T17:26:16.378 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_md4.ko 2023-11-28T17:26:16.378 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs 2023-11-28T17:26:16.378 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs/squashfs.ko 2023-11-28T17:26:16.378 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv 2023-11-28T17:26:16.378 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv/sysv.ko 2023-11-28T17:26:16.378 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf 2023-11-28T17:26:16.378 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf/udf.ko 2023-11-28T17:26:16.378 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs 2023-11-28T17:26:16.378 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs/ufs.ko 2023-11-28T17:26:16.378 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs 2023-11-28T17:26:16.378 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs/xfs.ko 2023-11-28T17:26:16.379 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel 2023-11-28T17:26:16.379 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu 2023-11-28T17:26:16.379 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu/rcutorture.ko 2023-11-28T17:26:16.379 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/torture.ko 2023-11-28T17:26:16.379 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace 2023-11-28T17:26:16.379 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-11-28T17:26:16.379 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib 2023-11-28T17:26:16.379 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/asn1_encoder.ko 2023-11-28T17:26:16.379 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc-itu-t.ko 2023-11-28T17:26:16.379 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc8.ko 2023-11-28T17:26:16.379 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto 2023-11-28T17:26:16.379 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libarc4.ko 2023-11-28T17:26:16.379 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libdes.ko 2023-11-28T17:26:16.380 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/libcrc32c.ko 2023-11-28T17:26:16.380 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lru_cache.ko 2023-11-28T17:26:16.380 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4 2023-11-28T17:26:16.380 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4_compress.ko 2023-11-28T17:26:16.380 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4hc_compress.ko 2023-11-28T17:26:16.380 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math 2023-11-28T17:26:16.380 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math/cordic.ko 2023-11-28T17:26:16.380 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6 2023-11-28T17:26:16.380 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6/raid6_pq.ko 2023-11-28T17:26:16.380 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon 2023-11-28T17:26:16.380 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon/reed_solomon.ko 2023-11-28T17:26:16.380 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_bm.ko 2023-11-28T17:26:16.381 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_fsm.ko 2023-11-28T17:26:16.381 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_kmp.ko 2023-11-28T17:26:16.381 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd 2023-11-28T17:26:16.381 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd/zstd_compress.ko 2023-11-28T17:26:16.381 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm 2023-11-28T17:26:16.381 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm/hwpoison-inject.ko 2023-11-28T17:26:16.381 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net 2023-11-28T17:26:16.381 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan 2023-11-28T17:26:16.381 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/6lowpan.ko 2023-11-28T17:26:16.381 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_dest.ko 2023-11-28T17:26:16.381 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_fragment.ko 2023-11-28T17:26:16.381 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_hop.ko 2023-11-28T17:26:16.382 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_ipv6.ko 2023-11-28T17:26:16.382 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_mobility.ko 2023-11-28T17:26:16.382 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_routing.ko 2023-11-28T17:26:16.382 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_udp.ko 2023-11-28T17:26:16.382 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802 2023-11-28T17:26:16.382 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/garp.ko 2023-11-28T17:26:16.382 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/mrp.ko 2023-11-28T17:26:16.382 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/p8022.ko 2023-11-28T17:26:16.382 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/psnap.ko 2023-11-28T17:26:16.382 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/stp.ko 2023-11-28T17:26:16.382 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q 2023-11-28T17:26:16.382 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q/8021q.ko 2023-11-28T17:26:16.383 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p 2023-11-28T17:26:16.383 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet.ko 2023-11-28T17:26:16.383 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_fd.ko 2023-11-28T17:26:16.383 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_rdma.ko 2023-11-28T17:26:16.383 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_virtio.ko 2023-11-28T17:26:16.383 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv 2023-11-28T17:26:16.383 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv/batman-adv.ko 2023-11-28T17:26:16.383 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge 2023-11-28T17:26:16.383 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/br_netfilter.ko 2023-11-28T17:26:16.383 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/bridge.ko 2023-11-28T17:26:16.383 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter 2023-11-28T17:26:16.383 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-11-28T17:26:16.383 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_among.ko 2023-11-28T17:26:16.384 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arp.ko 2023-11-28T17:26:16.384 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-11-28T17:26:16.384 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-11-28T17:26:16.384 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip.ko 2023-11-28T17:26:16.384 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-11-28T17:26:16.384 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_limit.ko 2023-11-28T17:26:16.384 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_log.ko 2023-11-28T17:26:16.384 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark.ko 2023-11-28T17:26:16.384 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-11-28T17:26:16.384 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-11-28T17:26:16.384 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-11-28T17:26:16.384 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-11-28T17:26:16.385 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_snat.ko 2023-11-28T17:26:16.385 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_stp.ko 2023-11-28T17:26:16.385 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-11-28T17:26:16.385 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-11-28T17:26:16.385 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-11-28T17:26:16.385 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-11-28T17:26:16.385 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtables.ko 2023-11-28T17:26:16.385 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-11-28T17:26:16.385 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-11-28T17:26:16.385 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-11-28T17:26:16.385 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph 2023-11-28T17:26:16.385 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph/libceph.ko 2023-11-28T17:26:16.386 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core 2023-11-28T17:26:16.386 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/failover.ko 2023-11-28T17:26:16.386 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/pktgen.ko 2023-11-28T17:26:16.386 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp 2023-11-28T17:26:16.386 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp.ko 2023-11-28T17:26:16.386 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_diag.ko 2023-11-28T17:26:16.386 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv4.ko 2023-11-28T17:26:16.386 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv6.ko 2023-11-28T17:26:16.386 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver 2023-11-28T17:26:16.386 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver/dns_resolver.ko 2023-11-28T17:26:16.386 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa 2023-11-28T17:26:16.386 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/dsa_core.ko 2023-11-28T17:26:16.387 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_brcm.ko 2023-11-28T17:26:16.387 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_dsa.ko 2023-11-28T17:26:16.387 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_none.ko 2023-11-28T17:26:16.387 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_trailer.ko 2023-11-28T17:26:16.387 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154 2023-11-28T17:26:16.387 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan 2023-11-28T17:26:16.387 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-11-28T17:26:16.387 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154.ko 2023-11-28T17:26:16.387 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154_socket.ko 2023-11-28T17:26:16.387 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4 2023-11-28T17:26:16.387 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ah4.ko 2023-11-28T17:26:16.387 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/esp4.ko 2023-11-28T17:26:16.388 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/fou.ko 2023-11-28T17:26:16.388 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/gre.ko 2023-11-28T17:26:16.388 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/inet_diag.ko 2023-11-28T17:26:16.388 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_gre.ko 2023-11-28T17:26:16.388 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_tunnel.ko 2023-11-28T17:26:16.388 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_vti.ko 2023-11-28T17:26:16.388 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipcomp.ko 2023-11-28T17:26:16.388 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipip.ko 2023-11-28T17:26:16.388 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter 2023-11-28T17:26:16.388 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arp_tables.ko 2023-11-28T17:26:16.388 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-11-28T17:26:16.388 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-11-28T17:26:16.388 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-11-28T17:26:16.389 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-11-28T17:26:16.389 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-11-28T17:26:16.389 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-11-28T17:26:16.389 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-11-28T17:26:16.389 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-11-28T17:26:16.389 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-11-28T17:26:16.389 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_security.ko 2023-11-28T17:26:16.389 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-11-28T17:26:16.389 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-11-28T17:26:16.389 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-11-28T17:26:16.389 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-11-28T17:26:16.389 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-11-28T17:26:16.390 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-11-28T17:26:16.390 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-11-28T17:26:16.390 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-11-28T17:26:16.390 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-11-28T17:26:16.390 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-11-28T17:26:16.390 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_bic.ko 2023-11-28T17:26:16.390 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_dctcp.ko 2023-11-28T17:26:16.390 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_diag.ko 2023-11-28T17:26:16.390 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_highspeed.ko 2023-11-28T17:26:16.390 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_htcp.ko 2023-11-28T17:26:16.390 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_hybla.ko 2023-11-28T17:26:16.390 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_illinois.ko 2023-11-28T17:26:16.391 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_lp.ko 2023-11-28T17:26:16.391 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_scalable.ko 2023-11-28T17:26:16.391 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_vegas.ko 2023-11-28T17:26:16.391 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_veno.ko 2023-11-28T17:26:16.391 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_westwood.ko 2023-11-28T17:26:16.391 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_yeah.ko 2023-11-28T17:26:16.391 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tunnel4.ko 2023-11-28T17:26:16.391 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_diag.ko 2023-11-28T17:26:16.391 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_tunnel.ko 2023-11-28T17:26:16.391 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/xfrm4_tunnel.ko 2023-11-28T17:26:16.391 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6 2023-11-28T17:26:16.391 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ah6.ko 2023-11-28T17:26:16.392 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/esp6.ko 2023-11-28T17:26:16.392 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/fou6.ko 2023-11-28T17:26:16.392 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_tunnel.ko 2023-11-28T17:26:16.392 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-11-28T17:26:16.392 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_vti.ko 2023-11-28T17:26:16.392 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ipcomp6.ko 2023-11-28T17:26:16.392 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter 2023-11-28T17:26:16.392 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-11-28T17:26:16.392 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-11-28T17:26:16.392 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-11-28T17:26:16.392 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-11-28T17:26:16.392 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-11-28T17:26:16.393 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-11-28T17:26:16.393 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-11-28T17:26:16.393 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-11-28T17:26:16.393 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-11-28T17:26:16.393 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-11-28T17:26:16.393 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-11-28T17:26:16.393 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-11-28T17:26:16.393 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-11-28T17:26:16.393 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-11-28T17:26:16.393 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-11-28T17:26:16.393 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-11-28T17:26:16.393 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-11-28T17:26:16.393 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-11-28T17:26:16.394 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-11-28T17:26:16.394 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-11-28T17:26:16.394 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-11-28T17:26:16.394 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-11-28T17:26:16.394 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-11-28T17:26:16.394 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-11-28T17:26:16.394 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/sit.ko 2023-11-28T17:26:16.394 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/tunnel6.ko 2023-11-28T17:26:16.394 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/xfrm6_tunnel.ko 2023-11-28T17:26:16.394 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key 2023-11-28T17:26:16.394 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key/af_key.ko 2023-11-28T17:26:16.394 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp 2023-11-28T17:26:16.395 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_core.ko 2023-11-28T17:26:16.395 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_debugfs.ko 2023-11-28T17:26:16.395 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_eth.ko 2023-11-28T17:26:16.395 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip.ko 2023-11-28T17:26:16.395 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip6.ko 2023-11-28T17:26:16.395 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_netlink.ko 2023-11-28T17:26:16.395 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ppp.ko 2023-11-28T17:26:16.395 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc 2023-11-28T17:26:16.395 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc/llc.ko 2023-11-28T17:26:16.395 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154 2023-11-28T17:26:16.395 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154/mac802154.ko 2023-11-28T17:26:16.395 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls 2023-11-28T17:26:16.396 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls/mpls_gso.ko 2023-11-28T17:26:16.396 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter 2023-11-28T17:26:16.396 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset 2023-11-28T17:26:16.396 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set.ko 2023-11-28T17:26:16.396 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-11-28T17:26:16.396 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-11-28T17:26:16.396 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-11-28T17:26:16.396 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-11-28T17:26:16.396 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-11-28T17:26:16.396 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-11-28T17:26:16.396 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-11-28T17:26:16.396 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-11-28T17:26:16.396 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-11-28T17:26:16.397 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-11-28T17:26:16.397 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-11-28T17:26:16.397 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-11-28T17:26:16.397 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-11-28T17:26:16.397 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-11-28T17:26:16.397 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-11-28T17:26:16.397 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs 2023-11-28T17:26:16.397 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs.ko 2023-11-28T17:26:16.397 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-11-28T17:26:16.397 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-11-28T17:26:16.397 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-11-28T17:26:16.397 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-11-28T17:26:16.398 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-11-28T17:26:16.398 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-11-28T17:26:16.398 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-11-28T17:26:16.398 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-11-28T17:26:16.398 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-11-28T17:26:16.398 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-11-28T17:26:16.398 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-11-28T17:26:16.398 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-11-28T17:26:16.398 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-11-28T17:26:16.398 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conncount.ko 2023-11-28T17:26:16.398 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack.ko 2023-11-28T17:26:16.398 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-11-28T17:26:16.399 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-11-28T17:26:16.399 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-11-28T17:26:16.399 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_h323.ko 2023-11-28T17:26:16.399 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_irc.ko 2023-11-28T17:26:16.399 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-11-28T17:26:16.399 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-11-28T17:26:16.399 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-11-28T17:26:16.399 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sane.ko 2023-11-28T17:26:16.399 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sip.ko 2023-11-28T17:26:16.399 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-11-28T17:26:16.399 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-11-28T17:26:16.399 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_dup_netdev.ko 2023-11-28T17:26:16.399 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table.ko 2023-11-28T17:26:16.400 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table_inet.ko 2023-11-28T17:26:16.400 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_log_syslog.ko 2023-11-28T17:26:16.400 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat.ko 2023-11-28T17:26:16.400 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_amanda.ko 2023-11-28T17:26:16.400 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_ftp.ko 2023-11-28T17:26:16.400 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_irc.ko 2023-11-28T17:26:16.400 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_sip.ko 2023-11-28T17:26:16.400 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_tftp.ko 2023-11-28T17:26:16.400 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_synproxy_core.ko 2023-11-28T17:26:16.400 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_tables.ko 2023-11-28T17:26:16.400 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink.ko 2023-11-28T17:26:16.400 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_acct.ko 2023-11-28T17:26:16.401 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_log.ko 2023-11-28T17:26:16.401 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_osf.ko 2023-11-28T17:26:16.401 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_queue.ko 2023-11-28T17:26:16.401 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_chain_nat.ko 2023-11-28T17:26:16.401 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_compat.ko 2023-11-28T17:26:16.401 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_ct.ko 2023-11-28T17:26:16.401 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_dup_netdev.ko 2023-11-28T17:26:16.401 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fib.ko 2023-11-28T17:26:16.401 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fwd_netdev.ko 2023-11-28T17:26:16.401 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_hash.ko 2023-11-28T17:26:16.401 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_limit.ko 2023-11-28T17:26:16.401 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_log.ko 2023-11-28T17:26:16.402 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_masq.ko 2023-11-28T17:26:16.402 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_nat.ko 2023-11-28T17:26:16.402 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_numgen.ko 2023-11-28T17:26:16.402 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_queue.ko 2023-11-28T17:26:16.402 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_quota.ko 2023-11-28T17:26:16.402 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_redir.ko 2023-11-28T17:26:16.402 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject.ko 2023-11-28T17:26:16.402 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject_inet.ko 2023-11-28T17:26:16.402 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_socket.ko 2023-11-28T17:26:16.402 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_tunnel.ko 2023-11-28T17:26:16.402 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_AUDIT.ko 2023-11-28T17:26:16.402 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CHECKSUM.ko 2023-11-28T17:26:16.403 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CLASSIFY.ko 2023-11-28T17:26:16.403 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-11-28T17:26:16.403 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CT.ko 2023-11-28T17:26:16.403 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_DSCP.ko 2023-11-28T17:26:16.403 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HL.ko 2023-11-28T17:26:16.403 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HMARK.ko 2023-11-28T17:26:16.403 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_IDLETIMER.ko 2023-11-28T17:26:16.403 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LED.ko 2023-11-28T17:26:16.403 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LOG.ko 2023-11-28T17:26:16.403 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_MASQUERADE.ko 2023-11-28T17:26:16.403 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NETMAP.ko 2023-11-28T17:26:16.403 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFLOG.ko 2023-11-28T17:26:16.403 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFQUEUE.ko 2023-11-28T17:26:16.404 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_RATEEST.ko 2023-11-28T17:26:16.404 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_REDIRECT.ko 2023-11-28T17:26:16.404 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_SECMARK.ko 2023-11-28T17:26:16.404 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPMSS.ko 2023-11-28T17:26:16.404 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-11-28T17:26:16.404 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TEE.ko 2023-11-28T17:26:16.404 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TPROXY.ko 2023-11-28T17:26:16.404 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TRACE.ko 2023-11-28T17:26:16.404 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_addrtype.ko 2023-11-28T17:26:16.404 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_bpf.ko 2023-11-28T17:26:16.404 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cgroup.ko 2023-11-28T17:26:16.404 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cluster.ko 2023-11-28T17:26:16.405 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_comment.ko 2023-11-28T17:26:16.405 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connbytes.ko 2023-11-28T17:26:16.405 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlabel.ko 2023-11-28T17:26:16.405 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlimit.ko 2023-11-28T17:26:16.405 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connmark.ko 2023-11-28T17:26:16.405 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_conntrack.ko 2023-11-28T17:26:16.405 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cpu.ko 2023-11-28T17:26:16.405 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dccp.ko 2023-11-28T17:26:16.405 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_devgroup.ko 2023-11-28T17:26:16.405 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dscp.ko 2023-11-28T17:26:16.405 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ecn.ko 2023-11-28T17:26:16.405 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_esp.ko 2023-11-28T17:26:16.405 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hashlimit.ko 2023-11-28T17:26:16.406 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_helper.ko 2023-11-28T17:26:16.406 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hl.ko 2023-11-28T17:26:16.406 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipcomp.ko 2023-11-28T17:26:16.406 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_iprange.ko 2023-11-28T17:26:16.406 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipvs.ko 2023-11-28T17:26:16.406 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_l2tp.ko 2023-11-28T17:26:16.406 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_length.ko 2023-11-28T17:26:16.406 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_limit.ko 2023-11-28T17:26:16.406 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mac.ko 2023-11-28T17:26:16.406 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mark.ko 2023-11-28T17:26:16.406 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_multiport.ko 2023-11-28T17:26:16.406 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nat.ko 2023-11-28T17:26:16.407 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nfacct.ko 2023-11-28T17:26:16.407 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_osf.ko 2023-11-28T17:26:16.407 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_owner.ko 2023-11-28T17:26:16.407 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_physdev.ko 2023-11-28T17:26:16.407 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_pkttype.ko 2023-11-28T17:26:16.407 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_policy.ko 2023-11-28T17:26:16.407 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_quota.ko 2023-11-28T17:26:16.407 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_rateest.ko 2023-11-28T17:26:16.407 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_realm.ko 2023-11-28T17:26:16.407 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_recent.ko 2023-11-28T17:26:16.407 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_sctp.ko 2023-11-28T17:26:16.407 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_set.ko 2023-11-28T17:26:16.408 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_socket.ko 2023-11-28T17:26:16.408 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_state.ko 2023-11-28T17:26:16.408 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_statistic.ko 2023-11-28T17:26:16.408 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_string.ko 2023-11-28T17:26:16.408 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_tcpmss.ko 2023-11-28T17:26:16.408 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_time.ko 2023-11-28T17:26:16.408 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_u32.ko 2023-11-28T17:26:16.408 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink 2023-11-28T17:26:16.408 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink/netlink_diag.ko 2023-11-28T17:26:16.408 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh 2023-11-28T17:26:16.408 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh/nsh.ko 2023-11-28T17:26:16.408 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch 2023-11-28T17:26:16.408 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/openvswitch.ko 2023-11-28T17:26:16.409 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-geneve.ko 2023-11-28T17:26:16.409 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-gre.ko 2023-11-28T17:26:16.409 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-vxlan.ko 2023-11-28T17:26:16.409 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet 2023-11-28T17:26:16.409 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet/af_packet_diag.ko 2023-11-28T17:26:16.409 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds 2023-11-28T17:26:16.409 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds.ko 2023-11-28T17:26:16.409 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_rdma.ko 2023-11-28T17:26:16.409 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_tcp.ko 2023-11-28T17:26:16.409 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched 2023-11-28T17:26:16.409 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_csum.ko 2023-11-28T17:26:16.409 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_gact.ko 2023-11-28T17:26:16.410 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_ipt.ko 2023-11-28T17:26:16.410 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_mirred.ko 2023-11-28T17:26:16.410 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_nat.ko 2023-11-28T17:26:16.410 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_pedit.ko 2023-11-28T17:26:16.410 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_police.ko 2023-11-28T17:26:16.410 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_simple.ko 2023-11-28T17:26:16.410 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_skbedit.ko 2023-11-28T17:26:16.410 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_vlan.ko 2023-11-28T17:26:16.410 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_basic.ko 2023-11-28T17:26:16.410 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_bpf.ko 2023-11-28T17:26:16.410 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_flow.ko 2023-11-28T17:26:16.410 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_fw.ko 2023-11-28T17:26:16.411 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_route.ko 2023-11-28T17:26:16.411 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_u32.ko 2023-11-28T17:26:16.411 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_cmp.ko 2023-11-28T17:26:16.411 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_ipset.ko 2023-11-28T17:26:16.411 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_meta.ko 2023-11-28T17:26:16.411 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_nbyte.ko 2023-11-28T17:26:16.411 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_text.ko 2023-11-28T17:26:16.411 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_u32.ko 2023-11-28T17:26:16.411 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_choke.ko 2023-11-28T17:26:16.411 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_codel.ko 2023-11-28T17:26:16.411 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_drr.ko 2023-11-28T17:26:16.411 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_fq.ko 2023-11-28T17:26:16.411 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_gred.ko 2023-11-28T17:26:16.412 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hfsc.ko 2023-11-28T17:26:16.412 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hhf.ko 2023-11-28T17:26:16.412 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_htb.ko 2023-11-28T17:26:16.412 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_ingress.ko 2023-11-28T17:26:16.412 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio.ko 2023-11-28T17:26:16.412 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio_lib.ko 2023-11-28T17:26:16.412 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_multiq.ko 2023-11-28T17:26:16.412 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_netem.ko 2023-11-28T17:26:16.412 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_pie.ko 2023-11-28T17:26:16.412 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_plug.ko 2023-11-28T17:26:16.412 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_prio.ko 2023-11-28T17:26:16.412 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_qfq.ko 2023-11-28T17:26:16.413 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_red.ko 2023-11-28T17:26:16.413 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfb.ko 2023-11-28T17:26:16.413 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfq.ko 2023-11-28T17:26:16.413 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_tbf.ko 2023-11-28T17:26:16.413 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_teql.ko 2023-11-28T17:26:16.413 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp 2023-11-28T17:26:16.413 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp.ko 2023-11-28T17:26:16.413 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp_diag.ko 2023-11-28T17:26:16.413 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc 2023-11-28T17:26:16.413 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss 2023-11-28T17:26:16.413 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-11-28T17:26:16.413 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-11-28T17:26:16.414 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/sunrpc.ko 2023-11-28T17:26:16.414 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma 2023-11-28T17:26:16.414 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-11-28T17:26:16.414 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc 2023-11-28T17:26:16.414 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/diag.ko 2023-11-28T17:26:16.414 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/tipc.ko 2023-11-28T17:26:16.414 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix 2023-11-28T17:26:16.414 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix/unix_diag.ko 2023-11-28T17:26:16.414 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock 2023-11-28T17:26:16.414 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-11-28T17:26:16.414 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-11-28T17:26:16.414 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock.ko 2023-11-28T17:26:16.414 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_diag.ko 2023-11-28T17:26:16.415 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_loopback.ko 2023-11-28T17:26:16.415 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm 2023-11-28T17:26:16.415 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/xfrm_ipcomp.ko 2023-11-28T17:26:16.415 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security 2023-11-28T17:26:16.415 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys 2023-11-28T17:26:16.415 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys 2023-11-28T17:26:16.415 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-11-28T17:26:16.415 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys 2023-11-28T17:26:16.415 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys/trusted.ko 2023-11-28T17:26:16.415 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt 2023-11-28T17:26:16.415 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib 2023-11-28T17:26:16.415 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib/irqbypass.ko 2023-11-28T17:26:16.416 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias 2023-11-28T17:26:16.416 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias.bin 2023-11-28T17:26:16.416 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin 2023-11-28T17:26:16.416 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.alias.bin 2023-11-28T17:26:16.416 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.bin 2023-11-28T17:26:16.416 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.modinfo 2023-11-28T17:26:16.416 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep 2023-11-28T17:26:16.416 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep.bin 2023-11-28T17:26:16.416 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.devname 2023-11-28T17:26:16.416 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.order 2023-11-28T17:26:16.416 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.softdep 2023-11-28T17:26:16.416 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols 2023-11-28T17:26:16.416 INFO:teuthology.orchestra.run.smithi098.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols.bin 2023-11-28T17:26:16.417 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-g8247bbb5b470 2023-11-28T17:26:16.417 DEBUG:teuthology.orchestra.run.smithi098:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-11-28T17:26:16.439 INFO:teuthology.orchestra.run.smithi098.stdout:postinstall scriptlet (using /bin/sh): 2023-11-28T17:26:16.439 INFO:teuthology.orchestra.run.smithi098.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-g8247bbb5b470 -a -r /boot/System.map-6.6.0-g8247bbb5b470 ]; then 2023-11-28T17:26:16.439 INFO:teuthology.orchestra.run.smithi098.stdout:cp /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm 2023-11-28T17:26:16.439 INFO:teuthology.orchestra.run.smithi098.stdout:cp /boot/System.map-6.6.0-g8247bbb5b470 /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T17:26:16.439 INFO:teuthology.orchestra.run.smithi098.stdout:rm -f /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/System.map-6.6.0-g8247bbb5b470 2023-11-28T17:26:16.440 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/installkernel 6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T17:26:16.440 INFO:teuthology.orchestra.run.smithi098.stdout:rm -f /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T17:26:16.440 INFO:teuthology.orchestra.run.smithi098.stdout:fi 2023-11-28T17:26:16.440 INFO:teuthology.orchestra.run.smithi098.stdout:preuninstall scriptlet (using /bin/sh): 2023-11-28T17:26:16.440 INFO:teuthology.orchestra.run.smithi098.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-11-28T17:26:16.440 INFO:teuthology.orchestra.run.smithi098.stdout:new-kernel-pkg --remove 6.6.0-g8247bbb5b470 --rminitrd --initrdfile=/boot/initramfs-6.6.0-g8247bbb5b470.img 2023-11-28T17:26:16.440 INFO:teuthology.orchestra.run.smithi098.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-11-28T17:26:16.440 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-install remove 6.6.0-g8247bbb5b470 2023-11-28T17:26:16.440 INFO:teuthology.orchestra.run.smithi098.stdout:fi 2023-11-28T17:26:16.440 INFO:teuthology.orchestra.run.smithi098.stdout:postuninstall scriptlet (using /bin/sh): 2023-11-28T17:26:16.440 INFO:teuthology.orchestra.run.smithi098.stdout:if [ -x /sbin/update-bootloader ]; then 2023-11-28T17:26:16.440 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/update-bootloader --remove 6.6.0-g8247bbb5b470 2023-11-28T17:26:16.440 INFO:teuthology.orchestra.run.smithi098.stdout:fi 2023-11-28T17:26:16.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2023-11-28T17:26:16.520 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2023-11-28T17:26:16.521 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2023-11-28T17:26:16.521 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2023-11-28T17:26:16.521 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 68.el9 2023-11-28T17:26:16.521 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2023-11-28T17:26:16.521 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-28T17:26:16.521 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2023-11-28T17:26:16.521 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8264254 2023-11-28T17:26:16.521 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2023-11-28T17:26:16.521 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-28T17:26:16.521 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-28T17:26:16.521 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-28T17:26:16.522 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-28T17:26:16.522 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2023-11-28T17:26:16.522 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2023-11-28T17:26:16.522 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-28T17:26:16.522 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2023-11-28T17:26:16.522 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2023-11-28T17:26:16.522 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:26:16.522 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-28T17:26:16.522 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-28T17:26:16.522 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-28T17:26:16.522 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2023-11-28T17:26:16.522 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:26:16.523 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2023-11-28T17:26:16.527 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-28T17:26:16.527 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 6.6.0-g8247bbb5b470 2023-11-28T17:26:16.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-28T17:26:17.104 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2023-11-28T17:26:18.450 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-28T17:26:18.493 INFO:teuthology.orchestra.run.smithi098.stderr:done 2023-11-28T17:26:18.495 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2023-11-28T17:26:18.511 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.2IiVuBDl4f 2023-11-28T17:26:18.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2IiVuBDl4f 2023-11-28T17:26:18.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.2IiVuBDl4f 2023-11-28T17:26:18.711 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.2IiVuBDl4f is 6KB 2023-11-28T17:26:18.759 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.2IiVuBDl4f 2023-11-28T17:26:18.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2023-11-28T17:26:18.837 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-28T17:26:18.838 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-28T17:26:18.838 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-28T17:26:18.838 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470.conf 2023-11-28T17:26:18.839 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470 2023-11-28T17:26:19.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2023-11-28T17:26:25.042 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-11-28T17:26:25.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2023-11-28T17:26:25.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T17:26:43.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2023-11-28T17:26:49.026 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-28T17:26:49.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-11-28T17:26:49.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T17:26:52.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2023-11-28T17:26:52.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T17:26:55.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2023-11-28T17:27:07.362 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2023-11-28T17:27:07.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2023-11-28T17:27:07.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T17:27:10.690 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2023-11-28T17:27:16.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-11-28T17:27:16.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T17:27:19.426 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2023-11-28T17:27:25.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2023-11-28T17:27:25.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T17:27:31.430 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-11-28T17:27:31.431 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T17:27:44.233 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2023-11-28T17:28:02.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2023-11-28T17:28:02.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T17:28:02.679 DEBUG:teuthology.orchestra.run.smithi113:> true 2023-11-28T17:28:03.302 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi113.front.sepia.ceph.com' 2023-11-28T17:28:03.302 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.6.0-g8247bbb5b470"... 2023-11-28T17:28:03.302 DEBUG:teuthology.orchestra.run.smithi113:> uname -r 2023-11-28T17:28:03.303 DEBUG:teuthology.orchestra.run.smithi098:> true 2023-11-28T17:28:03.343 INFO:teuthology.orchestra.run.smithi113.stdout:6.6.0-g8247bbb5b470 2023-11-28T17:28:03.344 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-g8247bbb5b470 vs 6.6.0-g8247bbb5b470 2023-11-28T17:28:03.344 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-28T17:28:03.344 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-28T17:28:03.923 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-11-28T17:28:03.924 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.6.0-g8247bbb5b470"... 2023-11-28T17:28:03.924 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2023-11-28T17:28:03.972 INFO:teuthology.orchestra.run.smithi098.stdout:6.6.0-g8247bbb5b470 2023-11-28T17:28:03.973 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-g8247bbb5b470 vs 6.6.0-g8247bbb5b470 2023-11-28T17:28:03.973 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-28T17:28:03.974 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-28T17:28:04.345 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-11-28T17:28:04.345 DEBUG:teuthology.orchestra.run.smithi113:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-28T17:28:04.625 INFO:teuthology.orchestra.run.smithi113.stdout:ttyS1 2023-11-28T17:28:04.672 DEBUG:teuthology.parallel:result is None 2023-11-28T17:28:04.975 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-28T17:28:04.975 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-28T17:28:05.095 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2023-11-28T17:28:05.140 DEBUG:teuthology.parallel:result is None 2023-11-28T17:28:05.141 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-28T17:28:05.148 INFO:teuthology.task.internal:Creating test directory... 2023-11-28T17:28:05.149 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-28T17:28:05.153 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-28T17:28:05.207 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-28T17:28:05.215 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-28T17:28:05.247 INFO:teuthology.task.internal:Creating archive directory... 2023-11-28T17:28:05.247 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-28T17:28:05.256 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-28T17:28:05.318 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-28T17:28:05.326 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-28T17:28:05.326 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-28T17:28:05.331 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-28T17:28:05.412 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T17:28:05.434 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T17:28:05.450 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T17:28:05.473 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T17:28:05.477 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-28T17:28:05.484 INFO:teuthology.task.internal:Configuring sudo... 2023-11-28T17:28:05.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-28T17:28:05.493 DEBUG:teuthology.orchestra.run.smithi113:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-28T17:28:05.577 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-28T17:28:05.587 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-28T17:28:05.587 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-28T17:28:05.593 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-28T17:28:05.635 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T17:28:05.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T17:28:05.803 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T17:28:05.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T17:28:05.950 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:28:05.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-28T17:28:06.023 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T17:28:06.091 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T17:28:06.189 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T17:28:06.260 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T17:28:06.343 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:28:06.343 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-28T17:28:06.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2023-11-28T17:28:06.421 DEBUG:teuthology.orchestra.run.smithi113:> sudo service rsyslog restart 2023-11-28T17:28:06.525 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T17:28:06.574 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T17:28:06.941 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-28T17:28:06.949 INFO:teuthology.task.internal:Starting timer... 2023-11-28T17:28:06.949 INFO:teuthology.run_tasks:Running task pcp... 2023-11-28T17:28:07.001 INFO:teuthology.run_tasks:Running task selinux... 2023-11-28T17:28:07.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2023-11-28T17:28:07.167 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2023-11-28T17:28:07.170 DEBUG:teuthology.orchestra.run.smithi113:> sudo service auditd rotate 2023-11-28T17:28:07.275 INFO:teuthology.orchestra.run.smithi113.stdout:Rotating logs: 2023-11-28T17:28:07.278 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-28T17:28:07.278 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2023-11-28T17:28:07.327 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2023-11-28T17:28:07.328 DEBUG:teuthology.orchestra.run.smithi113:> /usr/sbin/getenforce 2023-11-28T17:28:07.375 INFO:teuthology.orchestra.run.smithi113.stdout:Permissive 2023-11-28T17:28:07.375 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi113.front.sepia.ceph.com': 'permissive'} 2023-11-28T17:28:07.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T17:28:07.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:28:07.451 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T17:28:07.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:28:07.528 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-28T17:28:07.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2023-11-28T17:28:07.615 DEBUG:teuthology.orchestra.run.smithi113:> sudo /usr/sbin/setenforce permissive 2023-11-28T17:28:07.695 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-28T17:28:07.705 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-11-28T17:28:07.710 INFO:teuthology.repo_utils:Fetching main from origin 2023-11-28T17:28:07.819 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-28T17:28:07.837 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-28T17:28:07.838 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi113.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-28T17:38:06.748 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi113.front.sepia.ceph.com')] 2023-11-28T17:38:06.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-11-28T17:38:06.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T17:38:06.892 DEBUG:teuthology.orchestra.run.smithi098:> true 2023-11-28T17:38:07.046 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-11-28T17:38:07.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2023-11-28T17:38:07.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T17:38:07.185 DEBUG:teuthology.orchestra.run.smithi113:> true 2023-11-28T17:38:07.332 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi113.front.sepia.ceph.com' 2023-11-28T17:38:07.332 INFO:teuthology.run_tasks:Running task clock... 2023-11-28T17:38:07.347 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-28T17:38:07.348 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-28T17:38:07.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T17:38:07.358 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-28T17:38:07.358 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T17:38:07.458 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-28T17:38:07.462 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-28T17:38:07.509 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-28T17:38:07.515 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-28T17:38:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2023-11-28T17:38:07.605 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: ntpd: command not found 2023-11-28T17:38:07.644 INFO:teuthology.orchestra.run.smithi113.stdout:506 Cannot talk to daemon 2023-11-28T17:38:07.689 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2023-11-28T17:38:07.691 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-28T17:38:07.696 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-28T17:38:07.740 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-28T17:38:07.750 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-28T17:38:07.864 INFO:teuthology.orchestra.run.smithi113.stderr:bash: line 1: ntpq: command not found 2023-11-28T17:38:07.870 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2023-11-28T17:38:07.872 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T17:38:07.872 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2023-11-28T17:38:07.872 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:38:07.872 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:38:07.873 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:38:07.873 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:38:07.876 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T17:38:07.876 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2023-11-28T17:38:07.876 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:38:07.876 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:38:07.876 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:38:07.876 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:38:07.877 INFO:teuthology.run_tasks:Running task install... 2023-11-28T17:38:07.889 DEBUG:teuthology.task.install:project ceph 2023-11-28T17:38:07.889 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}} 2023-11-28T17:38:07.889 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'} 2023-11-28T17:38:07.889 INFO:teuthology.task.install:Using flavor: default 2023-11-28T17:38:07.906 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-28T17:38:07.906 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-11-28T17:38:07.907 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'tag': None, 'wait_for_package': False} 2023-11-28T17:38:07.907 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T17:38:07.913 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'tag': None, 'wait_for_package': False} 2023-11-28T17:38:07.913 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T17:38:08.064 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/ 2023-11-28T17:38:08.064 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7531.g47c1c547 2023-11-28T17:38:08.072 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/ 2023-11-28T17:38:08.072 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7531.g47c1c547 2023-11-28T17:38:08.189 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-28T17:38:08.190 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:38:08.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-28T17:38:08.193 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-28T17:38:08.194 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:38:08.194 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-28T17:38:08.298 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-28T17:38:08.298 DEBUG:teuthology.orchestra.run.smithi113:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/47c1c547372ed5bff7984647dc200bac0e019caf/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-28T17:38:08.301 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-28T17:38:08.301 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/47c1c547372ed5bff7984647dc200bac0e019caf/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-28T17:38:08.440 DEBUG:teuthology.orchestra.run.smithi113:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-28T17:38:08.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-28T17:38:08.572 DEBUG:teuthology.orchestra.run.smithi113:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-28T17:38:08.579 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-28T17:38:08.666 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2023-11-28T17:38:08.666 INFO:teuthology.orchestra.run.smithi113.stdout:check_obsoletes = 1 2023-11-28T17:38:08.669 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum clean all 2023-11-28T17:38:08.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2023-11-28T17:38:09.021 INFO:teuthology.orchestra.run.smithi113.stdout:73 files removed 2023-11-28T17:38:09.030 INFO:teuthology.orchestra.run.smithi098.stdout:73 files removed 2023-11-28T17:38:09.063 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-28T17:38:09.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-28T17:38:10.073 INFO:teuthology.orchestra.run.smithi113.stdout:ceph packages for x86_64 158 kB/s | 78 kB 00:00 2023-11-28T17:38:10.156 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 135 kB/s | 78 kB 00:00 2023-11-28T17:38:10.681 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 22 kB/s | 9.5 kB 00:00 2023-11-28T17:38:10.699 INFO:teuthology.orchestra.run.smithi113.stdout:ceph noarch packages 18 kB/s | 9.5 kB 00:00 2023-11-28T17:38:11.183 INFO:teuthology.orchestra.run.smithi113.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2023-11-28T17:38:11.223 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2023-11-28T17:38:12.227 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 6.9 MB/s | 6.4 MB 00:00 2023-11-28T17:38:12.344 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS-9 - AppStream 5.9 MB/s | 6.4 MB 00:01 2023-11-28T17:38:14.012 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2023-11-28T17:38:21.119 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 7.9 MB 00:00 2023-11-28T17:38:21.119 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - AppStream 10 MB/s | 18 MB 00:01 2023-11-28T17:38:21.119 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2023-11-28T17:38:21.119 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 7.9 MB 00:00 2023-11-28T17:38:21.119 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 24 MB/s | 18 MB 00:00 2023-11-28T17:38:22.361 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 5.8 MB 00:00 2023-11-28T17:38:24.615 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 5.8 MB 00:03 2023-11-28T17:38:25.013 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 15 kB 00:01 2023-11-28T17:38:26.249 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 15 kB 00:00 2023-11-28T17:38:26.320 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 18 MB/s | 20 MB 00:01 2023-11-28T17:38:30.998 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 20 MB 00:04 2023-11-28T17:38:31.841 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-11-28T17:38:34.707 INFO:teuthology.orchestra.run.smithi113.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-28T17:38:34.709 INFO:teuthology.orchestra.run.smithi113.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-28T17:38:34.730 INFO:teuthology.orchestra.run.smithi113.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-11-28T17:38:34.731 INFO:teuthology.orchestra.run.smithi113.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-11-28T17:38:34.732 INFO:teuthology.orchestra.run.smithi113.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-11-28T17:38:34.734 INFO:teuthology.orchestra.run.smithi113.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-11-28T17:38:34.736 INFO:teuthology.orchestra.run.smithi113.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-11-28T17:38:34.740 INFO:teuthology.orchestra.run.smithi113.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-11-28T17:38:34.741 INFO:teuthology.orchestra.run.smithi113.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-11-28T17:38:34.820 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2023-11-28T17:38:34.846 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2023-11-28T17:38:34.846 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2023-11-28T17:38:34.846 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2023-11-28T17:38:34.846 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2023-11-28T17:38:34.847 INFO:teuthology.orchestra.run.smithi113.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-11-28T17:38:34.847 INFO:teuthology.orchestra.run.smithi113.stdout: ceph x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 6.6 k 2023-11-28T17:38:34.847 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 5.2 M 2023-11-28T17:38:34.847 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 859 k 2023-11-28T17:38:34.847 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 141 k 2023-11-28T17:38:34.847 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 1.5 M 2023-11-28T17:38:34.847 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 133 k 2023-11-28T17:38:34.848 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 1.8 M 2023-11-28T17:38:34.848 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 7.4 M 2023-11-28T17:38:34.848 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 50 k 2023-11-28T17:38:34.848 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 8.7 M 2023-11-28T17:38:34.848 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 41 M 2023-11-28T17:38:34.848 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-volume noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 262 k 2023-11-28T17:38:34.848 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 773 k 2023-11-28T17:38:34.848 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-mirror x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 217 k 2023-11-28T17:38:34.848 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-top noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 18 k 2023-11-28T17:38:34.849 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-devel x86_64 0.190-1.el9 CentOS-AppStream 21 k 2023-11-28T17:38:34.849 INFO:teuthology.orchestra.run.smithi113.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-11-28T17:38:34.849 INFO:teuthology.orchestra.run.smithi113.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-11-28T17:38:34.849 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 31 k 2023-11-28T17:38:34.849 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 719 k 2023-11-28T17:38:34.849 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 126 k 2023-11-28T17:38:34.849 INFO:teuthology.orchestra.run.smithi113.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-11-28T17:38:34.849 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2023-11-28T17:38:34.850 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-devel x86_64 1:3.0.7-25.el9 appstream 4.1 M 2023-11-28T17:38:34.850 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-11-28T17:38:34.850 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-11-28T17:38:34.850 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-11-28T17:38:34.850 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 162 k 2023-11-28T17:38:34.850 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 321 k 2023-11-28T17:38:34.850 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 298 k 2023-11-28T17:38:34.850 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 99 k 2023-11-28T17:38:34.851 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 85 k 2023-11-28T17:38:34.851 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 3.0 M 2023-11-28T17:38:34.851 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 170 k 2023-11-28T17:38:34.851 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 CentOS-AppStream 85 k 2023-11-28T17:38:34.851 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrading: 2023-11-28T17:38:34.851 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager x86_64 1:1.45.6-1.el9 CentOS-BaseOS 2.2 M 2023-11-28T17:38:34.851 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-libnm x86_64 1:1.45.6-1.el9 CentOS-BaseOS 1.8 M 2023-11-28T17:38:34.852 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-team x86_64 1:1.45.6-1.el9 CentOS-BaseOS 42 k 2023-11-28T17:38:34.852 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-tui x86_64 1:1.45.6-1.el9 CentOS-BaseOS 245 k 2023-11-28T17:38:34.852 INFO:teuthology.orchestra.run.smithi113.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-11-28T17:38:34.852 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-debuginfod-client x86_64 0.190-1.el9 CentOS-BaseOS 36 k 2023-11-28T17:38:34.852 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf x86_64 0.190-1.el9 CentOS-BaseOS 192 k 2023-11-28T17:38:34.852 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libs x86_64 0.190-1.el9 CentOS-BaseOS 255 k 2023-11-28T17:38:34.852 INFO:teuthology.orchestra.run.smithi113.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-11-28T17:38:34.852 INFO:teuthology.orchestra.run.smithi113.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-11-28T17:38:34.853 INFO:teuthology.orchestra.run.smithi113.stdout: librados2 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 3.3 M 2023-11-28T17:38:34.853 INFO:teuthology.orchestra.run.smithi113.stdout: librbd1 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 3.0 M 2023-11-28T17:38:34.853 INFO:teuthology.orchestra.run.smithi113.stdout: openssl x86_64 1:3.0.7-25.el9 baseos 1.2 M 2023-11-28T17:38:34.853 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-libs x86_64 1:3.0.7-25.el9 baseos 2.2 M 2023-11-28T17:38:34.853 INFO:teuthology.orchestra.run.smithi113.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-11-28T17:38:34.853 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2023-11-28T17:38:34.853 INFO:teuthology.orchestra.run.smithi113.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2023-11-28T17:38:34.853 INFO:teuthology.orchestra.run.smithi113.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-28T17:38:34.854 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 19 M 2023-11-28T17:38:34.854 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 28 k 2023-11-28T17:38:34.854 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 2.1 M 2023-11-28T17:38:34.854 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 243 k 2023-11-28T17:38:34.855 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 4.4 M 2023-11-28T17:38:34.855 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 18 M 2023-11-28T17:38:34.855 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 15 k 2023-11-28T17:38:34.855 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 24 k 2023-11-28T17:38:34.855 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-28T17:38:34.856 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-28T17:38:34.856 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-28T17:38:34.856 INFO:teuthology.orchestra.run.smithi113.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-28T17:38:34.856 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-28T17:38:34.856 INFO:teuthology.orchestra.run.smithi113.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-28T17:38:34.856 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-28T17:38:34.857 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-28T17:38:34.857 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 163 k 2023-11-28T17:38:34.857 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-28T17:38:34.857 INFO:teuthology.orchestra.run.smithi113.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-28T17:38:34.857 INFO:teuthology.orchestra.run.smithi113.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-28T17:38:34.857 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-28T17:38:34.858 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-11-28T17:38:34.858 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 476 k 2023-11-28T17:38:34.858 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-28T17:38:34.858 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 4.8 M 2023-11-28T17:38:34.858 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-28T17:38:34.859 INFO:teuthology.orchestra.run.smithi113.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-11-28T17:38:34.859 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-28T17:38:34.859 INFO:teuthology.orchestra.run.smithi113.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-11-28T17:38:34.859 INFO:teuthology.orchestra.run.smithi113.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-11-28T17:38:34.859 INFO:teuthology.orchestra.run.smithi113.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-11-28T17:38:34.860 INFO:teuthology.orchestra.run.smithi113.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-28T17:38:34.860 INFO:teuthology.orchestra.run.smithi113.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-28T17:38:34.860 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-28T17:38:34.860 INFO:teuthology.orchestra.run.smithi113.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-28T17:38:34.860 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-11-28T17:38:34.860 INFO:teuthology.orchestra.run.smithi113.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-28T17:38:34.861 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-28T17:38:34.861 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-28T17:38:34.861 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-28T17:38:34.861 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 45 k 2023-11-28T17:38:34.861 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 129 k 2023-11-28T17:38:34.862 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-28T17:38:34.862 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-28T17:38:34.862 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-28T17:38:34.862 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-28T17:38:34.863 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-11-28T17:38:34.863 INFO:teuthology.orchestra.run.smithi113.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-11-28T17:38:34.863 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-11-28T17:38:34.863 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-28T17:38:34.863 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-28T17:38:34.864 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-28T17:38:34.864 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-28T17:38:34.864 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-28T17:38:34.864 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-28T17:38:34.864 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-28T17:38:34.864 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-28T17:38:34.864 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-28T17:38:34.865 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-28T17:38:34.865 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-28T17:38:34.865 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-28T17:38:34.865 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-28T17:38:34.865 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-28T17:38:34.866 INFO:teuthology.orchestra.run.smithi113.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-28T17:38:34.866 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-28T17:38:34.866 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-28T17:38:34.866 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-28T17:38:34.866 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-28T17:38:34.866 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-28T17:38:34.867 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-28T17:38:34.867 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-28T17:38:34.867 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-28T17:38:34.867 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-28T17:38:34.868 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-28T17:38:34.868 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-28T17:38:34.868 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-11-28T17:38:34.868 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-28T17:38:34.868 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-28T17:38:34.869 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-28T17:38:34.869 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-28T17:38:34.869 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-28T17:38:34.869 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-28T17:38:34.869 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-28T17:38:34.869 INFO:teuthology.orchestra.run.smithi113.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-28T17:38:34.870 INFO:teuthology.orchestra.run.smithi113.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-28T17:38:34.870 INFO:teuthology.orchestra.run.smithi113.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-28T17:38:34.870 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-28T17:38:34.870 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-28T17:38:34.870 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-28T17:38:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-28T17:38:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-28T17:38:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-28T17:38:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-28T17:38:34.871 INFO:teuthology.orchestra.run.smithi113.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-11-28T17:38:34.872 INFO:teuthology.orchestra.run.smithi113.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-28T17:38:34.872 INFO:teuthology.orchestra.run.smithi113.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2023-11-28T17:38:34.872 INFO:teuthology.orchestra.run.smithi113.stdout:Installing weak dependencies: 2023-11-28T17:38:34.872 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-28T17:38:34.872 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:38:34.872 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2023-11-28T17:38:34.872 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2023-11-28T17:38:34.872 INFO:teuthology.orchestra.run.smithi113.stdout:Install 135 Packages 2023-11-28T17:38:34.872 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrade 15 Packages 2023-11-28T17:38:34.872 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:38:34.873 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 205 M 2023-11-28T17:38:34.873 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2023-11-28T17:38:36.134 INFO:teuthology.orchestra.run.smithi113.stdout:(1/150): ceph-18.0.0-7531.g47c1c547.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2023-11-28T17:38:36.542 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2023-11-28T17:38:36.727 INFO:teuthology.orchestra.run.smithi113.stdout:(2/150): ceph-base-18.0.0-7531.g47c1c547.el9.x8 6.0 MB/s | 5.2 MB 00:00 2023-11-28T17:38:36.768 INFO:teuthology.orchestra.run.smithi113.stdout:(3/150): ceph-fuse-18.0.0-7531.g47c1c547.el9.x8 1.3 MB/s | 859 kB 00:00 2023-11-28T17:38:36.801 INFO:teuthology.orchestra.run.smithi113.stdout:(4/150): ceph-immutable-object-cache-18.0.0-753 1.9 MB/s | 141 kB 00:00 2023-11-28T17:38:37.052 INFO:teuthology.orchestra.run.smithi113.stdout:(5/150): ceph-mgr-18.0.0-7531.g47c1c547.el9.x86 5.9 MB/s | 1.5 MB 00:00 2023-11-28T17:38:37.510 INFO:teuthology.orchestra.run.smithi113.stdout:(6/150): ceph-mon-18.0.0-7531.g47c1c547.el9.x86 9.6 MB/s | 4.4 MB 00:00 2023-11-28T17:38:37.569 INFO:teuthology.orchestra.run.smithi113.stdout:(7/150): ceph-mds-18.0.0-7531.g47c1c547.el9.x86 2.6 MB/s | 2.1 MB 00:00 2023-11-28T17:38:38.336 INFO:teuthology.orchestra.run.smithi113.stdout:(8/150): ceph-common-18.0.0-7531.g47c1c547.el9. 7.7 MB/s | 19 MB 00:02 2023-11-28T17:38:38.386 INFO:teuthology.orchestra.run.smithi113.stdout:(9/150): ceph-selinux-18.0.0-7531.g47c1c547.el9 493 kB/s | 24 kB 00:00 2023-11-28T17:38:38.562 INFO:teuthology.orchestra.run.smithi113.stdout:(10/150): ceph-radosgw-18.0.0-7531.g47c1c547.el 8.8 MB/s | 8.7 MB 00:00 2023-11-28T17:38:38.653 INFO:teuthology.orchestra.run.smithi113.stdout:(11/150): cephfs-mirror-18.0.0-7531.g47c1c547.e 2.3 MB/s | 217 kB 00:00 2023-11-28T17:38:38.737 INFO:teuthology.orchestra.run.smithi113.stdout:(12/150): libcephfs-devel-18.0.0-7531.g47c1c547 379 kB/s | 31 kB 00:00 2023-11-28T17:38:38.921 INFO:teuthology.orchestra.run.smithi113.stdout:(13/150): libcephfs2-18.0.0-7531.g47c1c547.el9. 3.9 MB/s | 719 kB 00:00 2023-11-28T17:38:39.012 INFO:teuthology.orchestra.run.smithi113.stdout:(14/150): libcephsqlite-18.0.0-7531.g47c1c547.e 1.8 MB/s | 163 kB 00:00 2023-11-28T17:38:39.104 INFO:teuthology.orchestra.run.smithi113.stdout:(15/150): librados-devel-18.0.0-7531.g47c1c547. 1.4 MB/s | 126 kB 00:00 2023-11-28T17:38:39.238 INFO:teuthology.orchestra.run.smithi113.stdout:(16/150): libradosstriper1-18.0.0-7531.g47c1c54 3.5 MB/s | 476 kB 00:00 2023-11-28T17:38:39.364 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-28T17:38:39.366 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-28T17:38:39.384 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-11-28T17:38:39.386 INFO:teuthology.orchestra.run.smithi098.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-11-28T17:38:39.387 INFO:teuthology.orchestra.run.smithi098.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-11-28T17:38:39.388 INFO:teuthology.orchestra.run.smithi098.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-11-28T17:38:39.391 INFO:teuthology.orchestra.run.smithi098.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-11-28T17:38:39.394 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-11-28T17:38:39.396 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-11-28T17:38:39.474 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2023-11-28T17:38:39.496 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2023-11-28T17:38:39.496 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2023-11-28T17:38:39.496 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2023-11-28T17:38:39.496 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2023-11-28T17:38:39.496 INFO:teuthology.orchestra.run.smithi098.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-11-28T17:38:39.496 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 6.6 k 2023-11-28T17:38:39.496 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 5.2 M 2023-11-28T17:38:39.496 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 859 k 2023-11-28T17:38:39.496 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 141 k 2023-11-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 1.5 M 2023-11-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 133 k 2023-11-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 1.8 M 2023-11-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 7.4 M 2023-11-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 50 k 2023-11-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 8.7 M 2023-11-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 41 M 2023-11-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 262 k 2023-11-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 773 k 2023-11-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 217 k 2023-11-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 18 k 2023-11-28T17:38:39.497 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel x86_64 0.190-1.el9 CentOS-AppStream 21 k 2023-11-28T17:38:39.498 INFO:teuthology.orchestra.run.smithi098.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-11-28T17:38:39.498 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-11-28T17:38:39.498 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 31 k 2023-11-28T17:38:39.498 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 719 k 2023-11-28T17:38:39.498 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 126 k 2023-11-28T17:38:39.498 INFO:teuthology.orchestra.run.smithi098.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-11-28T17:38:39.498 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2023-11-28T17:38:39.498 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel x86_64 1:3.0.7-25.el9 appstream 4.1 M 2023-11-28T17:38:39.498 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-11-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-11-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-11-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 162 k 2023-11-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 321 k 2023-11-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 298 k 2023-11-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 99 k 2023-11-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 85 k 2023-11-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 3.0 M 2023-11-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 170 k 2023-11-28T17:38:39.499 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 CentOS-AppStream 85 k 2023-11-28T17:38:39.500 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2023-11-28T17:38:39.500 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager x86_64 1:1.45.6-1.el9 CentOS-BaseOS 2.2 M 2023-11-28T17:38:39.500 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm x86_64 1:1.45.6-1.el9 CentOS-BaseOS 1.8 M 2023-11-28T17:38:39.500 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team x86_64 1:1.45.6-1.el9 CentOS-BaseOS 42 k 2023-11-28T17:38:39.500 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui x86_64 1:1.45.6-1.el9 CentOS-BaseOS 245 k 2023-11-28T17:38:39.500 INFO:teuthology.orchestra.run.smithi098.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-11-28T17:38:39.500 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client x86_64 0.190-1.el9 CentOS-BaseOS 36 k 2023-11-28T17:38:39.500 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf x86_64 0.190-1.el9 CentOS-BaseOS 192 k 2023-11-28T17:38:39.501 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs x86_64 0.190-1.el9 CentOS-BaseOS 255 k 2023-11-28T17:38:39.501 INFO:teuthology.orchestra.run.smithi098.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-11-28T17:38:39.501 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-11-28T17:38:39.501 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 3.3 M 2023-11-28T17:38:39.501 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 3.0 M 2023-11-28T17:38:39.501 INFO:teuthology.orchestra.run.smithi098.stdout: openssl x86_64 1:3.0.7-25.el9 baseos 1.2 M 2023-11-28T17:38:39.502 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs x86_64 1:3.0.7-25.el9 baseos 2.2 M 2023-11-28T17:38:39.502 INFO:teuthology.orchestra.run.smithi098.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-11-28T17:38:39.502 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2023-11-28T17:38:39.502 INFO:teuthology.orchestra.run.smithi098.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2023-11-28T17:38:39.502 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-28T17:38:39.503 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 19 M 2023-11-28T17:38:39.503 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 28 k 2023-11-28T17:38:39.503 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 2.1 M 2023-11-28T17:38:39.503 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 243 k 2023-11-28T17:38:39.503 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 4.4 M 2023-11-28T17:38:39.503 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 18 M 2023-11-28T17:38:39.503 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 15 k 2023-11-28T17:38:39.503 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 24 k 2023-11-28T17:38:39.504 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-28T17:38:39.504 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-28T17:38:39.504 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-28T17:38:39.504 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-28T17:38:39.504 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-28T17:38:39.504 INFO:teuthology.orchestra.run.smithi098.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-28T17:38:39.504 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-28T17:38:39.505 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-28T17:38:39.505 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 163 k 2023-11-28T17:38:39.505 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-28T17:38:39.505 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-28T17:38:39.505 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-28T17:38:39.505 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-28T17:38:39.505 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-11-28T17:38:39.505 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 476 k 2023-11-28T17:38:39.506 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-28T17:38:39.506 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 4.8 M 2023-11-28T17:38:39.506 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-28T17:38:39.506 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-11-28T17:38:39.506 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-28T17:38:39.506 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-11-28T17:38:39.506 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-11-28T17:38:39.507 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-11-28T17:38:39.507 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-28T17:38:39.507 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-28T17:38:39.507 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-28T17:38:39.507 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-28T17:38:39.507 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-11-28T17:38:39.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-28T17:38:39.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-28T17:38:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-28T17:38:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-28T17:38:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 45 k 2023-11-28T17:38:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 129 k 2023-11-28T17:38:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-28T17:38:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-28T17:38:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-28T17:38:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-28T17:38:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-11-28T17:38:39.508 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-11-28T17:38:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-11-28T17:38:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-28T17:38:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-28T17:38:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-28T17:38:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-28T17:38:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-28T17:38:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-28T17:38:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-28T17:38:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-28T17:38:39.509 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-28T17:38:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-28T17:38:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-28T17:38:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-28T17:38:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-28T17:38:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-28T17:38:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-28T17:38:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-28T17:38:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-28T17:38:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-28T17:38:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-28T17:38:39.510 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-28T17:38:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-28T17:38:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-28T17:38:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-28T17:38:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-28T17:38:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-28T17:38:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-28T17:38:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-11-28T17:38:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-28T17:38:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-28T17:38:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-28T17:38:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-28T17:38:39.511 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-28T17:38:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-28T17:38:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-28T17:38:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-28T17:38:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-28T17:38:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-28T17:38:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-28T17:38:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-28T17:38:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-28T17:38:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-28T17:38:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-28T17:38:39.512 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-28T17:38:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-28T17:38:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-11-28T17:38:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-28T17:38:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2023-11-28T17:38:39.513 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2023-11-28T17:38:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-28T17:38:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:38:39.513 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2023-11-28T17:38:39.513 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2023-11-28T17:38:39.513 INFO:teuthology.orchestra.run.smithi098.stdout:Install 135 Packages 2023-11-28T17:38:39.513 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 15 Packages 2023-11-28T17:38:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:38:39.514 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 205 M 2023-11-28T17:38:39.514 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2023-11-28T17:38:39.855 INFO:teuthology.orchestra.run.smithi113.stdout:(17/150): ceph-osd-18.0.0-7531.g47c1c547.el9.x8 7.6 MB/s | 18 MB 00:02 2023-11-28T17:38:39.905 INFO:teuthology.orchestra.run.smithi113.stdout:(18/150): python3-ceph-argparse-18.0.0-7531.g47 927 kB/s | 45 kB 00:00 2023-11-28T17:38:40.039 INFO:teuthology.orchestra.run.smithi113.stdout:(19/150): librgw2-18.0.0-7531.g47c1c547.el9.x86 6.0 MB/s | 4.8 MB 00:00 2023-11-28T17:38:40.073 INFO:teuthology.orchestra.run.smithi113.stdout:(20/150): python3-ceph-common-18.0.0-7531.g47c1 776 kB/s | 129 kB 00:00 2023-11-28T17:38:40.106 INFO:teuthology.orchestra.run.smithi113.stdout:(21/150): python3-cephfs-18.0.0-7531.g47c1c547. 2.4 MB/s | 162 kB 00:00 2023-11-28T17:38:40.148 INFO:teuthology.orchestra.run.smithi113.stdout:(22/150): python3-rados-18.0.0-7531.g47c1c547.e 4.2 MB/s | 321 kB 00:00 2023-11-28T17:38:40.182 INFO:teuthology.orchestra.run.smithi113.stdout:(23/150): python3-rbd-18.0.0-7531.g47c1c547.el9 3.9 MB/s | 298 kB 00:00 2023-11-28T17:38:40.215 INFO:teuthology.orchestra.run.smithi113.stdout:(24/150): python3-rgw-18.0.0-7531.g47c1c547.el9 1.5 MB/s | 99 kB 00:00 2023-11-28T17:38:40.240 INFO:teuthology.orchestra.run.smithi113.stdout:(25/150): rbd-fuse-18.0.0-7531.g47c1c547.el9.x8 1.4 MB/s | 85 kB 00:00 2023-11-28T17:38:40.299 INFO:teuthology.orchestra.run.smithi113.stdout:(26/150): rbd-nbd-18.0.0-7531.g47c1c547.el9.x86 2.9 MB/s | 170 kB 00:00 2023-11-28T17:38:40.400 INFO:teuthology.orchestra.run.smithi113.stdout:(27/150): ceph-grafana-dashboards-18.0.0-7531.g 283 kB/s | 28 kB 00:00 2023-11-28T17:38:40.500 INFO:teuthology.orchestra.run.smithi113.stdout:(28/150): rbd-mirror-18.0.0-7531.g47c1c547.el9. 11 MB/s | 3.0 MB 00:00 2023-11-28T17:38:40.534 INFO:teuthology.orchestra.run.smithi113.stdout:(29/150): ceph-mgr-cephadm-18.0.0-7531.g47c1c54 999 kB/s | 133 kB 00:00 2023-11-28T17:38:40.734 INFO:teuthology.orchestra.run.smithi113.stdout:(30/150): ceph-mgr-dashboard-18.0.0-7531.g47c1c 7.6 MB/s | 1.8 MB 00:00 2023-11-28T17:38:40.817 INFO:teuthology.orchestra.run.smithi113.stdout:(31/150): ceph-mgr-modules-core-18.0.0-7531.g47 2.9 MB/s | 243 kB 00:00 2023-11-28T17:38:40.893 INFO:teuthology.orchestra.run.smithi113.stdout:(32/150): ceph-mgr-rook-18.0.0-7531.g47c1c547.e 679 kB/s | 50 kB 00:00 2023-11-28T17:38:40.927 INFO:teuthology.orchestra.run.smithi098.stdout:(1/150): ceph-18.0.0-7531.g47c1c547.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2023-11-28T17:38:40.968 INFO:teuthology.orchestra.run.smithi113.stdout:(33/150): ceph-prometheus-alerts-18.0.0-7531.g4 204 kB/s | 15 kB 00:00 2023-11-28T17:38:41.080 INFO:teuthology.orchestra.run.smithi113.stdout:(34/150): ceph-volume-18.0.0-7531.g47c1c547.el9 2.3 MB/s | 262 kB 00:00 2023-11-28T17:38:41.252 INFO:teuthology.orchestra.run.smithi113.stdout:(35/150): cephadm-18.0.0-7531.g47c1c547.el9.noa 4.4 MB/s | 773 kB 00:00 2023-11-28T17:38:41.402 INFO:teuthology.orchestra.run.smithi113.stdout:(36/150): ceph-mgr-diskprediction-local-18.0.0- 8.5 MB/s | 7.4 MB 00:00 2023-11-28T17:38:41.427 INFO:teuthology.orchestra.run.smithi113.stdout:(37/150): cephfs-top-18.0.0-7531.g47c1c547.el9. 104 kB/s | 18 kB 00:00 2023-11-28T17:38:41.636 INFO:teuthology.orchestra.run.smithi113.stdout:(38/150): automake-1.16.2-8.el9.noarch.rpm 2.8 MB/s | 662 kB 00:00 2023-11-28T17:38:41.669 INFO:teuthology.orchestra.run.smithi113.stdout:(39/150): bison-3.7.4-5.el9.x86_64.rpm 3.7 MB/s | 920 kB 00:00 2023-11-28T17:38:41.686 INFO:teuthology.orchestra.run.smithi113.stdout:(40/150): boost-program-options-1.75.0-8.el9.x8 2.2 MB/s | 106 kB 00:00 2023-11-28T17:38:41.694 INFO:teuthology.orchestra.run.smithi098.stdout:(2/150): ceph-fuse-18.0.0-7531.g47c1c547.el9.x8 1.1 MB/s | 859 kB 00:00 2023-11-28T17:38:41.712 INFO:teuthology.orchestra.run.smithi113.stdout:(41/150): elfutils-libelf-devel-0.190-1.el9.x86 527 kB/s | 21 kB 00:00 2023-11-28T17:38:41.745 INFO:teuthology.orchestra.run.smithi113.stdout:(42/150): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2023-11-28T17:38:41.762 INFO:teuthology.orchestra.run.smithi113.stdout:(43/150): flexiblas-3.0.4-8.el9.x86_64.rpm 672 kB/s | 32 kB 00:00 2023-11-28T17:38:41.794 INFO:teuthology.orchestra.run.smithi098.stdout:(3/150): ceph-immutable-object-cache-18.0.0-753 1.4 MB/s | 141 kB 00:00 2023-11-28T17:38:41.930 INFO:teuthology.orchestra.run.smithi113.stdout:(44/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2023-11-28T17:38:41.946 INFO:teuthology.orchestra.run.smithi113.stdout:(45/150): flexiblas-openblas-openmp-3.0.4-8.el9 96 kB/s | 17 kB 00:00 2023-11-28T17:38:41.980 INFO:teuthology.orchestra.run.smithi113.stdout:(46/150): jq-1.6-16.el9.x86_64.rpm 3.7 MB/s | 186 kB 00:00 2023-11-28T17:38:42.005 INFO:teuthology.orchestra.run.smithi113.stdout:(47/150): libaio-devel-0.3.111-13.el9.x86_64.rp 179 kB/s | 10 kB 00:00 2023-11-28T17:38:42.030 INFO:teuthology.orchestra.run.smithi113.stdout:(48/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 159 kB 00:00 2023-11-28T17:38:42.056 INFO:teuthology.orchestra.run.smithi113.stdout:(49/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 921 kB/s | 44 kB 00:00 2023-11-28T17:38:42.098 INFO:teuthology.orchestra.run.smithi113.stdout:(50/150): librdkafka-1.6.1-102.el9.x86_64.rpm 9.9 MB/s | 661 kB 00:00 2023-11-28T17:38:42.158 INFO:teuthology.orchestra.run.smithi113.stdout:(51/150): libtool-2.4.6-45.el9.x86_64.rpm 5.9 MB/s | 578 kB 00:00 2023-11-28T17:38:42.182 INFO:teuthology.orchestra.run.smithi113.stdout:(52/150): libuuid-devel-2.37.4-15.el9.x86_64.rp 314 kB/s | 24 kB 00:00 2023-11-28T17:38:42.207 INFO:teuthology.orchestra.run.smithi113.stdout:(53/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 982 kB/s | 46 kB 00:00 2023-11-28T17:38:42.241 INFO:teuthology.orchestra.run.smithi113.stdout:(54/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2023-11-28T17:38:42.274 INFO:teuthology.orchestra.run.smithi113.stdout:(55/150): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2023-11-28T17:38:42.308 INFO:teuthology.orchestra.run.smithi113.stdout:(56/150): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2023-11-28T17:38:42.333 INFO:teuthology.orchestra.run.smithi113.stdout:(57/150): openblas-0.3.21-2.el9.x86_64.rpm 612 kB/s | 34 kB 00:00 2023-11-28T17:38:42.403 INFO:teuthology.orchestra.run.smithi098.stdout:(4/150): ceph-base-18.0.0-7531.g47c1c547.el9.x8 2.8 MB/s | 5.2 MB 00:01 2023-11-28T17:38:42.484 INFO:teuthology.orchestra.run.smithi113.stdout:(58/150): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2023-11-28T17:38:42.550 INFO:teuthology.orchestra.run.smithi113.stdout:(59/150): postgresql-13.11-1.el9.x86_64.rpm 7.5 MB/s | 1.5 MB 00:00 2023-11-28T17:38:42.601 INFO:teuthology.orchestra.run.smithi113.stdout:(60/150): postgresql-contrib-13.11-1.el9.x86_64 7.7 MB/s | 811 kB 00:00 2023-11-28T17:38:42.626 INFO:teuthology.orchestra.run.smithi113.stdout:(61/150): postgresql-private-libs-13.11-1.el9.x 1.9 MB/s | 134 kB 00:00 2023-11-28T17:38:42.686 INFO:teuthology.orchestra.run.smithi098.stdout:(5/150): ceph-mgr-18.0.0-7531.g47c1c547.el9.x86 5.2 MB/s | 1.5 MB 00:00 2023-11-28T17:38:42.753 INFO:teuthology.orchestra.run.smithi098.stdout:(6/150): ceph-mds-18.0.0-7531.g47c1c547.el9.x86 2.2 MB/s | 2.1 MB 00:00 2023-11-28T17:38:42.801 INFO:teuthology.orchestra.run.smithi113.stdout:(62/150): postgresql-server-13.11-1.el9.x86_64. 29 MB/s | 5.7 MB 00:00 2023-11-28T17:38:42.943 INFO:teuthology.orchestra.run.smithi113.stdout:(63/150): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2023-11-28T17:38:42.968 INFO:teuthology.orchestra.run.smithi113.stdout:(64/150): python3-jinja2-2.11.3-4.el9.noarch.rp 1.5 MB/s | 228 kB 00:00 2023-11-28T17:38:42.994 INFO:teuthology.orchestra.run.smithi113.stdout:(65/150): python3-jsonpatch-1.21-16.el9.noarch. 498 kB/s | 24 kB 00:00 2023-11-28T17:38:43.019 INFO:teuthology.orchestra.run.smithi113.stdout:(66/150): python3-jsonpointer-2.0-4.el9.noarch. 369 kB/s | 18 kB 00:00 2023-11-28T17:38:43.044 INFO:teuthology.orchestra.run.smithi113.stdout:(67/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2023-11-28T17:38:43.070 INFO:teuthology.orchestra.run.smithi113.stdout:(68/150): python3-markupsafe-1.1.1-12.el9.x86_6 656 kB/s | 32 kB 00:00 2023-11-28T17:38:43.445 INFO:teuthology.orchestra.run.smithi098.stdout:(7/150): ceph-mon-18.0.0-7531.g47c1c547.el9.x86 5.8 MB/s | 4.4 MB 00:00 2023-11-28T17:38:43.564 INFO:teuthology.orchestra.run.smithi113.stdout:(69/150): ceph-test-18.0.0-7531.g47c1c547.el9.x 8.0 MB/s | 41 MB 00:05 2023-11-28T17:38:43.687 INFO:teuthology.orchestra.run.smithi113.stdout:(70/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.5 MB/s | 4.8 MB 00:00 2023-11-28T17:38:43.713 INFO:teuthology.orchestra.run.smithi113.stdout:(71/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 349 kB/s | 217 kB 00:00 2023-11-28T17:38:43.755 INFO:teuthology.orchestra.run.smithi113.stdout:(72/150): python3-oauthlib-3.1.1-5.el9.noarch.r 979 kB/s | 174 kB 00:00 2023-11-28T17:38:43.781 INFO:teuthology.orchestra.run.smithi113.stdout:(73/150): python3-prettytable-0.7.2-27.el9.noar 637 kB/s | 41 kB 00:00 2023-11-28T17:38:43.796 INFO:teuthology.orchestra.run.smithi113.stdout:(74/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2023-11-28T17:38:43.830 INFO:teuthology.orchestra.run.smithi113.stdout:(75/150): python3-pyasn1-modules-0.4.8-6.el9.no 2.8 MB/s | 211 kB 00:00 2023-11-28T17:38:43.855 INFO:teuthology.orchestra.run.smithi113.stdout:(76/150): python3-requests-oauthlib-1.3.0-12.el 771 kB/s | 43 kB 00:00 2023-11-28T17:38:43.897 INFO:teuthology.orchestra.run.smithi113.stdout:(77/150): python3-pytz-2021.1-5.el9.noarch.rpm 484 kB/s | 47 kB 00:00 2023-11-28T17:38:44.081 INFO:teuthology.orchestra.run.smithi113.stdout:(78/150): python3-toml-0.10.2-6.el9.noarch.rpm 188 kB/s | 39 kB 00:00 2023-11-28T17:38:44.115 INFO:teuthology.orchestra.run.smithi113.stdout:(79/150): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2023-11-28T17:38:44.306 INFO:teuthology.orchestra.run.smithi113.stdout:(80/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2023-11-28T17:38:44.356 INFO:teuthology.orchestra.run.smithi113.stdout:(81/150): uuid-1.6.2-55.el9.x86_64.rpm 206 kB/s | 56 kB 00:00 2023-11-28T17:38:44.373 INFO:teuthology.orchestra.run.smithi113.stdout:(82/150): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 333 kB/s | 85 kB 00:00 2023-11-28T17:38:44.478 INFO:teuthology.orchestra.run.smithi113.stdout:(83/150): zlib-devel-1.2.11-40.el9.x86_64.rpm 369 kB/s | 44 kB 00:00 2023-11-28T17:38:44.507 INFO:teuthology.orchestra.run.smithi113.stdout:(84/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 421 kB/s | 63 kB 00:00 2023-11-28T17:38:44.533 INFO:teuthology.orchestra.run.smithi113.stdout:(85/150): fuse-2.9.9-15.el9.x86_64.rpm 496 kB/s | 79 kB 00:00 2023-11-28T17:38:44.583 INFO:teuthology.orchestra.run.smithi113.stdout:(86/150): libgfortran-11.4.1-2.3.el9.x86_64.rpm 7.6 MB/s | 803 kB 00:00 2023-11-28T17:38:44.616 INFO:teuthology.orchestra.run.smithi113.stdout:(87/150): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2023-11-28T17:38:44.642 INFO:teuthology.orchestra.run.smithi113.stdout:(88/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 241 kB 00:00 2023-11-28T17:38:44.654 INFO:teuthology.orchestra.run.smithi098.stdout:(8/150): ceph-radosgw-18.0.0-7531.g47c1c547.el9 7.2 MB/s | 8.7 MB 00:01 2023-11-28T17:38:44.701 INFO:teuthology.orchestra.run.smithi113.stdout:(89/150): python3-cryptography-36.0.1-4.el9.x86 9.9 MB/s | 1.1 MB 00:00 2023-11-28T17:38:44.726 INFO:teuthology.orchestra.run.smithi113.stdout:(90/150): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2023-11-28T17:38:44.729 INFO:teuthology.orchestra.run.smithi098.stdout:(9/150): ceph-selinux-18.0.0-7531.g47c1c547.el9 326 kB/s | 24 kB 00:00 2023-11-28T17:38:44.751 INFO:teuthology.orchestra.run.smithi113.stdout:(91/150): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 124 kB 00:00 2023-11-28T17:38:44.784 INFO:teuthology.orchestra.run.smithi113.stdout:(92/150): python3-requests-2.25.1-7.el9.noarch. 1.4 MB/s | 113 kB 00:00 2023-11-28T17:38:44.818 INFO:teuthology.orchestra.run.smithi113.stdout:(93/150): python3-urllib3-1.26.5-4.el9.noarch.r 2.8 MB/s | 187 kB 00:00 2023-11-28T17:38:44.993 INFO:teuthology.orchestra.run.smithi113.stdout:(94/150): lua-devel-5.4.4-4.el9.x86_64.rpm 128 kB/s | 22 kB 00:00 2023-11-28T17:38:45.552 INFO:teuthology.orchestra.run.smithi113.stdout:(95/150): python3-devel-3.9.17-2.el9.x86_64.rpm 321 kB/s | 245 kB 00:00 2023-11-28T17:38:45.694 INFO:teuthology.orchestra.run.smithi113.stdout:(96/150): gperftools-libs-2.9.1-2.el9.x86_64.rp 442 kB/s | 309 kB 00:00 2023-11-28T17:38:45.936 INFO:teuthology.orchestra.run.smithi113.stdout:(97/150): libarrow-doc-9.0.0-9.el9.noarch.rpm 102 kB/s | 24 kB 00:00 2023-11-28T17:38:46.086 INFO:teuthology.orchestra.run.smithi113.stdout:(98/150): libarrow-9.0.0-9.el9.x86_64.rpm 8.3 MB/s | 4.4 MB 00:00 2023-11-28T17:38:46.088 INFO:teuthology.orchestra.run.smithi098.stdout:(10/150): ceph-osd-18.0.0-7531.g47c1c547.el9.x8 5.3 MB/s | 18 MB 00:03 2023-11-28T17:38:46.111 INFO:teuthology.orchestra.run.smithi113.stdout:(99/150): liboath-2.6.7-2.el9.x86_64.rpm 296 kB/s | 49 kB 00:00 2023-11-28T17:38:46.163 INFO:teuthology.orchestra.run.smithi098.stdout:(11/150): cephfs-mirror-18.0.0-7531.g47c1c547.e 2.9 MB/s | 217 kB 00:00 2023-11-28T17:38:46.213 INFO:teuthology.orchestra.run.smithi098.stdout:(12/150): libcephfs-devel-18.0.0-7531.g47c1c547 634 kB/s | 31 kB 00:00 2023-11-28T17:38:46.220 INFO:teuthology.orchestra.run.smithi113.stdout:(100/150): openssl-devel-3.0.7-25.el9.x86_64.rp 2.8 MB/s | 4.1 MB 00:01 2023-11-28T17:38:46.245 INFO:teuthology.orchestra.run.smithi113.stdout:(101/150): libunwind-1.6.2-1.el9.x86_64.rpm 428 kB/s | 67 kB 00:00 2023-11-28T17:38:46.270 INFO:teuthology.orchestra.run.smithi113.stdout:(102/150): luarocks-3.9.2-1.el9.noarch.rpm 956 kB/s | 151 kB 00:00 2023-11-28T17:38:46.364 INFO:teuthology.orchestra.run.smithi098.stdout:(13/150): libcephfs2-18.0.0-7531.g47c1c547.el9. 4.7 MB/s | 719 kB 00:00 2023-11-28T17:38:46.414 INFO:teuthology.orchestra.run.smithi098.stdout:(14/150): libcephsqlite-18.0.0-7531.g47c1c547.e 3.3 MB/s | 163 kB 00:00 2023-11-28T17:38:46.464 INFO:teuthology.orchestra.run.smithi098.stdout:(15/150): librados-devel-18.0.0-7531.g47c1c547. 2.5 MB/s | 126 kB 00:00 2023-11-28T17:38:46.573 INFO:teuthology.orchestra.run.smithi098.stdout:(16/150): libradosstriper1-18.0.0-7531.g47c1c54 4.3 MB/s | 476 kB 00:00 2023-11-28T17:38:46.804 INFO:teuthology.orchestra.run.smithi113.stdout:(103/150): parquet-libs-9.0.0-9.el9.x86_64.rpm 1.4 MB/s | 837 kB 00:00 2023-11-28T17:38:46.846 INFO:teuthology.orchestra.run.smithi113.stdout:(104/150): python3-asyncssh-2.13.2-3.el9.noarch 953 kB/s | 548 kB 00:00 2023-11-28T17:38:46.871 INFO:teuthology.orchestra.run.smithi113.stdout:(105/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 72 kB/s | 43 kB 00:00 2023-11-28T17:38:46.913 INFO:teuthology.orchestra.run.smithi113.stdout:(106/150): python3-cachetools-4.2.4-1.el9.noarc 314 kB/s | 32 kB 00:00 2023-11-28T17:38:46.972 INFO:teuthology.orchestra.run.smithi113.stdout:(107/150): python3-certifi-2021.10.8-2.el9.noar 151 kB/s | 15 kB 00:00 2023-11-28T17:38:47.005 INFO:teuthology.orchestra.run.smithi113.stdout:(108/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2023-11-28T17:38:47.039 INFO:teuthology.orchestra.run.smithi113.stdout:(109/150): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2023-11-28T17:38:47.105 INFO:teuthology.orchestra.run.smithi113.stdout:(110/150): python3-google-auth-2.23.4-1.el9.noa 1.5 MB/s | 197 kB 00:00 2023-11-28T17:38:47.131 INFO:teuthology.orchestra.run.smithi113.stdout:(111/150): python3-jaraco-8.2.1-3.el9.noarch.rp 88 kB/s | 11 kB 00:00 2023-11-28T17:38:47.148 INFO:teuthology.orchestra.run.smithi113.stdout:(112/150): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2023-11-28T17:38:47.198 INFO:teuthology.orchestra.run.smithi113.stdout:(113/150): python3-jaraco-collections-3.0.0-8.e 253 kB/s | 23 kB 00:00 2023-11-28T17:38:47.215 INFO:teuthology.orchestra.run.smithi098.stdout:(17/150): librgw2-18.0.0-7531.g47c1c547.el9.x86 7.5 MB/s | 4.8 MB 00:00 2023-11-28T17:38:47.223 INFO:teuthology.orchestra.run.smithi113.stdout:(114/150): python3-jaraco-functools-3.5.0-2.el9 213 kB/s | 19 kB 00:00 2023-11-28T17:38:47.240 INFO:teuthology.orchestra.run.smithi113.stdout:(115/150): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2023-11-28T17:38:47.265 INFO:teuthology.orchestra.run.smithi098.stdout:(18/150): python3-ceph-argparse-18.0.0-7531.g47 922 kB/s | 45 kB 00:00 2023-11-28T17:38:47.291 INFO:teuthology.orchestra.run.smithi113.stdout:(116/150): python3-jwt+crypto-2.4.0-1.el9.noarc 99 kB/s | 9.0 kB 00:00 2023-11-28T17:38:47.316 INFO:teuthology.orchestra.run.smithi113.stdout:(117/150): python3-jwt-2.4.0-1.el9.noarch.rpm 446 kB/s | 41 kB 00:00 2023-11-28T17:38:47.324 INFO:teuthology.orchestra.run.smithi098.stdout:(19/150): python3-ceph-common-18.0.0-7531.g47c1 2.2 MB/s | 129 kB 00:00 2023-11-28T17:38:47.374 INFO:teuthology.orchestra.run.smithi113.stdout:(118/150): python3-kubernetes-26.1.0-2.el9.noar 7.7 MB/s | 1.0 MB 00:00 2023-11-28T17:38:47.382 INFO:teuthology.orchestra.run.smithi098.stdout:(20/150): python3-cephfs-18.0.0-7531.g47c1c547. 2.8 MB/s | 162 kB 00:00 2023-11-28T17:38:47.399 INFO:teuthology.orchestra.run.smithi113.stdout:(119/150): python3-logutils-0.3.5-21.el9.noarch 428 kB/s | 46 kB 00:00 2023-11-28T17:38:47.425 INFO:teuthology.orchestra.run.smithi113.stdout:(120/150): python3-more-itertools-8.12.0-2.el9. 728 kB/s | 79 kB 00:00 2023-11-28T17:38:47.483 INFO:teuthology.orchestra.run.smithi098.stdout:(21/150): python3-rados-18.0.0-7531.g47c1c547.e 3.2 MB/s | 321 kB 00:00 2023-11-28T17:38:47.483 INFO:teuthology.orchestra.run.smithi113.stdout:(121/150): python3-natsort-7.1.1-5.el9.noarch.r 535 kB/s | 58 kB 00:00 2023-11-28T17:38:47.518 INFO:teuthology.orchestra.run.smithi113.stdout:(122/150): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2023-11-28T17:38:47.543 INFO:teuthology.orchestra.run.smithi113.stdout:(123/150): python3-portend-3.1.0-2.el9.noarch.r 141 kB/s | 16 kB 00:00 2023-11-28T17:38:47.558 INFO:teuthology.orchestra.run.smithi098.stdout:(22/150): python3-rbd-18.0.0-7531.g47c1c547.el9 3.9 MB/s | 298 kB 00:00 2023-11-28T17:38:47.617 INFO:teuthology.orchestra.run.smithi098.stdout:(23/150): python3-rgw-18.0.0-7531.g47c1c547.el9 1.7 MB/s | 99 kB 00:00 2023-11-28T17:38:47.617 INFO:teuthology.orchestra.run.smithi113.stdout:(124/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 689 kB/s | 90 kB 00:00 2023-11-28T17:38:47.643 INFO:teuthology.orchestra.run.smithi113.stdout:(125/150): python3-repoze-lru-0.7-16.el9.noarch 247 kB/s | 31 kB 00:00 2023-11-28T17:38:47.667 INFO:teuthology.orchestra.run.smithi098.stdout:(24/150): rbd-fuse-18.0.0-7531.g47c1c547.el9.x8 1.7 MB/s | 85 kB 00:00 2023-11-28T17:38:47.676 INFO:teuthology.orchestra.run.smithi113.stdout:(126/150): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2023-11-28T17:38:47.719 INFO:teuthology.orchestra.run.smithi113.stdout:(127/150): python3-rsa-4.9-2.el9.noarch.rpm 592 kB/s | 59 kB 00:00 2023-11-28T17:38:47.752 INFO:teuthology.orchestra.run.smithi113.stdout:(128/150): python3-tempora-5.0.0-2.el9.noarch.r 336 kB/s | 36 kB 00:00 2023-11-28T17:38:47.777 INFO:teuthology.orchestra.run.smithi113.stdout:(129/150): python3-typing-extensions-4.4.0-2.el 510 kB/s | 51 kB 00:00 2023-11-28T17:38:47.836 INFO:teuthology.orchestra.run.smithi113.stdout:(130/150): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2023-11-28T17:38:47.861 INFO:teuthology.orchestra.run.smithi113.stdout:(131/150): python3-websocket-client-1.2.3-2.el9 829 kB/s | 90 kB 00:00 2023-11-28T17:38:47.906 INFO:teuthology.orchestra.run.smithi113.stdout:(132/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2023-11-28T17:38:47.954 INFO:teuthology.orchestra.run.smithi113.stdout:(133/150): python3-zc-lockfile-2.0-10.el9.noarc 171 kB/s | 20 kB 00:00 2023-11-28T17:38:47.987 INFO:teuthology.orchestra.run.smithi113.stdout:(134/150): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2023-11-28T17:38:48.067 INFO:teuthology.orchestra.run.smithi098.stdout:(25/150): rbd-mirror-18.0.0-7531.g47c1c547.el9. 7.5 MB/s | 3.0 MB 00:00 2023-11-28T17:38:48.126 INFO:teuthology.orchestra.run.smithi098.stdout:(26/150): rbd-nbd-18.0.0-7531.g47c1c547.el9.x86 2.9 MB/s | 170 kB 00:00 2023-11-28T17:38:48.154 INFO:teuthology.orchestra.run.smithi113.stdout:(135/150): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2023-11-28T17:38:48.176 INFO:teuthology.orchestra.run.smithi098.stdout:(27/150): ceph-grafana-dashboards-18.0.0-7531.g 572 kB/s | 28 kB 00:00 2023-11-28T17:38:48.205 INFO:teuthology.orchestra.run.smithi113.stdout:(136/150): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.3 MB/s | 68 kB 00:00 2023-11-28T17:38:48.260 INFO:teuthology.orchestra.run.smithi098.stdout:(28/150): ceph-mgr-cephadm-18.0.0-7531.g47c1c54 1.6 MB/s | 133 kB 00:00 2023-11-28T17:38:48.330 INFO:teuthology.orchestra.run.smithi113.stdout:(137/150): NetworkManager-1.45.6-1.el9.x86_64.r 18 MB/s | 2.2 MB 00:00 2023-11-28T17:38:48.447 INFO:teuthology.orchestra.run.smithi113.stdout:(138/150): NetworkManager-libnm-1.45.6-1.el9.x8 15 MB/s | 1.8 MB 00:00 2023-11-28T17:38:48.522 INFO:teuthology.orchestra.run.smithi113.stdout:(139/150): librados2-18.0.0-7531.g47c1c547.el9. 5.8 MB/s | 3.3 MB 00:00 2023-11-28T17:38:48.597 INFO:teuthology.orchestra.run.smithi113.stdout:(140/150): librbd1-18.0.0-7531.g47c1c547.el9.x8 5.0 MB/s | 3.0 MB 00:00 2023-11-28T17:38:48.602 INFO:teuthology.orchestra.run.smithi098.stdout:(29/150): ceph-mgr-dashboard-18.0.0-7531.g47c1c 5.2 MB/s | 1.8 MB 00:00 2023-11-28T17:38:48.622 INFO:teuthology.orchestra.run.smithi113.stdout:(141/150): NetworkManager-team-1.45.6-1.el9.x86 240 kB/s | 42 kB 00:00 2023-11-28T17:38:48.656 INFO:teuthology.orchestra.run.smithi113.stdout:(142/150): NetworkManager-tui-1.45.6-1.el9.x86_ 1.8 MB/s | 245 kB 00:00 2023-11-28T17:38:48.706 INFO:teuthology.orchestra.run.smithi113.stdout:(143/150): acl-2.3.1-4.el9.x86_64.rpm 856 kB/s | 69 kB 00:00 2023-11-28T17:38:48.732 INFO:teuthology.orchestra.run.smithi113.stdout:(144/150): elfutils-debuginfod-client-0.190-1.e 346 kB/s | 36 kB 00:00 2023-11-28T17:38:48.765 INFO:teuthology.orchestra.run.smithi113.stdout:(145/150): elfutils-libelf-0.190-1.el9.x86_64.r 1.7 MB/s | 192 kB 00:00 2023-11-28T17:38:48.790 INFO:teuthology.orchestra.run.smithi113.stdout:(146/150): libacl-2.3.1-4.el9.x86_64.rpm 394 kB/s | 22 kB 00:00 2023-11-28T17:38:48.832 INFO:teuthology.orchestra.run.smithi113.stdout:(147/150): elfutils-libs-0.190-1.el9.x86_64.rpm 2.5 MB/s | 255 kB 00:00 2023-11-28T17:38:48.859 INFO:teuthology.orchestra.run.smithi113.stdout:(148/150): xfsdump-3.1.12-4.el9.x86_64.rpm 3.4 MB/s | 321 kB 00:00 2023-11-28T17:38:48.885 INFO:teuthology.orchestra.run.smithi098.stdout:(30/150): ceph-common-18.0.0-7531.g47c1c547.el9 2.3 MB/s | 19 MB 00:08 2023-11-28T17:38:49.108 INFO:teuthology.orchestra.run.smithi113.stdout:(149/150): openssl-3.0.7-25.el9.x86_64.rpm 3.9 MB/s | 1.2 MB 00:00 2023-11-28T17:38:49.250 INFO:teuthology.orchestra.run.smithi113.stdout:(150/150): openssl-libs-3.0.7-25.el9.x86_64.rpm 5.5 MB/s | 2.2 MB 00:00 2023-11-28T17:38:49.260 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2023-11-28T17:38:49.260 INFO:teuthology.orchestra.run.smithi113.stdout:Total 14 MB/s | 205 MB 00:14 2023-11-28T17:38:49.419 INFO:teuthology.orchestra.run.smithi098.stdout:(31/150): ceph-test-18.0.0-7531.g47c1c547.el9.x 8.8 MB/s | 41 MB 00:04 2023-11-28T17:38:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:(32/150): ceph-mgr-modules-core-18.0.0-7531.g47 429 kB/s | 243 kB 00:00 2023-11-28T17:38:49.478 INFO:teuthology.orchestra.run.smithi098.stdout:(33/150): ceph-mgr-rook-18.0.0-7531.g47c1c547.e 873 kB/s | 50 kB 00:00 2023-11-28T17:38:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:(34/150): ceph-prometheus-alerts-18.0.0-7531.g4 228 kB/s | 15 kB 00:00 2023-11-28T17:38:49.526 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2023-11-28T17:38:49.670 INFO:teuthology.orchestra.run.smithi098.stdout:(35/150): ceph-volume-18.0.0-7531.g47c1c547.el9 1.3 MB/s | 262 kB 00:00 2023-11-28T17:38:49.721 INFO:teuthology.orchestra.run.smithi098.stdout:(36/150): cephfs-top-18.0.0-7531.g47c1c547.el9. 369 kB/s | 18 kB 00:00 2023-11-28T17:38:49.804 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2023-11-28T17:38:49.805 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2023-11-28T17:38:49.879 INFO:teuthology.orchestra.run.smithi098.stdout:(37/150): cephadm-18.0.0-7531.g47c1c547.el9.noa 2.1 MB/s | 773 kB 00:00 2023-11-28T17:38:50.080 INFO:teuthology.orchestra.run.smithi098.stdout:(38/150): ceph-mgr-diskprediction-local-18.0.0- 5.0 MB/s | 7.4 MB 00:01 2023-11-28T17:38:50.121 INFO:teuthology.orchestra.run.smithi098.stdout:(39/150): automake-1.16.2-8.el9.noarch.rpm 1.6 MB/s | 662 kB 00:00 2023-11-28T17:38:50.222 INFO:teuthology.orchestra.run.smithi098.stdout:(40/150): bison-3.7.4-5.el9.x86_64.rpm 2.6 MB/s | 920 kB 00:00 2023-11-28T17:38:50.247 INFO:teuthology.orchestra.run.smithi098.stdout:(41/150): boost-program-options-1.75.0-8.el9.x8 661 kB/s | 106 kB 00:00 2023-11-28T17:38:50.272 INFO:teuthology.orchestra.run.smithi098.stdout:(42/150): elfutils-libelf-devel-0.190-1.el9.x86 146 kB/s | 21 kB 00:00 2023-11-28T17:38:50.306 INFO:teuthology.orchestra.run.smithi098.stdout:(43/150): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 307 kB 00:00 2023-11-28T17:38:50.331 INFO:teuthology.orchestra.run.smithi098.stdout:(44/150): flexiblas-3.0.4-8.el9.x86_64.rpm 575 kB/s | 32 kB 00:00 2023-11-28T17:38:50.506 INFO:teuthology.orchestra.run.smithi098.stdout:(45/150): flexiblas-openblas-openmp-3.0.4-8.el9 86 kB/s | 17 kB 00:00 2023-11-28T17:38:50.632 INFO:teuthology.orchestra.run.smithi098.stdout:(46/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.4 MB/s | 3.0 MB 00:00 2023-11-28T17:38:50.657 INFO:teuthology.orchestra.run.smithi098.stdout:(47/150): jq-1.6-16.el9.x86_64.rpm 577 kB/s | 186 kB 00:00 2023-11-28T17:38:50.682 INFO:teuthology.orchestra.run.smithi098.stdout:(48/150): libaio-devel-0.3.111-13.el9.x86_64.rp 59 kB/s | 10 kB 00:00 2023-11-28T17:38:50.716 INFO:teuthology.orchestra.run.smithi098.stdout:(49/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2023-11-28T17:38:50.741 INFO:teuthology.orchestra.run.smithi098.stdout:(50/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 787 kB/s | 44 kB 00:00 2023-11-28T17:38:50.783 INFO:teuthology.orchestra.run.smithi098.stdout:(51/150): librdkafka-1.6.1-102.el9.x86_64.rpm 6.6 MB/s | 661 kB 00:00 2023-11-28T17:38:50.825 INFO:teuthology.orchestra.run.smithi098.stdout:(52/150): libtool-2.4.6-45.el9.x86_64.rpm 5.2 MB/s | 578 kB 00:00 2023-11-28T17:38:50.850 INFO:teuthology.orchestra.run.smithi098.stdout:(53/150): libuuid-devel-2.37.4-15.el9.x86_64.rp 233 kB/s | 24 kB 00:00 2023-11-28T17:38:50.875 INFO:teuthology.orchestra.run.smithi098.stdout:(54/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 530 kB/s | 46 kB 00:00 2023-11-28T17:38:50.909 INFO:teuthology.orchestra.run.smithi098.stdout:(55/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 282 kB 00:00 2023-11-28T17:38:50.942 INFO:teuthology.orchestra.run.smithi098.stdout:(56/150): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2023-11-28T17:38:50.976 INFO:teuthology.orchestra.run.smithi098.stdout:(57/150): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.2 MB/s | 217 kB 00:00 2023-11-28T17:38:50.993 INFO:teuthology.orchestra.run.smithi098.stdout:(58/150): openblas-0.3.21-2.el9.x86_64.rpm 413 kB/s | 34 kB 00:00 2023-11-28T17:38:51.360 INFO:teuthology.orchestra.run.smithi098.stdout:(59/150): openblas-openmp-0.3.21-2.el9.x86_64.r 11 MB/s | 4.7 MB 00:00 2023-11-28T17:38:51.410 INFO:teuthology.orchestra.run.smithi098.stdout:(60/150): postgresql-contrib-13.11-1.el9.x86_64 1.9 MB/s | 811 kB 00:00 2023-11-28T17:38:51.460 INFO:teuthology.orchestra.run.smithi098.stdout:(61/150): postgresql-13.11-1.el9.x86_64.rpm 3.3 MB/s | 1.5 MB 00:00 2023-11-28T17:38:51.485 INFO:teuthology.orchestra.run.smithi098.stdout:(62/150): postgresql-private-libs-13.11-1.el9.x 1.1 MB/s | 134 kB 00:00 2023-11-28T17:38:51.803 INFO:teuthology.orchestra.run.smithi098.stdout:(63/150): postgresql-server-13.11-1.el9.x86_64. 15 MB/s | 5.7 MB 00:00 2023-11-28T17:38:51.836 INFO:teuthology.orchestra.run.smithi098.stdout:(64/150): python3-jinja2-2.11.3-4.el9.noarch.rp 677 kB/s | 228 kB 00:00 2023-11-28T17:38:51.967 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2023-11-28T17:38:51.967 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2023-11-28T17:38:52.136 INFO:teuthology.orchestra.run.smithi098.stdout:(65/150): python3-babel-2.9.1-2.el9.noarch.rpm 9.1 MB/s | 5.8 MB 00:00 2023-11-28T17:38:52.161 INFO:teuthology.orchestra.run.smithi098.stdout:(66/150): python3-jsonpatch-1.21-16.el9.noarch. 69 kB/s | 24 kB 00:00 2023-11-28T17:38:52.187 INFO:teuthology.orchestra.run.smithi098.stdout:(67/150): python3-jsonpointer-2.0-4.el9.noarch. 54 kB/s | 18 kB 00:00 2023-11-28T17:38:52.254 INFO:teuthology.orchestra.run.smithi098.stdout:(68/150): python3-markupsafe-1.1.1-12.el9.x86_6 484 kB/s | 32 kB 00:00 2023-11-28T17:38:52.287 INFO:teuthology.orchestra.run.smithi098.stdout:(69/150): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-11-28T17:38:52.530 INFO:teuthology.orchestra.run.smithi098.stdout:(70/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 4.8 MB 00:00 2023-11-28T17:38:52.563 INFO:teuthology.orchestra.run.smithi098.stdout:(71/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 716 kB/s | 217 kB 00:00 2023-11-28T17:38:52.588 INFO:teuthology.orchestra.run.smithi098.stdout:(72/150): python3-oauthlib-3.1.1-5.el9.noarch.r 589 kB/s | 174 kB 00:00 2023-11-28T17:38:52.605 INFO:teuthology.orchestra.run.smithi098.stdout:(73/150): python3-prettytable-0.7.2-27.el9.noar 579 kB/s | 41 kB 00:00 2023-11-28T17:38:52.639 INFO:teuthology.orchestra.run.smithi098.stdout:(74/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 132 kB 00:00 2023-11-28T17:38:52.672 INFO:teuthology.orchestra.run.smithi098.stdout:(75/150): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2023-11-28T17:38:52.689 INFO:teuthology.orchestra.run.smithi098.stdout:(76/150): python3-pytz-2021.1-5.el9.noarch.rpm 569 kB/s | 47 kB 00:00 2023-11-28T17:38:52.706 INFO:teuthology.orchestra.run.smithi098.stdout:(77/150): python3-requests-oauthlib-1.3.0-12.el 666 kB/s | 43 kB 00:00 2023-11-28T17:38:52.732 INFO:teuthology.orchestra.run.smithi098.stdout:(78/150): python3-toml-0.10.2-6.el9.noarch.rpm 954 kB/s | 39 kB 00:00 2023-11-28T17:38:52.965 INFO:teuthology.orchestra.run.smithi098.stdout:(79/150): uuid-1.6.2-55.el9.x86_64.rpm 254 kB/s | 56 kB 00:00 2023-11-28T17:38:52.999 INFO:teuthology.orchestra.run.smithi098.stdout:(80/150): socat-1.7.4.1-5.el9.x86_64.rpm 1.0 MB/s | 300 kB 00:00 2023-11-28T17:38:53.174 INFO:teuthology.orchestra.run.smithi098.stdout:(81/150): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 413 kB/s | 85 kB 00:00 2023-11-28T17:38:53.391 INFO:teuthology.orchestra.run.smithi098.stdout:(82/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2023-11-28T17:38:53.416 INFO:teuthology.orchestra.run.smithi098.stdout:(83/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 155 kB/s | 63 kB 00:00 2023-11-28T17:38:53.441 INFO:teuthology.orchestra.run.smithi098.stdout:(84/150): zlib-devel-1.2.11-40.el9.x86_64.rpm 174 kB/s | 44 kB 00:00 2023-11-28T17:38:53.475 INFO:teuthology.orchestra.run.smithi098.stdout:(85/150): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2023-11-28T17:38:53.525 INFO:teuthology.orchestra.run.smithi098.stdout:(86/150): libgfortran-11.4.1-2.3.el9.x86_64.rpm 7.3 MB/s | 803 kB 00:00 2023-11-28T17:38:53.550 INFO:teuthology.orchestra.run.smithi098.stdout:(87/150): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.8 MB/s | 195 kB 00:00 2023-11-28T17:38:53.584 INFO:teuthology.orchestra.run.smithi098.stdout:(88/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 241 kB 00:00 2023-11-28T17:38:53.601 INFO:teuthology.orchestra.run.smithi098.stdout:(89/150): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2023-11-28T17:38:53.660 INFO:teuthology.orchestra.run.smithi098.stdout:(90/150): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2023-11-28T17:38:53.685 INFO:teuthology.orchestra.run.smithi098.stdout:(91/150): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 124 kB 00:00 2023-11-28T17:38:53.710 INFO:teuthology.orchestra.run.smithi098.stdout:(92/150): python3-requests-2.25.1-7.el9.noarch. 1.1 MB/s | 113 kB 00:00 2023-11-28T17:38:53.744 INFO:teuthology.orchestra.run.smithi098.stdout:(93/150): python3-urllib3-1.26.5-4.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2023-11-28T17:38:54.319 INFO:teuthology.orchestra.run.smithi098.stdout:(94/150): python3-devel-3.9.17-2.el9.x86_64.rpm 405 kB/s | 245 kB 00:00 2023-11-28T17:38:54.436 INFO:teuthology.orchestra.run.smithi098.stdout:(95/150): gperftools-libs-2.9.1-2.el9.x86_64.rp 2.6 MB/s | 309 kB 00:00 2023-11-28T17:38:54.545 INFO:teuthology.orchestra.run.smithi098.stdout:(96/150): lua-devel-5.4.4-4.el9.x86_64.rpm 28 kB/s | 22 kB 00:00 2023-11-28T17:38:54.637 INFO:teuthology.orchestra.run.smithi098.stdout:(97/150): libarrow-9.0.0-9.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2023-11-28T17:38:54.679 INFO:teuthology.orchestra.run.smithi098.stdout:(98/150): liboath-2.6.7-2.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2023-11-28T17:38:54.822 INFO:teuthology.orchestra.run.smithi098.stdout:(99/150): libarrow-doc-9.0.0-9.el9.noarch.rpm 88 kB/s | 24 kB 00:00 2023-11-28T17:38:54.846 INFO:teuthology.orchestra.run.smithi098.stdout:(100/150): libunwind-1.6.2-1.el9.x86_64.rpm 405 kB/s | 67 kB 00:00 2023-11-28T17:38:54.888 INFO:teuthology.orchestra.run.smithi098.stdout:(101/150): luarocks-3.9.2-1.el9.noarch.rpm 2.3 MB/s | 151 kB 00:00 2023-11-28T17:38:54.947 INFO:teuthology.orchestra.run.smithi098.stdout:(102/150): parquet-libs-9.0.0-9.el9.x86_64.rpm 8.2 MB/s | 837 kB 00:00 2023-11-28T17:38:54.989 INFO:teuthology.orchestra.run.smithi098.stdout:(103/150): python3-asyncssh-2.13.2-3.el9.noarch 5.4 MB/s | 548 kB 00:00 2023-11-28T17:38:55.014 INFO:teuthology.orchestra.run.smithi098.stdout:(104/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 652 kB/s | 43 kB 00:00 2023-11-28T17:38:55.064 INFO:teuthology.orchestra.run.smithi098.stdout:(105/150): python3-cachetools-4.2.4-1.el9.noarc 659 kB/s | 32 kB 00:00 2023-11-28T17:38:55.089 INFO:teuthology.orchestra.run.smithi098.stdout:(106/150): python3-certifi-2021.10.8-2.el9.noar 202 kB/s | 15 kB 00:00 2023-11-28T17:38:55.155 INFO:teuthology.orchestra.run.smithi098.stdout:(107/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.9 MB/s | 172 kB 00:00 2023-11-28T17:38:55.190 INFO:teuthology.orchestra.run.smithi098.stdout:(108/150): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2023-11-28T17:38:55.224 INFO:teuthology.orchestra.run.smithi098.stdout:(109/150): python3-google-auth-2.23.4-1.el9.noa 2.9 MB/s | 197 kB 00:00 2023-11-28T17:38:55.249 INFO:teuthology.orchestra.run.smithi098.stdout:(110/150): python3-jaraco-8.2.1-3.el9.noarch.rp 184 kB/s | 11 kB 00:00 2023-11-28T17:38:55.274 INFO:teuthology.orchestra.run.smithi098.stdout:(111/150): python3-jaraco-classes-3.2.1-5.el9.n 357 kB/s | 18 kB 00:00 2023-11-28T17:38:55.300 INFO:teuthology.orchestra.run.smithi098.stdout:(112/150): python3-jaraco-collections-3.0.0-8.e 468 kB/s | 23 kB 00:00 2023-11-28T17:38:55.325 INFO:teuthology.orchestra.run.smithi098.stdout:(113/150): python3-jaraco-functools-3.5.0-2.el9 393 kB/s | 19 kB 00:00 2023-11-28T17:38:55.350 INFO:teuthology.orchestra.run.smithi098.stdout:(114/150): python3-jaraco-text-3.2.0-6.el9.noar 398 kB/s | 20 kB 00:00 2023-11-28T17:38:55.375 INFO:teuthology.orchestra.run.smithi098.stdout:(115/150): python3-jwt+crypto-2.4.0-1.el9.noarc 183 kB/s | 9.0 kB 00:00 2023-11-28T17:38:55.400 INFO:teuthology.orchestra.run.smithi098.stdout:(116/150): python3-jwt-2.4.0-1.el9.noarch.rpm 832 kB/s | 41 kB 00:00 2023-11-28T17:38:55.459 INFO:teuthology.orchestra.run.smithi098.stdout:(117/150): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2023-11-28T17:38:55.477 INFO:teuthology.orchestra.run.smithi098.stdout:(118/150): python3-logutils-0.3.5-21.el9.noarch 621 kB/s | 46 kB 00:00 2023-11-28T17:38:55.501 INFO:teuthology.orchestra.run.smithi098.stdout:(119/150): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2023-11-28T17:38:55.519 INFO:teuthology.orchestra.run.smithi098.stdout:(120/150): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2023-11-28T17:38:55.560 INFO:teuthology.orchestra.run.smithi098.stdout:(121/150): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2023-11-28T17:38:55.578 INFO:teuthology.orchestra.run.smithi098.stdout:(122/150): python3-portend-3.1.0-2.el9.noarch.r 283 kB/s | 16 kB 00:00 2023-11-28T17:38:55.628 INFO:teuthology.orchestra.run.smithi098.stdout:(123/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2023-11-28T17:38:55.653 INFO:teuthology.orchestra.run.smithi098.stdout:(124/150): python3-repoze-lru-0.7-16.el9.noarch 412 kB/s | 31 kB 00:00 2023-11-28T17:38:55.686 INFO:teuthology.orchestra.run.smithi098.stdout:(125/150): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2023-11-28T17:38:55.712 INFO:teuthology.orchestra.run.smithi098.stdout:(126/150): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2023-11-28T17:38:55.738 INFO:teuthology.orchestra.run.smithi098.stdout:(127/150): python3-tempora-5.0.0-2.el9.noarch.r 708 kB/s | 36 kB 00:00 2023-11-28T17:38:55.763 INFO:teuthology.orchestra.run.smithi098.stdout:(128/150): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2023-11-28T17:38:55.796 INFO:teuthology.orchestra.run.smithi098.stdout:(129/150): python3-webob-1.8.7-6.el9.noarch.rpm 3.9 MB/s | 230 kB 00:00 2023-11-28T17:38:55.822 INFO:teuthology.orchestra.run.smithi098.stdout:(130/150): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2023-11-28T17:38:55.864 INFO:teuthology.orchestra.run.smithi098.stdout:(131/150): python3-werkzeug-2.0.3-3.el9.1.noarc 6.3 MB/s | 427 kB 00:00 2023-11-28T17:38:55.889 INFO:teuthology.orchestra.run.smithi098.stdout:(132/150): python3-zc-lockfile-2.0-10.el9.noarc 302 kB/s | 20 kB 00:00 2023-11-28T17:38:55.923 INFO:teuthology.orchestra.run.smithi098.stdout:(133/150): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2023-11-28T17:38:55.990 INFO:teuthology.orchestra.run.smithi098.stdout:(134/150): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2023-11-28T17:38:56.323 INFO:teuthology.orchestra.run.smithi098.stdout:(135/150): librbd1-18.0.0-7531.g47c1c547.el9.x8 9.1 MB/s | 3.0 MB 00:00 2023-11-28T17:38:56.398 INFO:teuthology.orchestra.run.smithi098.stdout:(136/150): librados2-18.0.0-7531.g47c1c547.el9. 7.0 MB/s | 3.3 MB 00:00 2023-11-28T17:38:56.424 INFO:teuthology.orchestra.run.smithi098.stdout:(137/150): libacl-devel-2.3.1-4.el9.x86_64.rpm 682 kB/s | 68 kB 00:00 2023-11-28T17:38:56.524 INFO:teuthology.orchestra.run.smithi098.stdout:(138/150): NetworkManager-1.45.6-1.el9.x86_64.r 18 MB/s | 2.2 MB 00:00 2023-11-28T17:38:56.591 INFO:teuthology.orchestra.run.smithi098.stdout:(139/150): NetworkManager-libnm-1.45.6-1.el9.x8 11 MB/s | 1.8 MB 00:00 2023-11-28T17:38:56.616 INFO:teuthology.orchestra.run.smithi098.stdout:(140/150): NetworkManager-team-1.45.6-1.el9.x86 534 kB/s | 42 kB 00:00 2023-11-28T17:38:56.650 INFO:teuthology.orchestra.run.smithi098.stdout:(141/150): NetworkManager-tui-1.45.6-1.el9.x86_ 4.2 MB/s | 245 kB 00:00 2023-11-28T17:38:56.676 INFO:teuthology.orchestra.run.smithi098.stdout:(142/150): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 69 kB 00:00 2023-11-28T17:38:56.701 INFO:teuthology.orchestra.run.smithi098.stdout:(143/150): elfutils-debuginfod-client-0.190-1.e 747 kB/s | 36 kB 00:00 2023-11-28T17:38:56.726 INFO:teuthology.orchestra.run.smithi098.stdout:(144/150): elfutils-libelf-0.190-1.el9.x86_64.r 3.8 MB/s | 192 kB 00:00 2023-11-28T17:38:56.760 INFO:teuthology.orchestra.run.smithi098.stdout:(145/150): elfutils-libs-0.190-1.el9.x86_64.rpm 4.4 MB/s | 255 kB 00:00 2023-11-28T17:38:56.785 INFO:teuthology.orchestra.run.smithi098.stdout:(146/150): libacl-2.3.1-4.el9.x86_64.rpm 394 kB/s | 22 kB 00:00 2023-11-28T17:38:56.819 INFO:teuthology.orchestra.run.smithi098.stdout:(147/150): xfsdump-3.1.12-4.el9.x86_64.rpm 5.5 MB/s | 321 kB 00:00 2023-11-28T17:38:57.544 INFO:teuthology.orchestra.run.smithi098.stdout:(148/150): openssl-3.0.7-25.el9.x86_64.rpm 1.6 MB/s | 1.2 MB 00:00 2023-11-28T17:38:57.727 INFO:teuthology.orchestra.run.smithi098.stdout:(149/150): openssl-libs-3.0.7-25.el9.x86_64.rpm 2.4 MB/s | 2.2 MB 00:00 2023-11-28T17:38:58.278 INFO:teuthology.orchestra.run.smithi098.stdout:(150/150): openssl-devel-3.0.7-25.el9.x86_64.rp 920 kB/s | 4.1 MB 00:04 2023-11-28T17:38:58.286 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2023-11-28T17:38:58.287 INFO:teuthology.orchestra.run.smithi098.stdout:Total 11 MB/s | 205 MB 00:18 2023-11-28T17:38:58.448 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2023-11-28T17:38:58.536 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2023-11-28T17:38:58.603 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/165 2023-11-28T17:38:58.639 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/165 2023-11-28T17:38:58.709 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/165 2023-11-28T17:38:58.769 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/165 2023-11-28T17:38:58.799 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2023-11-28T17:38:58.813 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2023-11-28T17:38:58.814 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2023-11-28T17:38:58.890 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2023-11-28T17:38:59.309 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2023-11-28T17:38:59.363 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 8/165 2023-11-28T17:38:59.510 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 8/165 2023-11-28T17:38:59.571 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 9/165 2023-11-28T17:38:59.697 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 9/165 2023-11-28T17:38:59.737 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rados-2:18.0.0-7531.g47c1c547.el9.x86_64 10/165 2023-11-28T17:38:59.803 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/165 2023-11-28T17:39:00.096 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-libelf-0.190-1.el9.x86_64 12/165 2023-11-28T17:39:00.152 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-13.11-1.el9.x86_64 13/165 2023-11-28T17:39:00.177 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 14/165 2023-11-28T17:39:00.338 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 14/165 2023-11-28T17:39:00.370 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86 15/165 2023-11-28T17:39:00.453 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86 15/165 2023-11-28T17:39:00.791 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 16/165 2023-11-28T17:39:00.975 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-libnm-1:1.45.6-1.el9.x86_64 17/165 2023-11-28T17:39:00.976 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2023-11-28T17:39:00.977 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2023-11-28T17:39:01.113 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.6-1.el9.x86_64 17/165 2023-11-28T17:39:01.313 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.45.6-1.el9.x86_64 18/165 2023-11-28T17:39:01.378 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-1:1.45.6-1.el9.x86_64 18/165 2023-11-28T17:39:01.823 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.45.6-1.el9.x86_64 18/165 2023-11-28T17:39:01.867 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/165 2023-11-28T17:39:01.974 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/165 2023-11-28T17:39:02.038 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 21/165 2023-11-28T17:39:02.079 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 22/165 2023-11-28T17:39:02.135 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 23/165 2023-11-28T17:39:02.189 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 24/165 2023-11-28T17:39:02.294 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 25/165 2023-11-28T17:39:02.340 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 26/165 2023-11-28T17:39:02.385 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/165 2023-11-28T17:39:02.425 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2023-11-28T17:39:02.539 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2023-11-28T17:39:02.579 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-common-2:18.0.0-7531.g47c1c547.el9. 30/165 2023-11-28T17:39:02.630 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-argparse-2:18.0.0-7531.g47c1c547.el 31/165 2023-11-28T17:39:02.660 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cephfs-2:18.0.0-7531.g47c1c547.el9.x86_6 32/165 2023-11-28T17:39:02.769 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/165 2023-11-28T17:39:02.865 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/165 2023-11-28T17:39:03.043 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 35/165 2023-11-28T17:39:03.098 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 36/165 2023-11-28T17:39:03.784 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rsa-4.9-2.el9.noarch 37/165 2023-11-28T17:39:03.885 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 38/165 2023-11-28T17:39:04.127 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 39/165 2023-11-28T17:39:04.187 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-libs-0.190-1.el9.x86_64 40/165 2023-11-28T17:39:04.237 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librados-devel-2:18.0.0-7531.g47c1c547.el9.x86_6 41/165 2023-11-28T17:39:04.310 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 42/165 2023-11-28T17:39:04.524 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 43/165 2023-11-28T17:39:04.569 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 44/165 2023-11-28T17:39:04.712 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : re2-1:20211101-3.el9.x86_64 45/165 2023-11-28T17:39:04.756 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 46/165 2023-11-28T17:39:04.820 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/165 2023-11-28T17:39:04.906 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/165 2023-11-28T17:39:04.965 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/165 2023-11-28T17:39:05.003 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/165 2023-11-28T17:39:05.080 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/165 2023-11-28T17:39:05.136 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/165 2023-11-28T17:39:05.195 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 53/165 2023-11-28T17:39:05.388 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 54/165 2023-11-28T17:39:05.429 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 55/165 2023-11-28T17:39:05.482 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 56/165 2023-11-28T17:39:05.607 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 57/165 2023-11-28T17:39:05.657 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 58/165 2023-11-28T17:39:05.714 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 59/165 2023-11-28T17:39:05.745 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 60/165 2023-11-28T17:39:05.901 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 61/165 2023-11-28T17:39:05.958 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 62/165 2023-11-28T17:39:06.027 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ply-3.11-14.el9.noarch 63/165 2023-11-28T17:39:06.109 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 64/165 2023-11-28T17:39:06.361 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 65/165 2023-11-28T17:39:06.421 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 66/165 2023-11-28T17:39:06.617 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 67/165 2023-11-28T17:39:06.668 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 68/165 2023-11-28T17:39:06.698 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 69/165 2023-11-28T17:39:06.737 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 70/165 2023-11-28T17:39:06.780 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 71/165 2023-11-28T17:39:06.817 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : fuse-2.9.9-15.el9.x86_64 72/165 2023-11-28T17:39:06.859 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 73/165 2023-11-28T17:39:06.924 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 74/165 2023-11-28T17:39:07.045 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : uuid-1.6.2-55.el9.x86_64 75/165 2023-11-28T17:39:07.097 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/165 2023-11-28T17:39:07.107 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2023-11-28T17:39:07.235 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/165 2023-11-28T17:39:07.249 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/165 2023-11-28T17:39:07.288 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 78/165 2023-11-28T17:39:07.342 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 79/165 2023-11-28T17:39:07.410 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 80/165 2023-11-28T17:39:07.451 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 81/165 2023-11-28T17:39:07.453 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/165 2023-11-28T17:39:07.588 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/165 2023-11-28T17:39:07.598 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 82/165 2023-11-28T17:39:07.635 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/165 2023-11-28T17:39:07.640 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 83/165 2023-11-28T17:39:07.660 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2023-11-28T17:39:07.720 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2023-11-28T17:39:07.786 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/165 2023-11-28T17:39:07.846 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/165 2023-11-28T17:39:08.145 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2023-11-28T17:39:08.201 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 8/165 2023-11-28T17:39:08.351 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 8/165 2023-11-28T17:39:08.393 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 9/165 2023-11-28T17:39:08.508 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 9/165 2023-11-28T17:39:08.564 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:18.0.0-7531.g47c1c547.el9.x86_64 10/165 2023-11-28T17:39:08.642 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/165 2023-11-28T17:39:08.816 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/165 2023-11-28T17:39:08.862 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/165 2023-11-28T17:39:08.896 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/165 2023-11-28T17:39:08.929 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/165 2023-11-28T17:39:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libelf-0.190-1.el9.x86_64 12/165 2023-11-28T17:39:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-13.11-1.el9.x86_64 13/165 2023-11-28T17:39:09.032 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 14/165 2023-11-28T17:39:09.180 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 14/165 2023-11-28T17:39:09.184 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-0.3.21-2.el9.x86_64 90/165 2023-11-28T17:39:09.208 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86 15/165 2023-11-28T17:39:09.217 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 91/165 2023-11-28T17:39:09.282 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86 15/165 2023-11-28T17:39:09.311 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 92/165 2023-11-28T17:39:09.612 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 16/165 2023-11-28T17:39:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-libnm-1:1.45.6-1.el9.x86_64 17/165 2023-11-28T17:39:09.877 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.6-1.el9.x86_64 17/165 2023-11-28T17:39:10.083 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.45.6-1.el9.x86_64 18/165 2023-11-28T17:39:10.113 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 93/165 2023-11-28T17:39:10.141 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-1:1.45.6-1.el9.x86_64 18/165 2023-11-28T17:39:10.253 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 94/165 2023-11-28T17:39:10.568 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.45.6-1.el9.x86_64 18/165 2023-11-28T17:39:10.646 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/165 2023-11-28T17:39:10.761 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/165 2023-11-28T17:39:10.840 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 21/165 2023-11-28T17:39:10.876 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 22/165 2023-11-28T17:39:10.934 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 23/165 2023-11-28T17:39:10.987 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 24/165 2023-11-28T17:39:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 25/165 2023-11-28T17:39:11.136 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 26/165 2023-11-28T17:39:11.191 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/165 2023-11-28T17:39:11.222 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2023-11-28T17:39:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2023-11-28T17:39:11.376 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:18.0.0-7531.g47c1c547.el9. 30/165 2023-11-28T17:39:11.426 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:18.0.0-7531.g47c1c547.el 31/165 2023-11-28T17:39:11.457 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:18.0.0-7531.g47c1c547.el9.x86_6 32/165 2023-11-28T17:39:11.571 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/165 2023-11-28T17:39:11.694 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/165 2023-11-28T17:39:11.790 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 95/165 2023-11-28T17:39:11.873 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 35/165 2023-11-28T17:39:11.896 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 96/165 2023-11-28T17:39:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 97/165 2023-11-28T17:39:11.942 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 36/165 2023-11-28T17:39:11.966 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : jq-1.6-16.el9.x86_64 98/165 2023-11-28T17:39:12.009 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2023-11-28T17:39:12.052 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 100/165 2023-11-28T17:39:12.418 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2023-11-28T17:39:12.463 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 102/165 2023-11-28T17:39:12.584 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 102/165 2023-11-28T17:39:12.628 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rbd-2:18.0.0-7531.g47c1c547.el9.x86_64 103/165 2023-11-28T17:39:12.637 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 37/165 2023-11-28T17:39:12.686 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2023-11-28T17:39:12.737 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 38/165 2023-11-28T17:39:12.797 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 39/165 2023-11-28T17:39:12.858 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libs-0.190-1.el9.x86_64 40/165 2023-11-28T17:39:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:18.0.0-7531.g47c1c547.el9.x86_6 41/165 2023-11-28T17:39:12.970 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 42/165 2023-11-28T17:39:13.162 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 43/165 2023-11-28T17:39:13.170 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 105/165 2023-11-28T17:39:13.199 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 106/165 2023-11-28T17:39:13.208 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 44/165 2023-11-28T17:39:13.283 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 106/165 2023-11-28T17:39:13.310 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rgw-2:18.0.0-7531.g47c1c547.el9.x86_64 107/165 2023-11-28T17:39:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 45/165 2023-11-28T17:39:13.394 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 46/165 2023-11-28T17:39:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/165 2023-11-28T17:39:13.567 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/165 2023-11-28T17:39:13.604 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/165 2023-11-28T17:39:13.675 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/165 2023-11-28T17:39:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/165 2023-11-28T17:39:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/165 2023-11-28T17:39:13.872 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 53/165 2023-11-28T17:39:14.064 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 54/165 2023-11-28T17:39:14.111 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 55/165 2023-11-28T17:39:14.156 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 56/165 2023-11-28T17:39:14.277 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 57/165 2023-11-28T17:39:14.320 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 58/165 2023-11-28T17:39:14.367 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 59/165 2023-11-28T17:39:14.401 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 60/165 2023-11-28T17:39:14.550 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 61/165 2023-11-28T17:39:14.614 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 62/165 2023-11-28T17:39:14.681 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 63/165 2023-11-28T17:39:14.765 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 64/165 2023-11-28T17:39:15.022 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 65/165 2023-11-28T17:39:15.086 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 66/165 2023-11-28T17:39:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 67/165 2023-11-28T17:39:15.313 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 68/165 2023-11-28T17:39:15.338 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 69/165 2023-11-28T17:39:15.377 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 70/165 2023-11-28T17:39:15.419 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 71/165 2023-11-28T17:39:15.457 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 72/165 2023-11-28T17:39:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 73/165 2023-11-28T17:39:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 74/165 2023-11-28T17:39:15.604 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : uuid-1.6.2-55.el9.x86_64 75/165 2023-11-28T17:39:15.649 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/165 2023-11-28T17:39:15.797 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/165 2023-11-28T17:39:15.821 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 108/165 2023-11-28T17:39:15.836 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 78/165 2023-11-28T17:39:15.864 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 108/165 2023-11-28T17:39:15.870 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 79/165 2023-11-28T17:39:15.925 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 80/165 2023-11-28T17:39:15.965 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 81/165 2023-11-28T17:39:16.105 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 82/165 2023-11-28T17:39:16.155 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 83/165 2023-11-28T17:39:16.294 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/165 2023-11-28T17:39:16.370 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/165 2023-11-28T17:39:16.481 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 108/165 2023-11-28T17:39:16.526 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 109/165 2023-11-28T17:39:16.689 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 109/165 2023-11-28T17:39:16.689 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-28T17:39:16.689 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-28T17:39:16.689 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:39:16.727 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 110/165 2023-11-28T17:39:17.340 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/165 2023-11-28T17:39:17.394 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/165 2023-11-28T17:39:17.428 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/165 2023-11-28T17:39:17.461 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/165 2023-11-28T17:39:17.709 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 90/165 2023-11-28T17:39:17.782 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 91/165 2023-11-28T17:39:17.898 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 92/165 2023-11-28T17:39:18.713 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 93/165 2023-11-28T17:39:18.869 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 94/165 2023-11-28T17:39:20.405 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 95/165 2023-11-28T17:39:20.487 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 96/165 2023-11-28T17:39:20.525 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 97/165 2023-11-28T17:39:20.554 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : jq-1.6-16.el9.x86_64 98/165 2023-11-28T17:39:20.589 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2023-11-28T17:39:20.622 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 100/165 2023-11-28T17:39:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2023-11-28T17:39:21.312 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 102/165 2023-11-28T17:39:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 102/165 2023-11-28T17:39:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:18.0.0-7531.g47c1c547.el9.x86_64 103/165 2023-11-28T17:39:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2023-11-28T17:39:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 105/165 2023-11-28T17:39:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 106/165 2023-11-28T17:39:22.174 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 106/165 2023-11-28T17:39:22.201 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:18.0.0-7531.g47c1c547.el9.x86_64 107/165 2023-11-28T17:39:24.615 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 108/165 2023-11-28T17:39:24.655 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 108/165 2023-11-28T17:39:25.286 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 108/165 2023-11-28T17:39:25.334 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 109/165 2023-11-28T17:39:25.499 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 109/165 2023-11-28T17:39:25.499 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-28T17:39:25.500 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-28T17:39:25.500 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:39:25.544 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 110/165 2023-11-28T17:39:29.014 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 110/165 2023-11-28T17:39:29.014 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /sys 2023-11-28T17:39:29.014 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /proc 2023-11-28T17:39:29.014 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /mnt 2023-11-28T17:39:29.015 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /var/tmp 2023-11-28T17:39:29.015 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /home 2023-11-28T17:39:29.015 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /root 2023-11-28T17:39:29.015 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /tmp 2023-11-28T17:39:29.015 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:39:29.210 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 111/165 2023-11-28T17:39:29.255 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 111/165 2023-11-28T17:39:29.256 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:29.256 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-28T17:39:29.256 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:39:30.929 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 112/165 2023-11-28T17:39:31.013 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 112/165 2023-11-28T17:39:31.014 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:31.014 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-28T17:39:31.014 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T17:39:31.014 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T17:39:31.014 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:39:31.237 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 113/165 2023-11-28T17:39:31.325 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 113/165 2023-11-28T17:39:31.325 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:31.325 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-28T17:39:31.325 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T17:39:31.326 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T17:39:31.326 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:39:31.838 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 114/165 2023-11-28T17:39:31.922 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 114/165 2023-11-28T17:39:31.922 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:31.922 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-28T17:39:31.923 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T17:39:31.923 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T17:39:31.923 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:39:32.048 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2023-11-28T17:39:32.699 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 116/165 2023-11-28T17:39:32.732 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 116/165 2023-11-28T17:39:32.768 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7531.g47c1c547.e 117/165 2023-11-28T17:39:32.861 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7531.g47c1c547. 118/165 2023-11-28T17:39:32.887 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noa 119/165 2023-11-28T17:39:33.200 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noa 119/165 2023-11-28T17:39:33.246 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.n 120/165 2023-11-28T17:39:34.379 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.n 120/165 2023-11-28T17:39:34.405 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c 121/165 2023-11-28T17:39:34.549 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c 121/165 2023-11-28T17:39:34.737 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7531.g47c1c547.el 122/165 2023-11-28T17:39:34.782 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 123/165 2023-11-28T17:39:34.866 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 123/165 2023-11-28T17:39:34.867 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:34.867 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-28T17:39:34.867 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T17:39:34.867 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T17:39:34.867 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:39:34.924 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 124/165 2023-11-28T17:39:34.963 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 124/165 2023-11-28T17:39:35.054 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-2:18.0.0-7531.g47c1c547.el9.x86_64 125/165 2023-11-28T17:39:35.977 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libtool-2.4.6-45.el9.x86_64 126/165 2023-11-28T17:39:36.018 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 127/165 2023-11-28T17:39:36.094 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 127/165 2023-11-28T17:39:36.095 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:36.095 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-28T17:39:36.095 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T17:39:36.095 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T17:39:36.095 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:39:36.143 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7531.g47c1c 128/165 2023-11-28T17:39:36.226 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7531.g47c1c 128/165 2023-11-28T17:39:36.226 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:36.226 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-28T17:39:36.226 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:39:36.295 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 129/165 2023-11-28T17:39:36.378 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 129/165 2023-11-28T17:39:36.378 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:36.378 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-11-28T17:39:36.379 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-28T17:39:36.379 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-28T17:39:36.379 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:39:36.770 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 130/165 2023-11-28T17:39:36.845 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 130/165 2023-11-28T17:39:36.845 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:36.845 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-28T17:39:36.846 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T17:39:36.846 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T17:39:36.846 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:39:38.476 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 110/165 2023-11-28T17:39:38.476 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2023-11-28T17:39:38.476 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2023-11-28T17:39:38.476 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2023-11-28T17:39:38.476 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2023-11-28T17:39:38.476 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2023-11-28T17:39:38.476 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2023-11-28T17:39:38.476 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2023-11-28T17:39:38.476 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:39:38.686 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 111/165 2023-11-28T17:39:38.731 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 111/165 2023-11-28T17:39:38.731 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:38.732 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-28T17:39:38.732 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:39:40.455 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 112/165 2023-11-28T17:39:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 112/165 2023-11-28T17:39:40.567 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:40.567 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-28T17:39:40.567 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T17:39:40.567 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T17:39:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:39:40.789 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 113/165 2023-11-28T17:39:40.877 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 113/165 2023-11-28T17:39:40.877 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:40.877 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-28T17:39:40.878 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T17:39:40.878 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T17:39:40.878 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:39:41.046 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-test-2:18.0.0-7531.g47c1c547.el9.x86_64 131/165 2023-11-28T17:39:41.108 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 132/165 2023-11-28T17:39:41.186 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-nbd-2:18.0.0-7531.g47c1c547.el9.x86_64 133/165 2023-11-28T17:39:41.226 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 134/165 2023-11-28T17:39:41.365 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 114/165 2023-11-28T17:39:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 114/165 2023-11-28T17:39:41.457 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:41.457 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-28T17:39:41.457 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T17:39:41.457 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T17:39:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:39:41.518 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : elfutils-libelf-devel-0.190-1.el9.x86_64 135/165 2023-11-28T17:39:41.566 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2023-11-28T17:39:41.647 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 136/165 2023-11-28T17:39:41.679 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 137/165 2023-11-28T17:39:41.712 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs-devel-2:18.0.0-7531.g47c1c547.el9.x86_ 138/165 2023-11-28T17:39:41.751 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-debuginfod-client-0.190-1.el9.x86_64 139/165 2023-11-28T17:39:41.795 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-team-1:1.45.6-1.el9.x86_64 140/165 2023-11-28T17:39:41.911 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-tui-1:1.45.6-1.el9.x86_64 141/165 2023-11-28T17:39:41.976 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 142/165 2023-11-28T17:39:42.022 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 143/165 2023-11-28T17:39:42.128 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 116/165 2023-11-28T17:39:42.167 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 116/165 2023-11-28T17:39:42.212 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7531.g47c1c547.e 117/165 2023-11-28T17:39:42.307 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7531.g47c1c547. 118/165 2023-11-28T17:39:42.338 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noa 119/165 2023-11-28T17:39:42.660 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noa 119/165 2023-11-28T17:39:42.697 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.n 120/165 2023-11-28T17:39:42.968 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T17:39:43.012 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T17:39:43.070 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T17:39:43.844 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.n 120/165 2023-11-28T17:39:43.873 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c 121/165 2023-11-28T17:39:44.014 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c 121/165 2023-11-28T17:39:44.186 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7531.g47c1c547.el 122/165 2023-11-28T17:39:44.217 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 123/165 2023-11-28T17:39:44.288 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 123/165 2023-11-28T17:39:44.288 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:44.288 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-28T17:39:44.288 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T17:39:44.288 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T17:39:44.288 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:39:44.343 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 124/165 2023-11-28T17:39:44.382 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 124/165 2023-11-28T17:39:44.390 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephfs-top-2:18.0.0-7531.g47c1c547.el9.noarch 145/165 2023-11-28T17:39:44.444 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:18.0.0-7531.g47c1c547.el9.x86_64 125/165 2023-11-28T17:39:44.524 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 146/165 2023-11-28T17:39:44.575 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 147/165 2023-11-28T17:39:44.646 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/165 2023-11-28T17:39:44.802 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/165 2023-11-28T17:39:44.838 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/165 2023-11-28T17:39:44.871 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 151/165 2023-11-28T17:39:44.950 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 152/165 2023-11-28T17:39:44.951 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 153/165 2023-11-28T17:39:44.951 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 154/165 2023-11-28T17:39:45.096 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 154/165 2023-11-28T17:39:45.141 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 155/165 2023-11-28T17:39:45.185 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 156/165 2023-11-28T17:39:45.217 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 157/165 2023-11-28T17:39:45.217 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 158/165 2023-11-28T17:39:45.334 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libtool-2.4.6-45.el9.x86_64 126/165 2023-11-28T17:39:45.355 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 158/165 2023-11-28T17:39:45.361 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 127/165 2023-11-28T17:39:45.385 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 159/165 2023-11-28T17:39:45.430 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T17:39:45.430 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T17:39:45.437 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 127/165 2023-11-28T17:39:45.437 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:45.437 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-28T17:39:45.437 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T17:39:45.438 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T17:39:45.438 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:39:45.495 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7531.g47c1c 128/165 2023-11-28T17:39:45.584 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7531.g47c1c 128/165 2023-11-28T17:39:45.584 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:45.584 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-28T17:39:45.584 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:39:45.654 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 129/165 2023-11-28T17:39:45.733 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 129/165 2023-11-28T17:39:45.733 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:45.733 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-11-28T17:39:45.733 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-28T17:39:45.733 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-28T17:39:45.733 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:39:45.760 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T17:39:45.760 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2023-11-28T17:39:45.902 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2023-11-28T17:39:45.937 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 162/165 2023-11-28T17:39:45.972 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 163/165 2023-11-28T17:39:46.007 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 164/165 2023-11-28T17:39:46.113 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 130/165 2023-11-28T17:39:46.133 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 165/165 2023-11-28T17:39:46.193 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 130/165 2023-11-28T17:39:46.194 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:39:46.194 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-28T17:39:46.194 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T17:39:46.194 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T17:39:46.194 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:39:46.393 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 165/165 2023-11-28T17:39:46.394 INFO:teuthology.orchestra.run.smithi113.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-11-28T17:39:46.394 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:39:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:18.0.0-7531.g47c1c547.el9.x86_64 131/165 2023-11-28T17:39:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 132/165 2023-11-28T17:39:50.592 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:18.0.0-7531.g47c1c547.el9.x86_64 133/165 2023-11-28T17:39:50.628 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 134/165 2023-11-28T17:39:50.912 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : elfutils-libelf-devel-0.190-1.el9.x86_64 135/165 2023-11-28T17:39:51.037 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 136/165 2023-11-28T17:39:51.072 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 137/165 2023-11-28T17:39:51.106 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:18.0.0-7531.g47c1c547.el9.x86_ 138/165 2023-11-28T17:39:51.178 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-debuginfod-client-0.190-1.el9.x86_64 139/165 2023-11-28T17:39:51.210 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-2:18.0.0-7531.g47c1c547.el9.x86_64 1/165 2023-11-28T17:39:51.210 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 2/165 2023-11-28T17:39:51.210 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 3/165 2023-11-28T17:39:51.210 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 4/165 2023-11-28T17:39:51.211 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7531.g47c1c 5/165 2023-11-28T17:39:51.211 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 6/165 2023-11-28T17:39:51.211 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 7/165 2023-11-28T17:39:51.211 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 8/165 2023-11-28T17:39:51.211 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 9/165 2023-11-28T17:39:51.211 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 10/165 2023-11-28T17:39:51.211 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 11/165 2023-11-28T17:39:51.211 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-test-2:18.0.0-7531.g47c1c547.el9.x86_64 12/165 2023-11-28T17:39:51.211 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 13/165 2023-11-28T17:39:51.211 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs-devel-2:18.0.0-7531.g47c1c547.el9.x86_ 14/165 2023-11-28T17:39:51.211 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 15/165 2023-11-28T17:39:51.211 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 16/165 2023-11-28T17:39:51.212 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados-devel-2:18.0.0-7531.g47c1c547.el9.x86_6 17/165 2023-11-28T17:39:51.212 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86 18/165 2023-11-28T17:39:51.212 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 19/165 2023-11-28T17:39:51.212 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7531.g47c1c547.el 20/165 2023-11-28T17:39:51.212 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-common-2:18.0.0-7531.g47c1c547.el9. 21/165 2023-11-28T17:39:51.212 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cephfs-2:18.0.0-7531.g47c1c547.el9.x86_6 22/165 2023-11-28T17:39:51.212 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rados-2:18.0.0-7531.g47c1c547.el9.x86_64 23/165 2023-11-28T17:39:51.212 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rbd-2:18.0.0-7531.g47c1c547.el9.x86_64 24/165 2023-11-28T17:39:51.212 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rgw-2:18.0.0-7531.g47c1c547.el9.x86_64 25/165 2023-11-28T17:39:51.212 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 26/165 2023-11-28T17:39:51.212 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 27/165 2023-11-28T17:39:51.224 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-nbd-2:18.0.0-7531.g47c1c547.el9.x86_64 28/165 2023-11-28T17:39:51.224 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7531.g47c1c547. 29/165 2023-11-28T17:39:51.224 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noa 30/165 2023-11-28T17:39:51.225 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.n 31/165 2023-11-28T17:39:51.225 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c 32/165 2023-11-28T17:39:51.225 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7531.g47c1c547.el 33/165 2023-11-28T17:39:51.225 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 34/165 2023-11-28T17:39:51.225 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7531.g47c1c547.e 35/165 2023-11-28T17:39:51.225 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 36/165 2023-11-28T17:39:51.225 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 37/165 2023-11-28T17:39:51.225 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephfs-top-2:18.0.0-7531.g47c1c547.el9.noarch 38/165 2023-11-28T17:39:51.225 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/165 2023-11-28T17:39:51.225 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/165 2023-11-28T17:39:51.225 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/165 2023-11-28T17:39:51.225 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-devel-0.190-1.el9.x86_64 42/165 2023-11-28T17:39:51.226 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/165 2023-11-28T17:39:51.226 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/165 2023-11-28T17:39:51.226 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/165 2023-11-28T17:39:51.226 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/165 2023-11-28T17:39:51.226 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : jq-1.6-16.el9.x86_64 47/165 2023-11-28T17:39:51.226 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/165 2023-11-28T17:39:51.226 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/165 2023-11-28T17:39:51.226 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/165 2023-11-28T17:39:51.226 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/165 2023-11-28T17:39:51.226 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/165 2023-11-28T17:39:51.226 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 53/165 2023-11-28T17:39:51.226 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/165 2023-11-28T17:39:51.226 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/165 2023-11-28T17:39:51.227 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/165 2023-11-28T17:39:51.227 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 57/165 2023-11-28T17:39:51.227 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 58/165 2023-11-28T17:39:51.227 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 59/165 2023-11-28T17:39:51.227 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/165 2023-11-28T17:39:51.227 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/165 2023-11-28T17:39:51.227 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/165 2023-11-28T17:39:51.227 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/165 2023-11-28T17:39:51.227 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/165 2023-11-28T17:39:51.227 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 65/165 2023-11-28T17:39:51.227 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 66/165 2023-11-28T17:39:51.227 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 67/165 2023-11-28T17:39:51.228 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 68/165 2023-11-28T17:39:51.228 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 69/165 2023-11-28T17:39:51.228 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 70/165 2023-11-28T17:39:51.228 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 71/165 2023-11-28T17:39:51.228 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 72/165 2023-11-28T17:39:51.228 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/165 2023-11-28T17:39:51.228 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 74/165 2023-11-28T17:39:51.228 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 75/165 2023-11-28T17:39:51.228 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 76/165 2023-11-28T17:39:51.228 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 77/165 2023-11-28T17:39:51.228 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 78/165 2023-11-28T17:39:51.228 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 79/165 2023-11-28T17:39:51.228 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 80/165 2023-11-28T17:39:51.229 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 81/165 2023-11-28T17:39:51.229 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 82/165 2023-11-28T17:39:51.229 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 83/165 2023-11-28T17:39:51.229 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 84/165 2023-11-28T17:39:51.229 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 85/165 2023-11-28T17:39:51.229 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 86/165 2023-11-28T17:39:51.229 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 87/165 2023-11-28T17:39:51.229 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 88/165 2023-11-28T17:39:51.229 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 89/165 2023-11-28T17:39:51.229 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ply-3.11-14.el9.noarch 90/165 2023-11-28T17:39:51.229 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 91/165 2023-11-28T17:39:51.229 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 92/165 2023-11-28T17:39:51.230 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 93/165 2023-11-28T17:39:51.230 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 94/165 2023-11-28T17:39:51.230 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/165 2023-11-28T17:39:51.230 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/165 2023-11-28T17:39:51.230 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/165 2023-11-28T17:39:51.230 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 98/165 2023-11-28T17:39:51.230 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 99/165 2023-11-28T17:39:51.230 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/165 2023-11-28T17:39:51.230 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/165 2023-11-28T17:39:51.230 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/165 2023-11-28T17:39:51.230 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 103/165 2023-11-28T17:39:51.230 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/165 2023-11-28T17:39:51.231 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/165 2023-11-28T17:39:51.231 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/165 2023-11-28T17:39:51.231 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/165 2023-11-28T17:39:51.231 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/165 2023-11-28T17:39:51.231 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/165 2023-11-28T17:39:51.231 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 110/165 2023-11-28T17:39:51.231 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/165 2023-11-28T17:39:51.231 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/165 2023-11-28T17:39:51.231 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/165 2023-11-28T17:39:51.231 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/165 2023-11-28T17:39:51.231 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/165 2023-11-28T17:39:51.231 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/165 2023-11-28T17:39:51.232 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/165 2023-11-28T17:39:51.232 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/165 2023-11-28T17:39:51.232 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/165 2023-11-28T17:39:51.232 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/165 2023-11-28T17:39:51.232 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/165 2023-11-28T17:39:51.232 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/165 2023-11-28T17:39:51.232 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/165 2023-11-28T17:39:51.232 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/165 2023-11-28T17:39:51.232 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/165 2023-11-28T17:39:51.232 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/165 2023-11-28T17:39:51.232 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/165 2023-11-28T17:39:51.232 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/165 2023-11-28T17:39:51.232 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/165 2023-11-28T17:39:51.233 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/165 2023-11-28T17:39:51.233 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/165 2023-11-28T17:39:51.233 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/165 2023-11-28T17:39:51.233 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/165 2023-11-28T17:39:51.233 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/165 2023-11-28T17:39:51.233 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/165 2023-11-28T17:39:51.233 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 136/165 2023-11-28T17:39:51.233 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/165 2023-11-28T17:39:51.233 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 138/165 2023-11-28T17:39:51.233 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/165 2023-11-28T17:39:51.233 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/165 2023-11-28T17:39:51.233 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/165 2023-11-28T17:39:51.234 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-1:1.45.6-1.el9.x86_64 142/165 2023-11-28T17:39:51.234 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 143/165 2023-11-28T17:39:51.234 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-libnm-1:1.45.6-1.el9.x86_64 144/165 2023-11-28T17:39:51.234 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 145/165 2023-11-28T17:39:51.234 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-team-1:1.45.6-1.el9.x86_64 146/165 2023-11-28T17:39:51.234 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/165 2023-11-28T17:39:51.234 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-tui-1:1.45.6-1.el9.x86_64 148/165 2023-11-28T17:39:51.235 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 149/165 2023-11-28T17:39:51.235 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : acl-2.3.1-4.el9.x86_64 150/165 2023-11-28T17:39:51.235 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : acl-2.3.1-3.el9.x86_64 151/165 2023-11-28T17:39:51.235 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-debuginfod-client-0.190-1.el9.x86_64 152/165 2023-11-28T17:39:51.235 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/165 2023-11-28T17:39:51.235 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-0.190-1.el9.x86_64 154/165 2023-11-28T17:39:51.235 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 155/165 2023-11-28T17:39:51.235 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libs-0.190-1.el9.x86_64 156/165 2023-11-28T17:39:51.235 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 157/165 2023-11-28T17:39:51.235 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 158/165 2023-11-28T17:39:51.235 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 159/165 2023-11-28T17:39:51.235 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 160/165 2023-11-28T17:39:51.235 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 161/165 2023-11-28T17:39:51.236 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 162/165 2023-11-28T17:39:51.236 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 163/165 2023-11-28T17:39:51.236 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 164/165 2023-11-28T17:39:51.236 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-team-1:1.45.6-1.el9.x86_64 140/165 2023-11-28T17:39:51.318 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-tui-1:1.45.6-1.el9.x86_64 141/165 2023-11-28T17:39:51.382 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 142/165 2023-11-28T17:39:51.425 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 143/165 2023-11-28T17:39:52.610 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T17:39:52.657 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T17:39:52.715 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T17:39:53.372 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 165/165 2023-11-28T17:39:53.372 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:39:53.372 INFO:teuthology.orchestra.run.smithi113.stdout:Upgraded: 2023-11-28T17:39:53.372 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-1:1.45.6-1.el9.x86_64 2023-11-28T17:39:53.372 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-libnm-1:1.45.6-1.el9.x86_64 2023-11-28T17:39:53.372 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-team-1:1.45.6-1.el9.x86_64 2023-11-28T17:39:53.372 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-tui-1:1.45.6-1.el9.x86_64 2023-11-28T17:39:53.373 INFO:teuthology.orchestra.run.smithi113.stdout: acl-2.3.1-4.el9.x86_64 2023-11-28T17:39:53.373 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-debuginfod-client-0.190-1.el9.x86_64 2023-11-28T17:39:53.373 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-0.190-1.el9.x86_64 2023-11-28T17:39:53.373 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libs-0.190-1.el9.x86_64 2023-11-28T17:39:53.373 INFO:teuthology.orchestra.run.smithi113.stdout: libacl-2.3.1-4.el9.x86_64 2023-11-28T17:39:53.373 INFO:teuthology.orchestra.run.smithi113.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-11-28T17:39:53.373 INFO:teuthology.orchestra.run.smithi113.stdout: librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.373 INFO:teuthology.orchestra.run.smithi113.stdout: librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.373 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-1:3.0.7-25.el9.x86_64 2023-11-28T17:39:53.373 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2023-11-28T17:39:53.373 INFO:teuthology.orchestra.run.smithi113.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-11-28T17:39:53.374 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2023-11-28T17:39:53.374 INFO:teuthology.orchestra.run.smithi113.stdout: automake-1.16.2-8.el9.noarch 2023-11-28T17:39:53.374 INFO:teuthology.orchestra.run.smithi113.stdout: bison-3.7.4-5.el9.x86_64 2023-11-28T17:39:53.374 INFO:teuthology.orchestra.run.smithi113.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-28T17:39:53.374 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.374 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.374 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.374 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.374 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:39:53.374 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.375 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.375 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.375 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:39:53.375 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:39:53.375 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:39:53.375 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:39:53.375 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:39:53.375 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.375 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.375 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:39:53.376 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.376 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.376 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.376 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:39:53.376 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:39:53.376 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.376 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-top-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:39:53.377 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-devel-0.190-1.el9.x86_64 2023-11-28T17:39:53.377 INFO:teuthology.orchestra.run.smithi113.stdout: flex-2.6.4-9.el9.x86_64 2023-11-28T17:39:53.377 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-28T17:39:53.377 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-28T17:39:53.377 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-28T17:39:53.377 INFO:teuthology.orchestra.run.smithi113.stdout: fuse-2.9.9-15.el9.x86_64 2023-11-28T17:39:53.377 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-28T17:39:53.377 INFO:teuthology.orchestra.run.smithi113.stdout: jq-1.6-16.el9.x86_64 2023-11-28T17:39:53.377 INFO:teuthology.orchestra.run.smithi113.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-11-28T17:39:53.377 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-28T17:39:53.378 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-28T17:39:53.378 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.378 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.378 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.378 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-28T17:39:53.378 INFO:teuthology.orchestra.run.smithi113.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-28T17:39:53.378 INFO:teuthology.orchestra.run.smithi113.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-28T17:39:53.379 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-28T17:39:53.379 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-28T17:39:53.379 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.379 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.379 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-28T17:39:53.379 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.379 INFO:teuthology.orchestra.run.smithi113.stdout: libtool-2.4.6-45.el9.x86_64 2023-11-28T17:39:53.379 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-28T17:39:53.379 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-11-28T17:39:53.379 INFO:teuthology.orchestra.run.smithi113.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-11-28T17:39:53.379 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-28T17:39:53.379 INFO:teuthology.orchestra.run.smithi113.stdout: lua-5.4.4-4.el9.x86_64 2023-11-28T17:39:53.380 INFO:teuthology.orchestra.run.smithi113.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-11-28T17:39:53.380 INFO:teuthology.orchestra.run.smithi113.stdout: luarocks-3.9.2-1.el9.noarch 2023-11-28T17:39:53.380 INFO:teuthology.orchestra.run.smithi113.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-28T17:39:53.380 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-28T17:39:53.380 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-28T17:39:53.380 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2023-11-28T17:39:53.380 INFO:teuthology.orchestra.run.smithi113.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-28T17:39:53.381 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-13.11-1.el9.x86_64 2023-11-28T17:39:53.381 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-11-28T17:39:53.381 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-11-28T17:39:53.381 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-server-13.11-1.el9.x86_64 2023-11-28T17:39:53.381 INFO:teuthology.orchestra.run.smithi113.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-28T17:39:53.381 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-28T17:39:53.381 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-28T17:39:53.382 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-28T17:39:53.382 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.382 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.382 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.382 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-28T17:39:53.382 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-28T17:39:53.382 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-28T17:39:53.382 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-28T17:39:53.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-28T17:39:53.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-28T17:39:53.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-11-28T17:39:53.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-28T17:39:53.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-28T17:39:53.383 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-28T17:39:53.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-28T17:39:53.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-28T17:39:53.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-28T17:39:53.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-28T17:39:53.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-28T17:39:53.384 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-28T17:39:53.385 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-28T17:39:53.385 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-28T17:39:53.385 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-28T17:39:53.385 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-28T17:39:53.385 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-28T17:39:53.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-28T17:39:53.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-28T17:39:53.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-28T17:39:53.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-28T17:39:53.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-28T17:39:53.386 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-28T17:39:53.387 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ply-3.11-14.el9.noarch 2023-11-28T17:39:53.387 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-28T17:39:53.387 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-28T17:39:53.387 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-28T17:39:53.387 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-28T17:39:53.387 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-28T17:39:53.388 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-28T17:39:53.388 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-28T17:39:53.388 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.388 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.388 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-28T17:39:53.388 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-28T17:39:53.388 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-28T17:39:53.389 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.389 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-28T17:39:53.389 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-28T17:39:53.389 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-28T17:39:53.389 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-28T17:39:53.389 INFO:teuthology.orchestra.run.smithi113.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-28T17:39:53.389 INFO:teuthology.orchestra.run.smithi113.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-28T17:39:53.389 INFO:teuthology.orchestra.run.smithi113.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-28T17:39:53.389 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-28T17:39:53.389 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-28T17:39:53.389 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-28T17:39:53.389 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-28T17:39:53.390 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.390 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.390 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:39:53.390 INFO:teuthology.orchestra.run.smithi113.stdout: re2-1:20211101-3.el9.x86_64 2023-11-28T17:39:53.390 INFO:teuthology.orchestra.run.smithi113.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-28T17:39:53.390 INFO:teuthology.orchestra.run.smithi113.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-28T17:39:53.390 INFO:teuthology.orchestra.run.smithi113.stdout: uuid-1.6.2-55.el9.x86_64 2023-11-28T17:39:53.390 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-11-28T17:39:53.390 INFO:teuthology.orchestra.run.smithi113.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-28T17:39:53.391 INFO:teuthology.orchestra.run.smithi113.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-11-28T17:39:53.391 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:39:53.391 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2023-11-28T17:39:53.983 DEBUG:teuthology.parallel:result is None 2023-11-28T17:39:53.996 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-top-2:18.0.0-7531.g47c1c547.el9.noarch 145/165 2023-11-28T17:39:54.144 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 146/165 2023-11-28T17:39:54.246 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 147/165 2023-11-28T17:39:54.320 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/165 2023-11-28T17:39:54.491 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/165 2023-11-28T17:39:54.533 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/165 2023-11-28T17:39:54.568 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 151/165 2023-11-28T17:39:54.601 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 152/165 2023-11-28T17:39:54.643 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 153/165 2023-11-28T17:39:54.644 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 154/165 2023-11-28T17:39:54.775 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 154/165 2023-11-28T17:39:54.817 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 155/165 2023-11-28T17:39:54.864 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 156/165 2023-11-28T17:39:54.896 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 157/165 2023-11-28T17:39:54.896 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 158/165 2023-11-28T17:39:55.026 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 158/165 2023-11-28T17:39:55.056 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 159/165 2023-11-28T17:39:55.095 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T17:39:55.095 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T17:39:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T17:39:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2023-11-28T17:39:55.540 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2023-11-28T17:39:55.574 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 162/165 2023-11-28T17:39:55.617 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 163/165 2023-11-28T17:39:55.652 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 164/165 2023-11-28T17:39:55.789 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 165/165 2023-11-28T17:39:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 165/165 2023-11-28T17:39:56.036 INFO:teuthology.orchestra.run.smithi098.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-11-28T17:39:56.036 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:00.881 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:18.0.0-7531.g47c1c547.el9.x86_64 1/165 2023-11-28T17:40:00.881 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 2/165 2023-11-28T17:40:00.881 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 3/165 2023-11-28T17:40:00.881 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 4/165 2023-11-28T17:40:00.881 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7531.g47c1c 5/165 2023-11-28T17:40:00.881 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 6/165 2023-11-28T17:40:00.881 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 7/165 2023-11-28T17:40:00.881 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 8/165 2023-11-28T17:40:00.882 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 9/165 2023-11-28T17:40:00.882 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 10/165 2023-11-28T17:40:00.882 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 11/165 2023-11-28T17:40:00.882 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:18.0.0-7531.g47c1c547.el9.x86_64 12/165 2023-11-28T17:40:00.882 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 13/165 2023-11-28T17:40:00.882 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:18.0.0-7531.g47c1c547.el9.x86_ 14/165 2023-11-28T17:40:00.882 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 15/165 2023-11-28T17:40:00.882 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 16/165 2023-11-28T17:40:00.882 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:18.0.0-7531.g47c1c547.el9.x86_6 17/165 2023-11-28T17:40:00.882 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86 18/165 2023-11-28T17:40:00.882 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 19/165 2023-11-28T17:40:00.882 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7531.g47c1c547.el 20/165 2023-11-28T17:40:00.883 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:18.0.0-7531.g47c1c547.el9. 21/165 2023-11-28T17:40:00.883 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:18.0.0-7531.g47c1c547.el9.x86_6 22/165 2023-11-28T17:40:00.883 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:18.0.0-7531.g47c1c547.el9.x86_64 23/165 2023-11-28T17:40:00.883 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:18.0.0-7531.g47c1c547.el9.x86_64 24/165 2023-11-28T17:40:00.883 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:18.0.0-7531.g47c1c547.el9.x86_64 25/165 2023-11-28T17:40:00.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 26/165 2023-11-28T17:40:00.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 27/165 2023-11-28T17:40:00.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:18.0.0-7531.g47c1c547.el9.x86_64 28/165 2023-11-28T17:40:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7531.g47c1c547. 29/165 2023-11-28T17:40:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noa 30/165 2023-11-28T17:40:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.n 31/165 2023-11-28T17:40:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c 32/165 2023-11-28T17:40:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7531.g47c1c547.el 33/165 2023-11-28T17:40:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 34/165 2023-11-28T17:40:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7531.g47c1c547.e 35/165 2023-11-28T17:40:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 36/165 2023-11-28T17:40:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 37/165 2023-11-28T17:40:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-top-2:18.0.0-7531.g47c1c547.el9.noarch 38/165 2023-11-28T17:40:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/165 2023-11-28T17:40:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/165 2023-11-28T17:40:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/165 2023-11-28T17:40:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-devel-0.190-1.el9.x86_64 42/165 2023-11-28T17:40:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/165 2023-11-28T17:40:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/165 2023-11-28T17:40:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/165 2023-11-28T17:40:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/165 2023-11-28T17:40:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : jq-1.6-16.el9.x86_64 47/165 2023-11-28T17:40:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/165 2023-11-28T17:40:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/165 2023-11-28T17:40:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/165 2023-11-28T17:40:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/165 2023-11-28T17:40:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/165 2023-11-28T17:40:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 53/165 2023-11-28T17:40:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/165 2023-11-28T17:40:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/165 2023-11-28T17:40:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/165 2023-11-28T17:40:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 57/165 2023-11-28T17:40:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 58/165 2023-11-28T17:40:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 59/165 2023-11-28T17:40:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/165 2023-11-28T17:40:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/165 2023-11-28T17:40:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/165 2023-11-28T17:40:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/165 2023-11-28T17:40:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/165 2023-11-28T17:40:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 65/165 2023-11-28T17:40:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 66/165 2023-11-28T17:40:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 67/165 2023-11-28T17:40:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 68/165 2023-11-28T17:40:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 69/165 2023-11-28T17:40:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 70/165 2023-11-28T17:40:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 71/165 2023-11-28T17:40:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 72/165 2023-11-28T17:40:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/165 2023-11-28T17:40:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 74/165 2023-11-28T17:40:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 75/165 2023-11-28T17:40:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 76/165 2023-11-28T17:40:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 77/165 2023-11-28T17:40:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 78/165 2023-11-28T17:40:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 79/165 2023-11-28T17:40:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 80/165 2023-11-28T17:40:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 81/165 2023-11-28T17:40:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 82/165 2023-11-28T17:40:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 83/165 2023-11-28T17:40:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 84/165 2023-11-28T17:40:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 85/165 2023-11-28T17:40:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 86/165 2023-11-28T17:40:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 87/165 2023-11-28T17:40:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 88/165 2023-11-28T17:40:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 89/165 2023-11-28T17:40:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 90/165 2023-11-28T17:40:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 91/165 2023-11-28T17:40:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 92/165 2023-11-28T17:40:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 93/165 2023-11-28T17:40:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 94/165 2023-11-28T17:40:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/165 2023-11-28T17:40:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/165 2023-11-28T17:40:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/165 2023-11-28T17:40:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 98/165 2023-11-28T17:40:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 99/165 2023-11-28T17:40:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/165 2023-11-28T17:40:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/165 2023-11-28T17:40:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/165 2023-11-28T17:40:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 103/165 2023-11-28T17:40:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/165 2023-11-28T17:40:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/165 2023-11-28T17:40:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/165 2023-11-28T17:40:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/165 2023-11-28T17:40:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/165 2023-11-28T17:40:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/165 2023-11-28T17:40:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 110/165 2023-11-28T17:40:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/165 2023-11-28T17:40:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/165 2023-11-28T17:40:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/165 2023-11-28T17:40:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/165 2023-11-28T17:40:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/165 2023-11-28T17:40:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/165 2023-11-28T17:40:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/165 2023-11-28T17:40:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/165 2023-11-28T17:40:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/165 2023-11-28T17:40:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/165 2023-11-28T17:40:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/165 2023-11-28T17:40:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/165 2023-11-28T17:40:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/165 2023-11-28T17:40:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/165 2023-11-28T17:40:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/165 2023-11-28T17:40:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/165 2023-11-28T17:40:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/165 2023-11-28T17:40:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/165 2023-11-28T17:40:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/165 2023-11-28T17:40:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/165 2023-11-28T17:40:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/165 2023-11-28T17:40:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/165 2023-11-28T17:40:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/165 2023-11-28T17:40:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/165 2023-11-28T17:40:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/165 2023-11-28T17:40:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 136/165 2023-11-28T17:40:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/165 2023-11-28T17:40:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 138/165 2023-11-28T17:40:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/165 2023-11-28T17:40:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/165 2023-11-28T17:40:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/165 2023-11-28T17:40:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.45.6-1.el9.x86_64 142/165 2023-11-28T17:40:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 143/165 2023-11-28T17:40:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.45.6-1.el9.x86_64 144/165 2023-11-28T17:40:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 145/165 2023-11-28T17:40:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.45.6-1.el9.x86_64 146/165 2023-11-28T17:40:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/165 2023-11-28T17:40:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.45.6-1.el9.x86_64 148/165 2023-11-28T17:40:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 149/165 2023-11-28T17:40:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : acl-2.3.1-4.el9.x86_64 150/165 2023-11-28T17:40:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : acl-2.3.1-3.el9.x86_64 151/165 2023-11-28T17:40:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.190-1.el9.x86_64 152/165 2023-11-28T17:40:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/165 2023-11-28T17:40:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.190-1.el9.x86_64 154/165 2023-11-28T17:40:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 155/165 2023-11-28T17:40:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.190-1.el9.x86_64 156/165 2023-11-28T17:40:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 157/165 2023-11-28T17:40:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 158/165 2023-11-28T17:40:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 159/165 2023-11-28T17:40:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 160/165 2023-11-28T17:40:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 161/165 2023-11-28T17:40:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 162/165 2023-11-28T17:40:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 163/165 2023-11-28T17:40:00.908 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 164/165 2023-11-28T17:40:03.019 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 165/165 2023-11-28T17:40:03.019 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:03.019 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2023-11-28T17:40:03.019 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-1:1.45.6-1.el9.x86_64 2023-11-28T17:40:03.019 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm-1:1.45.6-1.el9.x86_64 2023-11-28T17:40:03.020 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team-1:1.45.6-1.el9.x86_64 2023-11-28T17:40:03.020 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui-1:1.45.6-1.el9.x86_64 2023-11-28T17:40:03.020 INFO:teuthology.orchestra.run.smithi098.stdout: acl-2.3.1-4.el9.x86_64 2023-11-28T17:40:03.020 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client-0.190-1.el9.x86_64 2023-11-28T17:40:03.020 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-0.190-1.el9.x86_64 2023-11-28T17:40:03.020 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs-0.190-1.el9.x86_64 2023-11-28T17:40:03.020 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-2.3.1-4.el9.x86_64 2023-11-28T17:40:03.020 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-11-28T17:40:03.020 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.020 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.020 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-1:3.0.7-25.el9.x86_64 2023-11-28T17:40:03.020 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2023-11-28T17:40:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-11-28T17:40:03.021 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2023-11-28T17:40:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: automake-1.16.2-8.el9.noarch 2023-11-28T17:40:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: bison-3.7.4-5.el9.x86_64 2023-11-28T17:40:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-28T17:40:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:40:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:40:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:40:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:40:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:40:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:40:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:40:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:40:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:40:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:40:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel-0.190-1.el9.x86_64 2023-11-28T17:40:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: flex-2.6.4-9.el9.x86_64 2023-11-28T17:40:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-28T17:40:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-28T17:40:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-28T17:40:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: fuse-2.9.9-15.el9.x86_64 2023-11-28T17:40:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-28T17:40:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: jq-1.6-16.el9.x86_64 2023-11-28T17:40:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-11-28T17:40:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-28T17:40:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-28T17:40:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-28T17:40:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-28T17:40:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-28T17:40:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-28T17:40:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-28T17:40:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-28T17:40:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: libtool-2.4.6-45.el9.x86_64 2023-11-28T17:40:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-28T17:40:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-11-28T17:40:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-11-28T17:40:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-28T17:40:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2023-11-28T17:40:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-11-28T17:40:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-1.el9.noarch 2023-11-28T17:40:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-28T17:40:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-28T17:40:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-28T17:40:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2023-11-28T17:40:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-28T17:40:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-13.11-1.el9.x86_64 2023-11-28T17:40:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-11-28T17:40:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-11-28T17:40:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server-13.11-1.el9.x86_64 2023-11-28T17:40:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-28T17:40:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-28T17:40:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-28T17:40:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-28T17:40:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-28T17:40:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-28T17:40:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-28T17:40:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-28T17:40:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-28T17:40:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-28T17:40:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-11-28T17:40:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-28T17:40:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-28T17:40:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-28T17:40:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-28T17:40:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-28T17:40:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-28T17:40:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-28T17:40:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-28T17:40:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-28T17:40:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-28T17:40:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-28T17:40:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-28T17:40:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-28T17:40:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-28T17:40:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-28T17:40:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-28T17:40:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-28T17:40:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-28T17:40:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-28T17:40:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-28T17:40:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2023-11-28T17:40:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-28T17:40:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-28T17:40:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-28T17:40:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-28T17:40:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-28T17:40:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-28T17:40:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-28T17:40:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-28T17:40:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-28T17:40:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-28T17:40:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-28T17:40:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-28T17:40:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-28T17:40:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-28T17:40:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-28T17:40:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-28T17:40:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-28T17:40:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-28T17:40:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-28T17:40:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-28T17:40:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-28T17:40:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:40:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2023-11-28T17:40:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-28T17:40:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-28T17:40:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: uuid-1.6.2-55.el9.x86_64 2023-11-28T17:40:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-11-28T17:40:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-28T17:40:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-11-28T17:40:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:03.034 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2023-11-28T17:40:03.619 DEBUG:teuthology.parallel:result is None 2023-11-28T17:40:03.620 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T17:40:03.761 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-28T17:40:03.812 INFO:teuthology.orchestra.run.smithi098.stdout:18.0.0-7531.g47c1c547.el9 2023-11-28T17:40:03.813 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7531.g47c1c547.el9 2023-11-28T17:40:03.813 INFO:teuthology.task.install:The correct ceph version 18.0.0-7531.g47c1c547 is installed. 2023-11-28T17:40:03.813 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T17:40:03.952 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-28T17:40:04.016 INFO:teuthology.orchestra.run.smithi113.stdout:18.0.0-7531.g47c1c547.el9 2023-11-28T17:40:04.017 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7531.g47c1c547.el9 2023-11-28T17:40:04.017 INFO:teuthology.task.install:The correct ceph version 18.0.0-7531.g47c1c547 is installed. 2023-11-28T17:40:04.017 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-28T17:40:04.017 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:04.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-28T17:40:04.103 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:04.103 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-28T17:40:04.188 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-28T17:40:04.189 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:04.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2023-11-28T17:40:04.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-28T17:40:04.378 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:04.378 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/daemon-helper 2023-11-28T17:40:04.475 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-28T17:40:04.569 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-28T17:40:04.570 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:04.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-28T17:40:04.642 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-28T17:40:04.753 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:04.753 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-28T17:40:04.850 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-28T17:40:04.987 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-28T17:40:04.988 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:04.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2023-11-28T17:40:05.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-28T17:40:05.192 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:05.193 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/stdin-killer 2023-11-28T17:40:05.299 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-28T17:40:05.393 INFO:teuthology.run_tasks:Running task ceph... 2023-11-28T17:40:05.467 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-28T17:40:05.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2023-11-28T17:40:05.474 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 777 /var/log/ceph 2023-11-28T17:40:05.574 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-28T17:40:05.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-28T17:40:05.580 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-28T17:40:05.676 INFO:tasks.ceph:Creating extra log directories... 2023-11-28T17:40:05.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-28T17:40:05.681 DEBUG:teuthology.orchestra.run.smithi113:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-28T17:40:05.785 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-11-28T17:40:05.786 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-11-28T17:40:05.786 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-11-28T17:40:05.786 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:05.786 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-28T17:40:05.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-28T17:40:06.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-28T17:40:06.120 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:06.120 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-28T17:40:06.180 DEBUG:teuthology.orchestra.run.smithi113:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-28T17:40:06.452 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-28T17:40:06.561 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-28T17:40:06.561 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'overall HEALTH_', '\\(MON_DOWN\\)', 'SLOW_OPS', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-28T17:40:06.562 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398', 'branch': 'wip-vshankar-testing-20231127.102654', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7469398', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20231127.102654'}, 'ceph': {'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'overall HEALTH_', '\\(MON_DOWN\\)', 'SLOW_OPS', 'slow request'], 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20231127.102654', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 1098, 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'sleep_before_teardown': 0, 'subset': '854/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20231127.102654', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi098.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+iCcxfSf82GEbV3qrLN2zk3B72kftTGkorTEp9DLx11Cnbjrg3wMKm4Bv4ZWsqV9EvbYu0IFCdKCD0aBeVm94=', 'smithi113.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdQNw65LjyaXDZutgOR30XOHDfvOBDbuZ+sVTLSEfsOXxgHHCpJpOhUpLvdBhD8YpfEX13hi6Gj6cLSu9FU5IE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'overall HEALTH_', '\\(MON_DOWN\\)', 'SLOW_OPS', 'slow request'], 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'mon_thrash': {'check_mds_failover': True, 'revive_delay': 20, 'thrash_delay': 10}}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '5ae09a5cc7064ca2bf994df19f3006b79794789a', 'timestamp': '2023-11-28_06:23:42', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-11-28T17:40:06.562 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-28T17:40:06.567 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-28T17:40:06.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2023-11-28T17:40:06.655 DEBUG:teuthology.orchestra.run.smithi113:> sudo install -d -m0777 -- /var/run/ceph 2023-11-28T17:40:06.757 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:06.757 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2023-11-28T17:40:06.798 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-28T17:40:06.798 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2023-11-28T17:40:06.880 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-28T17:40:06.881 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:40:06.881 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 632 Links: 1 2023-11-28T17:40:06.881 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:40:06.881 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:40:06.881 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-11-28 17:39:56.491003803 +0000 2023-11-28T17:40:06.881 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-11-28 17:37:18.762293472 +0000 2023-11-28T17:40:06.881 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-11-28 17:37:18.762293472 +0000 2023-11-28T17:40:06.881 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2023-11-28 17:37:18.762293472 +0000 2023-11-28T17:40:06.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-28T17:40:06.990 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-11-28T17:40:06.990 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-11-28T17:40:06.990 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000332179 s, 1.5 MB/s 2023-11-28T17:40:06.993 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-28T17:40:07.035 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2023-11-28T17:40:07.117 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-28T17:40:07.117 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:40:07.117 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 653 Links: 1 2023-11-28T17:40:07.117 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:40:07.117 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:40:07.117 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-11-28 17:39:56.491003803 +0000 2023-11-28T17:40:07.117 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-11-28 17:37:19.786280238 +0000 2023-11-28T17:40:07.117 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-11-28 17:37:19.786280238 +0000 2023-11-28T17:40:07.118 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2023-11-28 17:37:19.786280238 +0000 2023-11-28T17:40:07.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-28T17:40:07.229 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-11-28T17:40:07.229 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-11-28T17:40:07.229 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000310489 s, 1.6 MB/s 2023-11-28T17:40:07.232 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-28T17:40:07.271 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2023-11-28T17:40:07.353 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-28T17:40:07.354 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:40:07.354 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 672 Links: 1 2023-11-28T17:40:07.354 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:40:07.354 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:40:07.354 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-11-28 17:39:56.492003790 +0000 2023-11-28T17:40:07.354 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-11-28 17:37:20.804267082 +0000 2023-11-28T17:40:07.354 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-11-28 17:37:20.805267069 +0000 2023-11-28T17:40:07.354 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2023-11-28 17:37:20.804267082 +0000 2023-11-28T17:40:07.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-28T17:40:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-11-28T17:40:07.469 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-11-28T17:40:07.469 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.00028126 s, 1.8 MB/s 2023-11-28T17:40:07.469 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-28T17:40:07.516 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2023-11-28T17:40:07.554 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-28T17:40:07.555 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:40:07.555 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 691 Links: 1 2023-11-28T17:40:07.555 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:40:07.555 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:40:07.555 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-11-28 17:39:56.493003777 +0000 2023-11-28T17:40:07.555 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-11-28 17:37:21.792254313 +0000 2023-11-28T17:40:07.555 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-11-28 17:37:21.792254313 +0000 2023-11-28T17:40:07.555 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2023-11-28 17:37:21.792254313 +0000 2023-11-28T17:40:07.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-28T17:40:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-11-28T17:40:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-11-28T17:40:07.665 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000305214 s, 1.7 MB/s 2023-11-28T17:40:07.671 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-28T17:40:07.716 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T17:40:07.716 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:07.717 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/scratch_devs of=/dev/stdout 2023-11-28T17:40:07.776 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-28T17:40:07.777 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_1 2023-11-28T17:40:07.830 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-28T17:40:07.830 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:40:07.830 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 631 Links: 1 2023-11-28T17:40:07.830 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:40:07.830 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:40:07.830 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2023-11-28 17:39:46.850827777 +0000 2023-11-28T17:40:07.830 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2023-11-28 17:37:18.772634521 +0000 2023-11-28T17:40:07.830 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2023-11-28 17:37:18.772634521 +0000 2023-11-28T17:40:07.830 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2023-11-28 17:37:18.772634521 +0000 2023-11-28T17:40:07.831 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-28T17:40:07.913 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2023-11-28T17:40:07.913 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2023-11-28T17:40:07.913 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000349956 s, 1.5 MB/s 2023-11-28T17:40:07.916 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-28T17:40:07.971 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_2 2023-11-28T17:40:08.022 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-28T17:40:08.022 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:40:08.022 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 650 Links: 1 2023-11-28T17:40:08.022 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:40:08.022 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:40:08.023 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2023-11-28 17:39:46.851827757 +0000 2023-11-28T17:40:08.023 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2023-11-28 17:37:19.833612346 +0000 2023-11-28T17:40:08.023 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2023-11-28 17:37:19.833612346 +0000 2023-11-28T17:40:08.023 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2023-11-28 17:37:19.833612346 +0000 2023-11-28T17:40:08.023 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-28T17:40:08.107 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2023-11-28T17:40:08.107 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2023-11-28T17:40:08.107 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000352852 s, 1.5 MB/s 2023-11-28T17:40:08.110 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-28T17:40:08.164 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_3 2023-11-28T17:40:08.216 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-28T17:40:08.216 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:40:08.217 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 672 Links: 1 2023-11-28T17:40:08.217 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:40:08.217 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:40:08.217 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2023-11-28 17:39:46.852827736 +0000 2023-11-28T17:40:08.217 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2023-11-28 17:37:20.827591572 +0000 2023-11-28T17:40:08.217 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2023-11-28 17:37:20.828591552 +0000 2023-11-28T17:40:08.217 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2023-11-28 17:37:20.827591572 +0000 2023-11-28T17:40:08.217 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-28T17:40:08.306 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2023-11-28T17:40:08.307 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2023-11-28T17:40:08.307 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000353936 s, 1.4 MB/s 2023-11-28T17:40:08.310 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-28T17:40:08.365 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_4 2023-11-28T17:40:08.414 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-28T17:40:08.414 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:40:08.414 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 691 Links: 1 2023-11-28T17:40:08.414 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:40:08.414 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:40:08.414 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2023-11-28 17:39:46.852827736 +0000 2023-11-28T17:40:08.415 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2023-11-28 17:37:21.850570192 +0000 2023-11-28T17:40:08.415 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2023-11-28 17:37:21.850570192 +0000 2023-11-28T17:40:08.415 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2023-11-28 17:37:21.850570192 +0000 2023-11-28T17:40:08.415 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-28T17:40:08.504 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2023-11-28T17:40:08.505 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2023-11-28T17:40:08.505 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000361352 s, 1.4 MB/s 2023-11-28T17:40:08.508 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-28T17:40:08.561 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T17:40:08.562 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi113.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-28T17:40:08.562 INFO:tasks.ceph:Generating config... 2023-11-28T17:40:08.563 INFO:tasks.ceph:[client] client mount timeout = 600 2023-11-28T17:40:08.563 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2023-11-28T17:40:08.563 INFO:tasks.ceph:[client] debug client = 20 2023-11-28T17:40:08.563 INFO:tasks.ceph:[client] debug ms = 1 2023-11-28T17:40:08.563 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-11-28T17:40:08.564 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-11-28T17:40:08.564 INFO:tasks.ceph:[mds] debug mds = 20 2023-11-28T17:40:08.564 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-11-28T17:40:08.564 INFO:tasks.ceph:[mds] debug ms = 1 2023-11-28T17:40:08.564 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-11-28T17:40:08.564 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-11-28T17:40:08.565 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-11-28T17:40:08.565 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-11-28T17:40:08.565 INFO:tasks.ceph:[mds] mds debug frag = True 2023-11-28T17:40:08.565 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-11-28T17:40:08.565 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = False 2023-11-28T17:40:08.566 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-11-28T17:40:08.566 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-11-28T17:40:08.566 INFO:tasks.ceph:[mds] mds_dir_prefetch = False 2023-11-28T17:40:08.566 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-11-28T17:40:08.566 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2023-11-28T17:40:08.566 INFO:tasks.ceph:[mds] ms inject delay max = 1 2023-11-28T17:40:08.567 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2023-11-28T17:40:08.567 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2023-11-28T17:40:08.567 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2023-11-28T17:40:08.567 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-11-28T17:40:08.567 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-11-28T17:40:08.568 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-11-28T17:40:08.568 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-28T17:40:08.568 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-28T17:40:08.568 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-28T17:40:08.568 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-28T17:40:08.568 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-28T17:40:08.569 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-11-28T17:40:08.569 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-28T17:40:08.569 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-11-28T17:40:08.569 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-28T17:40:08.569 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-11-28T17:40:08.570 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-11-28T17:40:08.570 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-28T17:40:08.570 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-28T17:40:08.570 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-28T17:40:08.570 INFO:tasks.ceph:[osd] filestore flush min = 0 2023-11-28T17:40:08.571 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2023-11-28T17:40:08.571 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-28T17:40:08.571 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-28T17:40:08.571 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-28T17:40:08.571 INFO:tasks.ceph:[osd] ms inject delay max = 1 2023-11-28T17:40:08.572 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2023-11-28T17:40:08.572 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2023-11-28T17:40:08.572 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2023-11-28T17:40:08.572 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-11-28T17:40:08.572 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-28T17:40:08.573 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-11-28T17:40:08.573 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-28T17:40:08.573 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-11-28T17:40:08.574 INFO:tasks.ceph:Setting up mon.a... 2023-11-28T17:40:08.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-28T17:40:08.714 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.keyring 2023-11-28T17:40:08.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-28T17:40:08.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T17:40:08.926 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.98'), ('mon.b', '172.21.15.113'), ('mon.c', '[v2:172.21.15.113:3301,v1:172.21.15.113:6790]')] 2023-11-28T17:40:08.926 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.98,172.21.15.113,[v2:172.21.15.113:3301,v1:172.21.15.113:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-11-28T17:40:08.927 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:08.927 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-28T17:40:08.974 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.98 --add b 172.21.15.113 --addv c '[v2:172.21.15.113:3301,v1:172.21.15.113:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:40:09.106 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:40:09.106 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: generated fsid a317ea31-3452-4bbd-814c-526dd33c0124 2023-11-28T17:40:09.106 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = pacific 2023-11-28T17:40:09.106 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 0 2023-11-28T17:40:09.106 INFO:teuthology.orchestra.run.smithi098.stdout:fsid a317ea31-3452-4bbd-814c-526dd33c0124 2023-11-28T17:40:09.107 INFO:teuthology.orchestra.run.smithi098.stdout:last_changed 2023-11-28T17:40:09.105954+0000 2023-11-28T17:40:09.107 INFO:teuthology.orchestra.run.smithi098.stdout:created 2023-11-28T17:40:09.105954+0000 2023-11-28T17:40:09.107 INFO:teuthology.orchestra.run.smithi098.stdout:min_mon_release 16 (pacific) 2023-11-28T17:40:09.107 INFO:teuthology.orchestra.run.smithi098.stdout:election_strategy: 1 2023-11-28T17:40:09.107 INFO:teuthology.orchestra.run.smithi098.stdout:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2023-11-28T17:40:09.107 INFO:teuthology.orchestra.run.smithi098.stdout:1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.b 2023-11-28T17:40:09.107 INFO:teuthology.orchestra.run.smithi098.stdout:2: [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] mon.c 2023-11-28T17:40:09.107 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-11-28T17:40:09.109 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-28T17:40:09.155 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a317ea31-3452-4bbd-814c-526dd33c0124... 2023-11-28T17:40:09.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-28T17:40:09.200 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-28T17:40:09.354 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2023-11-28T17:40:09.355 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2023-11-28T17:40:09.355 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-28T17:40:09.355 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2023-11-28T17:40:09.355 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.355 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2023-11-28T17:40:09.355 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.355 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2023-11-28T17:40:09.355 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.355 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2023-11-28T17:40:09.355 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2023-11-28T17:40:09.355 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.356 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2023-11-28T17:40:09.356 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2023-11-28T17:40:09.356 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.356 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-28T17:40:09.356 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2023-11-28T17:40:09.356 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.356 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2023-11-28T17:40:09.356 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-28T17:40:09.357 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.357 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2023-11-28T17:40:09.357 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.357 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2023-11-28T17:40:09.357 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2023-11-28T17:40:09.357 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2023-11-28T17:40:09.357 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2023-11-28T17:40:09.357 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2023-11-28T17:40:09.357 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2023-11-28T17:40:09.357 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2023-11-28T17:40:09.357 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-28T17:40:09.358 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-28T17:40:09.358 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2023-11-28T17:40:09.358 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.358 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-28T17:40:09.358 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.358 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2023-11-28T17:40:09.358 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.358 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2023-11-28T17:40:09.358 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.358 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2023-11-28T17:40:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2023-11-28T17:40:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2023-11-28T17:40:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = "172.21.15.98,172.21.15.113,[v2:172.21.15.113:3301,v1:172.21.15.113:6790]" 2023-11-28T17:40:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = a317ea31-3452-4bbd-814c-526dd33c0124 2023-11-28T17:40:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.360 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2023-11-28T17:40:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2023-11-28T17:40:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2023-11-28T17:40:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2023-11-28T17:40:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock profile = high_recovery_ops 2023-11-28T17:40:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2023-11-28T17:40:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2023-11-28T17:40:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2023-11-28T17:40:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2023-11-28T17:40:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2023-11-28T17:40:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug trim objects = true 2023-11-28T17:40:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2023-11-28T17:40:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2023-11-28T17:40:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2023-11-28T17:40:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2023-11-28T17:40:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2023-11-28T17:40:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2023-11-28T17:40:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2023-11-28T17:40:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2023-11-28T17:40:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2023-11-28T17:40:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2023-11-28T17:40:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore compression mode = aggressive 2023-11-28T17:40:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2023-11-28T17:40:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 20 2023-11-28T17:40:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 20 2023-11-28T17:40:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2023-11-28T17:40:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2023-11-28T17:40:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 10 2023-11-28T17:40:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: filestore flush min = 0 2023-11-28T17:40:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: mon client directed command retry = 5 2023-11-28T17:40:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2023-11-28T17:40:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2023-11-28T17:40:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2023-11-28T17:40:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject delay max = 1 2023-11-28T17:40:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject delay probability = 0.005 2023-11-28T17:40:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject delay type = client mds 2023-11-28T17:40:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject socket failures = 2500 2023-11-28T17:40:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: osd blocked scrub grace period = 3600 2023-11-28T17:40:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2023-11-28T17:40:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: osd max markdown count = 1000 2023-11-28T17:40:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2023-11-28T17:40:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: osd op complaint time = 180 2023-11-28T17:40:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.367 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2023-11-28T17:40:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2023-11-28T17:40:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2023-11-28T17:40:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2023-11-28T17:40:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2023-11-28T17:40:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2023-11-28T17:40:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2023-11-28T17:40:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2023-11-28T17:40:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.368 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2023-11-28T17:40:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2023-11-28T17:40:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2023-11-28T17:40:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2023-11-28T17:40:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2023-11-28T17:40:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2023-11-28T17:40:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2023-11-28T17:40:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2023-11-28T17:40:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2023-11-28T17:40:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2023-11-28T17:40:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2023-11-28T17:40:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-28T17:40:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2023-11-28T17:40:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2023-11-28T17:40:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2023-11-28T17:40:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-28T17:40:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-28T17:40:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2023-11-28T17:40:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2023-11-28T17:40:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2023-11-28T17:40:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: mon op complaint time = 120 2023-11-28T17:40:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:40:09.370 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2023-11-28T17:40:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2023-11-28T17:40:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2023-11-28T17:40:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2023-11-28T17:40:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-28T17:40:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-28T17:40:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: client mount timeout = 600 2023-11-28T17:40:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: client_shutdown_timeout = 120 2023-11-28T17:40:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: debug client = 20 2023-11-28T17:40:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2023-11-28T17:40:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: rados mon op timeout = 900 2023-11-28T17:40:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: rados osd op timeout = 900 2023-11-28T17:40:09.373 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2023-11-28T17:40:09.373 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.b] 2023-11-28T17:40:09.373 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.c] 2023-11-28T17:40:09.373 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a] 2023-11-28T17:40:09.373 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c] 2023-11-28T17:40:09.373 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.e] 2023-11-28T17:40:09.373 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b] 2023-11-28T17:40:09.373 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d] 2023-11-28T17:40:09.373 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.f] 2023-11-28T17:40:09.373 INFO:teuthology.orchestra.run.smithi098.stdout:[mds] 2023-11-28T17:40:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: debug mds = 20 2023-11-28T17:40:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: debug mds balancer = 20 2023-11-28T17:40:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2023-11-28T17:40:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: mds bal fragment size max = 10000 2023-11-28T17:40:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: mds bal merge size = 5 2023-11-28T17:40:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: mds bal split bits = 3 2023-11-28T17:40:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: mds bal split size = 100 2023-11-28T17:40:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: mds debug frag = True 2023-11-28T17:40:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: mds debug scatterstat = True 2023-11-28T17:40:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: mds oft prefetch dirfrags = False 2023-11-28T17:40:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: mds op complaint time = 180 2023-11-28T17:40:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: mds verify scatter = True 2023-11-28T17:40:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: mds_dir_prefetch = False 2023-11-28T17:40:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: mds_sleep_rank_change = 5000000.0 2023-11-28T17:40:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: mon client directed command retry = 5 2023-11-28T17:40:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject delay max = 1 2023-11-28T17:40:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject delay probability = 0.005 2023-11-28T17:40:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject delay type = client mds osd 2023-11-28T17:40:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject socket failures = 2500 2023-11-28T17:40:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: osd op complaint time = 180 2023-11-28T17:40:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: rados mon op timeout = 900 2023-11-28T17:40:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: rados osd op timeout = 900 2023-11-28T17:40:09.378 INFO:teuthology.orchestra.run.smithi113.stdout:[global] 2023-11-28T17:40:09.378 INFO:teuthology.orchestra.run.smithi113.stdout: chdir = "" 2023-11-28T17:40:09.378 INFO:teuthology.orchestra.run.smithi113.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-28T17:40:09.378 INFO:teuthology.orchestra.run.smithi113.stdout: auth supported = cephx 2023-11-28T17:40:09.379 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.379 INFO:teuthology.orchestra.run.smithi113.stdout: filestore xattr use omap = true 2023-11-28T17:40:09.379 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.379 INFO:teuthology.orchestra.run.smithi113.stdout: mon clock drift allowed = 1.000 2023-11-28T17:40:09.379 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.379 INFO:teuthology.orchestra.run.smithi113.stdout: osd crush chooseleaf type = 0 2023-11-28T17:40:09.379 INFO:teuthology.orchestra.run.smithi113.stdout: auth debug = true 2023-11-28T17:40:09.379 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.379 INFO:teuthology.orchestra.run.smithi113.stdout: ms die on old message = true 2023-11-28T17:40:09.380 INFO:teuthology.orchestra.run.smithi113.stdout: ms die on bug = true 2023-11-28T17:40:09.380 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.380 INFO:teuthology.orchestra.run.smithi113.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-28T17:40:09.380 INFO:teuthology.orchestra.run.smithi113.stdout: mon pg warn max object skew = 0 2023-11-28T17:40:09.380 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.381 INFO:teuthology.orchestra.run.smithi113.stdout: # disable pg_autoscaler by default for new pools 2023-11-28T17:40:09.381 INFO:teuthology.orchestra.run.smithi113.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-28T17:40:09.382 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.382 INFO:teuthology.orchestra.run.smithi113.stdout: osd pool default size = 2 2023-11-28T17:40:09.382 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.383 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd allow primary affinity = true 2023-11-28T17:40:09.383 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd allow pg remap = true 2023-11-28T17:40:09.383 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on legacy crush tunables = false 2023-11-28T17:40:09.383 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on crush straw calc version zero = false 2023-11-28T17:40:09.383 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on no sortbitwise = false 2023-11-28T17:40:09.384 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on osd down out interval zero = false 2023-11-28T17:40:09.384 INFO:teuthology.orchestra.run.smithi113.stdout: mon warn on too few osds = false 2023-11-28T17:40:09.384 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-28T17:40:09.384 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-28T17:40:09.385 INFO:teuthology.orchestra.run.smithi113.stdout: mon_allow_pool_size_one = true 2023-11-28T17:40:09.385 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.385 INFO:teuthology.orchestra.run.smithi113.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-28T17:40:09.385 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.385 INFO:teuthology.orchestra.run.smithi113.stdout: osd default data pool replay window = 5 2023-11-28T17:40:09.385 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.386 INFO:teuthology.orchestra.run.smithi113.stdout: mon allow pool delete = true 2023-11-28T17:40:09.386 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.386 INFO:teuthology.orchestra.run.smithi113.stdout: mon cluster log file level = debug 2023-11-28T17:40:09.386 INFO:teuthology.orchestra.run.smithi113.stdout: debug asserts on shutdown = true 2023-11-28T17:40:09.386 INFO:teuthology.orchestra.run.smithi113.stdout: mon health detail to clog = false 2023-11-28T17:40:09.386 INFO:teuthology.orchestra.run.smithi113.stdout: mon host = "172.21.15.98,172.21.15.113,[v2:172.21.15.113:3301,v1:172.21.15.113:6790]" 2023-11-28T17:40:09.386 INFO:teuthology.orchestra.run.smithi113.stdout: fsid = a317ea31-3452-4bbd-814c-526dd33c0124 2023-11-28T17:40:09.386 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.386 INFO:teuthology.orchestra.run.smithi113.stdout:[osd] 2023-11-28T17:40:09.386 INFO:teuthology.orchestra.run.smithi113.stdout: osd journal size = 100 2023-11-28T17:40:09.387 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.387 INFO:teuthology.orchestra.run.smithi113.stdout: osd scrub load threshold = 5.0 2023-11-28T17:40:09.387 INFO:teuthology.orchestra.run.smithi113.stdout: osd scrub max interval = 600 2023-11-28T17:40:09.387 INFO:teuthology.orchestra.run.smithi113.stdout: osd mclock profile = high_recovery_ops 2023-11-28T17:40:09.387 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.387 INFO:teuthology.orchestra.run.smithi113.stdout: osd recover clone overlap = true 2023-11-28T17:40:09.387 INFO:teuthology.orchestra.run.smithi113.stdout: osd recovery max chunk = 1048576 2023-11-28T17:40:09.387 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.387 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug shutdown = true 2023-11-28T17:40:09.387 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug op order = true 2023-11-28T17:40:09.388 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug verify stray on activate = true 2023-11-28T17:40:09.388 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug trim objects = true 2023-11-28T17:40:09.388 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.388 INFO:teuthology.orchestra.run.smithi113.stdout: osd open classes on start = true 2023-11-28T17:40:09.388 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug pg log writeout = true 2023-11-28T17:40:09.388 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.388 INFO:teuthology.orchestra.run.smithi113.stdout: osd deep scrub update digest min age = 30 2023-11-28T17:40:09.388 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.388 INFO:teuthology.orchestra.run.smithi113.stdout: osd map max advance = 10 2023-11-28T17:40:09.388 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.389 INFO:teuthology.orchestra.run.smithi113.stdout: journal zero on create = true 2023-11-28T17:40:09.389 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.389 INFO:teuthology.orchestra.run.smithi113.stdout: filestore ondisk finisher threads = 3 2023-11-28T17:40:09.389 INFO:teuthology.orchestra.run.smithi113.stdout: filestore apply finisher threads = 3 2023-11-28T17:40:09.389 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.389 INFO:teuthology.orchestra.run.smithi113.stdout: bdev debug aio = true 2023-11-28T17:40:09.389 INFO:teuthology.orchestra.run.smithi113.stdout: osd debug misdirected ops = true 2023-11-28T17:40:09.389 INFO:teuthology.orchestra.run.smithi113.stdout: bluestore block size = 96636764160 2023-11-28T17:40:09.389 INFO:teuthology.orchestra.run.smithi113.stdout: bluestore compression mode = aggressive 2023-11-28T17:40:09.389 INFO:teuthology.orchestra.run.smithi113.stdout: bluestore fsck on mount = True 2023-11-28T17:40:09.390 INFO:teuthology.orchestra.run.smithi113.stdout: debug bluefs = 20 2023-11-28T17:40:09.390 INFO:teuthology.orchestra.run.smithi113.stdout: debug bluestore = 20 2023-11-28T17:40:09.390 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2023-11-28T17:40:09.390 INFO:teuthology.orchestra.run.smithi113.stdout: debug osd = 20 2023-11-28T17:40:09.390 INFO:teuthology.orchestra.run.smithi113.stdout: debug rocksdb = 10 2023-11-28T17:40:09.390 INFO:teuthology.orchestra.run.smithi113.stdout: filestore flush min = 0 2023-11-28T17:40:09.390 INFO:teuthology.orchestra.run.smithi113.stdout: mon client directed command retry = 5 2023-11-28T17:40:09.390 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd backfillfull_ratio = 0.85 2023-11-28T17:40:09.390 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd full ratio = 0.9 2023-11-28T17:40:09.391 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd nearfull ratio = 0.8 2023-11-28T17:40:09.391 INFO:teuthology.orchestra.run.smithi113.stdout: ms inject delay max = 1 2023-11-28T17:40:09.391 INFO:teuthology.orchestra.run.smithi113.stdout: ms inject delay probability = 0.005 2023-11-28T17:40:09.391 INFO:teuthology.orchestra.run.smithi113.stdout: ms inject delay type = client mds 2023-11-28T17:40:09.391 INFO:teuthology.orchestra.run.smithi113.stdout: ms inject socket failures = 2500 2023-11-28T17:40:09.391 INFO:teuthology.orchestra.run.smithi113.stdout: osd blocked scrub grace period = 3600 2023-11-28T17:40:09.391 INFO:teuthology.orchestra.run.smithi113.stdout: osd failsafe full ratio = 0.95 2023-11-28T17:40:09.391 INFO:teuthology.orchestra.run.smithi113.stdout: osd max markdown count = 1000 2023-11-28T17:40:09.391 INFO:teuthology.orchestra.run.smithi113.stdout: osd objectstore = bluestore 2023-11-28T17:40:09.391 INFO:teuthology.orchestra.run.smithi113.stdout: osd op complaint time = 180 2023-11-28T17:40:09.392 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.392 INFO:teuthology.orchestra.run.smithi113.stdout:[mgr] 2023-11-28T17:40:09.392 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2023-11-28T17:40:09.392 INFO:teuthology.orchestra.run.smithi113.stdout: debug mgr = 20 2023-11-28T17:40:09.392 INFO:teuthology.orchestra.run.smithi113.stdout: debug mon = 20 2023-11-28T17:40:09.392 INFO:teuthology.orchestra.run.smithi113.stdout: debug auth = 20 2023-11-28T17:40:09.392 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min pgs per osd = 4 2023-11-28T17:40:09.393 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min bytes per osd = 10 2023-11-28T17:40:09.393 INFO:teuthology.orchestra.run.smithi113.stdout: mgr/telemetry/nag = false 2023-11-28T17:40:09.393 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.393 INFO:teuthology.orchestra.run.smithi113.stdout:[mon] 2023-11-28T17:40:09.393 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2023-11-28T17:40:09.393 INFO:teuthology.orchestra.run.smithi113.stdout: debug mon = 20 2023-11-28T17:40:09.393 INFO:teuthology.orchestra.run.smithi113.stdout: debug paxos = 20 2023-11-28T17:40:09.394 INFO:teuthology.orchestra.run.smithi113.stdout: debug auth = 20 2023-11-28T17:40:09.394 INFO:teuthology.orchestra.run.smithi113.stdout: mon data avail warn = 5 2023-11-28T17:40:09.394 INFO:teuthology.orchestra.run.smithi113.stdout: mon mgr mkfs grace = 240 2023-11-28T17:40:09.394 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min pgs per osd = 4 2023-11-28T17:40:09.394 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd reporter subtree level = osd 2023-11-28T17:40:09.394 INFO:teuthology.orchestra.run.smithi113.stdout: mon osd prime pg temp = true 2023-11-28T17:40:09.395 INFO:teuthology.orchestra.run.smithi113.stdout: mon reweight min bytes per osd = 10 2023-11-28T17:40:09.395 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.395 INFO:teuthology.orchestra.run.smithi113.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-28T17:40:09.395 INFO:teuthology.orchestra.run.smithi113.stdout: auth mon ticket ttl = 660 # 11m 2023-11-28T17:40:09.395 INFO:teuthology.orchestra.run.smithi113.stdout: auth service ticket ttl = 240 # 4m 2023-11-28T17:40:09.395 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.396 INFO:teuthology.orchestra.run.smithi113.stdout: # don't complain about insecure global_id in the test suite 2023-11-28T17:40:09.396 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-28T17:40:09.396 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-28T17:40:09.396 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.396 INFO:teuthology.orchestra.run.smithi113.stdout: # 1m isn't quite enough 2023-11-28T17:40:09.396 INFO:teuthology.orchestra.run.smithi113.stdout: mon_down_mkfs_grace = 2m 2023-11-28T17:40:09.396 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.396 INFO:teuthology.orchestra.run.smithi113.stdout: mon_warn_on_filestore_osds = false 2023-11-28T17:40:09.396 INFO:teuthology.orchestra.run.smithi113.stdout: mon op complaint time = 120 2023-11-28T17:40:09.397 INFO:teuthology.orchestra.run.smithi113.stdout: 2023-11-28T17:40:09.397 INFO:teuthology.orchestra.run.smithi113.stdout:[client] 2023-11-28T17:40:09.397 INFO:teuthology.orchestra.run.smithi113.stdout: rgw cache enabled = true 2023-11-28T17:40:09.397 INFO:teuthology.orchestra.run.smithi113.stdout: rgw enable ops log = true 2023-11-28T17:40:09.397 INFO:teuthology.orchestra.run.smithi113.stdout: rgw enable usage log = true 2023-11-28T17:40:09.397 INFO:teuthology.orchestra.run.smithi113.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-28T17:40:09.397 INFO:teuthology.orchestra.run.smithi113.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-28T17:40:09.397 INFO:teuthology.orchestra.run.smithi113.stdout: client mount timeout = 600 2023-11-28T17:40:09.397 INFO:teuthology.orchestra.run.smithi113.stdout: client_shutdown_timeout = 120 2023-11-28T17:40:09.398 INFO:teuthology.orchestra.run.smithi113.stdout: debug client = 20 2023-11-28T17:40:09.398 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2023-11-28T17:40:09.398 INFO:teuthology.orchestra.run.smithi113.stdout: rados mon op timeout = 900 2023-11-28T17:40:09.398 INFO:teuthology.orchestra.run.smithi113.stdout: rados osd op timeout = 900 2023-11-28T17:40:09.398 INFO:teuthology.orchestra.run.smithi113.stdout:[mon.a] 2023-11-28T17:40:09.398 INFO:teuthology.orchestra.run.smithi113.stdout:[mon.b] 2023-11-28T17:40:09.398 INFO:teuthology.orchestra.run.smithi113.stdout:[mon.c] 2023-11-28T17:40:09.398 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.a] 2023-11-28T17:40:09.398 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.c] 2023-11-28T17:40:09.398 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.e] 2023-11-28T17:40:09.399 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.b] 2023-11-28T17:40:09.399 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.d] 2023-11-28T17:40:09.399 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.f] 2023-11-28T17:40:09.399 INFO:teuthology.orchestra.run.smithi113.stdout:[mds] 2023-11-28T17:40:09.399 INFO:teuthology.orchestra.run.smithi113.stdout: debug mds = 20 2023-11-28T17:40:09.399 INFO:teuthology.orchestra.run.smithi113.stdout: debug mds balancer = 20 2023-11-28T17:40:09.399 INFO:teuthology.orchestra.run.smithi113.stdout: debug ms = 1 2023-11-28T17:40:09.399 INFO:teuthology.orchestra.run.smithi113.stdout: mds bal fragment size max = 10000 2023-11-28T17:40:09.399 INFO:teuthology.orchestra.run.smithi113.stdout: mds bal merge size = 5 2023-11-28T17:40:09.399 INFO:teuthology.orchestra.run.smithi113.stdout: mds bal split bits = 3 2023-11-28T17:40:09.399 INFO:teuthology.orchestra.run.smithi113.stdout: mds bal split size = 100 2023-11-28T17:40:09.400 INFO:teuthology.orchestra.run.smithi113.stdout: mds debug frag = True 2023-11-28T17:40:09.400 INFO:teuthology.orchestra.run.smithi113.stdout: mds debug scatterstat = True 2023-11-28T17:40:09.400 INFO:teuthology.orchestra.run.smithi113.stdout: mds oft prefetch dirfrags = False 2023-11-28T17:40:09.400 INFO:teuthology.orchestra.run.smithi113.stdout: mds op complaint time = 180 2023-11-28T17:40:09.400 INFO:teuthology.orchestra.run.smithi113.stdout: mds verify scatter = True 2023-11-28T17:40:09.400 INFO:teuthology.orchestra.run.smithi113.stdout: mds_dir_prefetch = False 2023-11-28T17:40:09.400 INFO:teuthology.orchestra.run.smithi113.stdout: mds_sleep_rank_change = 5000000.0 2023-11-28T17:40:09.400 INFO:teuthology.orchestra.run.smithi113.stdout: mon client directed command retry = 5 2023-11-28T17:40:09.401 INFO:teuthology.orchestra.run.smithi113.stdout: ms inject delay max = 1 2023-11-28T17:40:09.401 INFO:teuthology.orchestra.run.smithi113.stdout: ms inject delay probability = 0.005 2023-11-28T17:40:09.401 INFO:teuthology.orchestra.run.smithi113.stdout: ms inject delay type = client mds osd 2023-11-28T17:40:09.401 INFO:teuthology.orchestra.run.smithi113.stdout: ms inject socket failures = 2500 2023-11-28T17:40:09.401 INFO:teuthology.orchestra.run.smithi113.stdout: osd op complaint time = 180 2023-11-28T17:40:09.401 INFO:teuthology.orchestra.run.smithi113.stdout: rados mon op timeout = 900 2023-11-28T17:40:09.401 INFO:teuthology.orchestra.run.smithi113.stdout: rados osd op timeout = 900 2023-11-28T17:40:09.423 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-28T17:40:09.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-28T17:40:09.573 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-28T17:40:09.573 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:09.573 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-28T17:40:09.621 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:09.621 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-28T17:40:09.713 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2023-11-28T17:40:09.713 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:09.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-28T17:40:09.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T17:40:09.903 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:09.903 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:40:10.002 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi113.front.sepia.ceph.com 2023-11-28T17:40:10.003 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:10.003 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-28T17:40:10.003 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T17:40:10.163 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:10.163 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:40:10.260 INFO:tasks.ceph:Setting up mon nodes... 2023-11-28T17:40:10.260 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-28T17:40:10.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-11-28T17:40:10.449 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-11-28T17:40:10.459 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-28T17:40:10.644 INFO:teuthology.orchestra.run.smithi113.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-28T17:40:10.654 INFO:tasks.ceph:Setting up mds nodes... 2023-11-28T17:40:10.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-11-28T17:40:10.846 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-11-28T17:40:10.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-11-28T17:40:10.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-11-28T17:40:11.134 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-11-28T17:40:11.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-11-28T17:40:11.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-11-28T17:40:11.434 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-11-28T17:40:11.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-11-28T17:40:11.539 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-11-28T17:40:11.730 INFO:teuthology.orchestra.run.smithi113.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-11-28T17:40:11.739 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-11-28T17:40:11.835 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-11-28T17:40:12.015 INFO:teuthology.orchestra.run.smithi113.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-11-28T17:40:12.024 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-11-28T17:40:12.119 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-11-28T17:40:12.303 INFO:teuthology.orchestra.run.smithi113.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-11-28T17:40:12.312 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-11-28T17:40:12.408 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-28T17:40:12.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-28T17:40:12.541 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-28T17:40:12.605 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-28T17:40:12.606 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi113.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-28T17:40:12.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-28T17:40:12.704 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T17:40:12.704 INFO:tasks.ceph:role: osd.0 2023-11-28T17:40:12.704 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com 2023-11-28T17:40:12.704 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-28T17:40:12.808 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:40:12.808 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:40:12.808 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:40:12.808 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:40:12.808 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:40:12.808 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:40:12.808 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:40:12.808 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:40:12.808 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:40:12.809 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:40:13.333 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2023-11-28T17:40:13.339 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2023-11-28T17:40:13.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-28T17:40:13.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-28T17:40:13.622 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-28T17:40:13.708 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T17:40:13.708 INFO:tasks.ceph:role: osd.1 2023-11-28T17:40:13.708 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com 2023-11-28T17:40:13.708 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-28T17:40:13.804 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:40:13.804 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:40:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:40:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:40:13.805 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:40:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:40:13.805 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:40:13.805 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:40:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:40:13.805 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:40:14.288 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2023-11-28T17:40:14.294 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2023-11-28T17:40:14.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-28T17:40:14.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-28T17:40:14.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-28T17:40:14.634 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T17:40:14.635 INFO:tasks.ceph:role: osd.2 2023-11-28T17:40:14.635 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com 2023-11-28T17:40:14.635 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-28T17:40:14.738 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:40:14.738 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:40:14.738 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:40:14.738 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:40:14.738 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:40:14.738 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:40:14.738 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:40:14.738 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:40:14.739 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:40:14.739 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:40:15.261 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2023-11-28T17:40:15.267 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2023-11-28T17:40:15.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-28T17:40:15.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-28T17:40:15.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-11-28T17:40:15.673 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T17:40:15.673 INFO:tasks.ceph:role: osd.3 2023-11-28T17:40:15.673 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com 2023-11-28T17:40:15.673 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-28T17:40:15.772 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:40:15.773 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:40:15.773 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:40:15.773 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:40:15.773 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:40:15.773 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:40:15.773 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:40:15.773 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:40:15.773 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:40:15.773 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:40:16.228 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2023-11-28T17:40:16.235 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2023-11-28T17:40:16.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-11-28T17:40:16.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-11-28T17:40:16.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:40:16.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:16.639+0000 7fb393c14500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-11-28T17:40:16.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:16.642+0000 7fb393c14500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-11-28T17:40:16.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:16.642+0000 7fb393c14500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-28T17:40:16.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:16.642+0000 7fb393c14500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-28T17:40:16.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:16.643+0000 7fb393c14500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-28T17:40:16.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:16.649+0000 7fb393c14500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-11-28T17:40:19.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-28T17:40:19.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:40:19.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:19.809+0000 7f9e83c17500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-11-28T17:40:19.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:19.809+0000 7f9e83c17500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-11-28T17:40:19.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:19.810+0000 7f9e83c17500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-28T17:40:19.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:19.810+0000 7f9e83c17500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-28T17:40:19.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:19.810+0000 7f9e83c17500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-28T17:40:19.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:19.812+0000 7f9e83c17500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-11-28T17:40:22.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-28T17:40:22.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:40:22.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:22.954+0000 7f7bfec38500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-11-28T17:40:22.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:22.955+0000 7f7bfec38500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-11-28T17:40:22.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:22.955+0000 7f7bfec38500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-28T17:40:22.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:22.955+0000 7f7bfec38500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-28T17:40:22.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:22.955+0000 7f7bfec38500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-28T17:40:22.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:22.957+0000 7f7bfec38500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-11-28T17:40:25.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-28T17:40:25.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:40:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:26.130+0000 7fc005002500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-11-28T17:40:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:26.131+0000 7fc005002500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-11-28T17:40:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:26.131+0000 7fc005002500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-28T17:40:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:26.131+0000 7fc005002500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-28T17:40:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:26.131+0000 7fc005002500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-28T17:40:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:26.132+0000 7fc005002500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-11-28T17:40:29.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-11-28T17:40:29.157 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-11-28T17:40:29.260 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T17:40:29.260 INFO:tasks.ceph:role: osd.4 2023-11-28T17:40:29.260 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi113.front.sepia.ceph.com 2023-11-28T17:40:29.261 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-28T17:40:29.359 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:40:29.360 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:40:29.360 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:40:29.360 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:40:29.360 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:40:29.360 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:40:29.360 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:40:29.360 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:40:29.360 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:40:29.360 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:40:29.889 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2023-11-28T17:40:29.896 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2023-11-28T17:40:29.896 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-11-28T17:40:30.057 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-11-28T17:40:30.160 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-11-28T17:40:30.246 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T17:40:30.247 INFO:tasks.ceph:role: osd.5 2023-11-28T17:40:30.247 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi113.front.sepia.ceph.com 2023-11-28T17:40:30.247 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-28T17:40:30.343 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:40:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:40:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:40:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:40:30.343 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:40:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:40:30.343 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:40:30.343 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:40:30.343 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:40:30.344 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:40:30.845 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2023-11-28T17:40:30.852 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2023-11-28T17:40:30.852 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-11-28T17:40:31.006 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-11-28T17:40:31.108 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-11-28T17:40:31.198 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T17:40:31.198 INFO:tasks.ceph:role: osd.6 2023-11-28T17:40:31.198 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi113.front.sepia.ceph.com 2023-11-28T17:40:31.199 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-28T17:40:31.297 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:40:31.297 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:40:31.297 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:40:31.297 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:40:31.297 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:40:31.297 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:40:31.297 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:40:31.298 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:40:31.298 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:40:31.298 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:40:31.822 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2023-11-28T17:40:31.828 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2023-11-28T17:40:31.829 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-11-28T17:40:31.981 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-11-28T17:40:32.081 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-11-28T17:40:32.171 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T17:40:32.171 INFO:tasks.ceph:role: osd.7 2023-11-28T17:40:32.171 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi113.front.sepia.ceph.com 2023-11-28T17:40:32.171 DEBUG:teuthology.orchestra.run.smithi113:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-28T17:40:32.272 INFO:teuthology.orchestra.run.smithi113.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:40:32.272 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:40:32.272 INFO:teuthology.orchestra.run.smithi113.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:40:32.273 INFO:teuthology.orchestra.run.smithi113.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:40:32.273 INFO:teuthology.orchestra.run.smithi113.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:40:32.273 INFO:teuthology.orchestra.run.smithi113.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:40:32.273 INFO:teuthology.orchestra.run.smithi113.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:40:32.273 INFO:teuthology.orchestra.run.smithi113.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:40:32.273 INFO:teuthology.orchestra.run.smithi113.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:40:32.273 INFO:teuthology.orchestra.run.smithi113.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:40:32.815 INFO:teuthology.orchestra.run.smithi113.stdout:Discarding blocks...Done. 2023-11-28T17:40:32.825 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi113.front.sepia.ceph.com -o noatime 2023-11-28T17:40:32.826 DEBUG:teuthology.orchestra.run.smithi113:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-11-28T17:40:32.983 DEBUG:teuthology.orchestra.run.smithi113:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-11-28T17:40:33.086 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:40:33.237 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:33.235+0000 7fec12e0e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-11-28T17:40:33.241 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:33.239+0000 7fec12e0e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-11-28T17:40:33.241 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:33.239+0000 7fec12e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-28T17:40:33.241 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:33.239+0000 7fec12e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-28T17:40:33.241 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:33.239+0000 7fec12e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-28T17:40:33.244 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:33.243+0000 7fec12e0e500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-11-28T17:40:36.181 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-11-28T17:40:36.281 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:40:36.420 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:36.418+0000 7fe6de81e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-11-28T17:40:36.422 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:36.419+0000 7fe6de81e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-11-28T17:40:36.422 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:36.419+0000 7fe6de81e500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-28T17:40:36.422 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:36.419+0000 7fe6de81e500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-28T17:40:36.422 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:36.419+0000 7fe6de81e500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-28T17:40:36.423 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:36.421+0000 7fe6de81e500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-11-28T17:40:36.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:40:36.573 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:40:39.578 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-11-28T17:40:39.679 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:40:39.818 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:39.816+0000 7fbae48f8500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-11-28T17:40:39.818 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:39.816+0000 7fbae48f8500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-11-28T17:40:39.818 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:39.816+0000 7fbae48f8500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-28T17:40:39.819 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:39.817+0000 7fbae48f8500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-28T17:40:39.819 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:39.817+0000 7fbae48f8500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-28T17:40:39.820 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:39.819+0000 7fbae48f8500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-11-28T17:40:42.999 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-11-28T17:40:43.097 DEBUG:teuthology.orchestra.run.smithi113:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:40:43.237 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:43.235+0000 7f2b0561c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-11-28T17:40:43.238 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:43.236+0000 7f2b0561c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-11-28T17:40:43.238 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:43.236+0000 7f2b0561c500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-28T17:40:43.238 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:43.236+0000 7f2b0561c500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-28T17:40:43.238 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:43.236+0000 7f2b0561c500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-28T17:40:43.239 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:43.238+0000 7f2b0561c500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-11-28T17:40:46.412 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-11-28T17:40:46.510 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-28T17:40:46.510 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:46.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-11-28T17:40:46.613 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:46.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-11-28T17:40:46.713 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:46.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-11-28T17:40:46.813 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:46.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-11-28T17:40:46.907 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:46.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-28T17:40:47.050 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:47.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-28T17:40:47.140 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:47.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-28T17:40:47.279 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:47.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-11-28T17:40:47.373 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:47.373 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-28T17:40:47.477 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:47.477 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-11-28T17:40:47.572 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:47.572 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-11-28T17:40:47.667 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:47.668 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-11-28T17:40:47.761 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:47.761 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-11-28T17:40:47.852 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:47.852 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-11-28T17:40:47.996 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:47.997 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-11-28T17:40:48.098 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:48.098 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-11-28T17:40:48.198 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:48.198 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-28T17:40:48.257 INFO:tasks.ceph:Adding keys to all mons... 2023-11-28T17:40:48.257 DEBUG:teuthology.orchestra.run.smithi098:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-28T17:40:48.262 DEBUG:teuthology.orchestra.run.smithi113:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-28T17:40:48.346 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.y] 2023-11-28T17:40:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQD6JWZlZHLFGhAA9wLEYJuMPsAAOMAYwkC+Ag== 2023-11-28T17:40:48.346 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a] 2023-11-28T17:40:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQD6JWZlk0N3MhAAVCvBwdfSRj2L1kvCWizJWA== 2023-11-28T17:40:48.346 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c] 2023-11-28T17:40:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQD7JWZl88UECBAAPpMp0EJnsKlJS4tnE6GkTw== 2023-11-28T17:40:48.346 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.e] 2023-11-28T17:40:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQD7JWZlNjnoGRAAUmQQCZaMgUckmLp5TaD4XQ== 2023-11-28T17:40:48.347 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2023-11-28T17:40:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAAJmZl5Ys2JhAAJlC3qZcthgOcnyKgQ2nsQQ== 2023-11-28T17:40:48.347 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2023-11-28T17:40:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQADJmZlqEBYMBAArvC6HyCpUX/Qyek3hAJuBQ== 2023-11-28T17:40:48.347 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2023-11-28T17:40:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAGJmZlA0MCORAA9vWuzXhKAwpIzbXfxee2yw== 2023-11-28T17:40:48.347 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.3] 2023-11-28T17:40:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAKJmZl/L7VBxAA8hn+a+P0bvxtRVkjRIG8MQ== 2023-11-28T17:40:48.347 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.x] 2023-11-28T17:40:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQD6JWZlE2tlJhAAgrvAG8UorNaewg7pS15hmw== 2023-11-28T17:40:48.347 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b] 2023-11-28T17:40:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQD7JWZluMWFKxAAvOwHPuX4aw9zQhs53TfAFA== 2023-11-28T17:40:48.348 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d] 2023-11-28T17:40:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQD8JWZlefPvABAAgpfrOhsQQ2QNpAVLBWokZQ== 2023-11-28T17:40:48.348 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.f] 2023-11-28T17:40:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQD8JWZlfN0WEhAAgVaUkd1xCZGnFuiH8tbS9w== 2023-11-28T17:40:48.348 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.4] 2023-11-28T17:40:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQARJmZlwoUpDhAAqGPZR6DjkkKm8X5648vpIw== 2023-11-28T17:40:48.348 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.5] 2023-11-28T17:40:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAUJmZlc6oQGRAAPqbpO+SF33lb9/GLtVpS0Q== 2023-11-28T17:40:48.348 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.6] 2023-11-28T17:40:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAXJmZlPqvBMBAAcDn8Bm3RyhhuqkMA5c/WbQ== 2023-11-28T17:40:48.348 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.7] 2023-11-28T17:40:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAbJmZlCJwlDhAA2C82u3/5O/wAKJ1S/pgtOQ== 2023-11-28T17:40:48.348 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2023-11-28T17:40:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQD8JWZlM1lJIBAADfmbwIVsSJBfWoBEwfhc/Q== 2023-11-28T17:40:48.355 INFO:teuthology.orchestra.run.smithi113.stdout:[mgr.y] 2023-11-28T17:40:48.355 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQD6JWZlZHLFGhAA9wLEYJuMPsAAOMAYwkC+Ag== 2023-11-28T17:40:48.355 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.a] 2023-11-28T17:40:48.355 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQD6JWZlk0N3MhAAVCvBwdfSRj2L1kvCWizJWA== 2023-11-28T17:40:48.355 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.c] 2023-11-28T17:40:48.355 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQD7JWZl88UECBAAPpMp0EJnsKlJS4tnE6GkTw== 2023-11-28T17:40:48.355 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.e] 2023-11-28T17:40:48.355 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQD7JWZlNjnoGRAAUmQQCZaMgUckmLp5TaD4XQ== 2023-11-28T17:40:48.355 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.0] 2023-11-28T17:40:48.356 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAAJmZl5Ys2JhAAJlC3qZcthgOcnyKgQ2nsQQ== 2023-11-28T17:40:48.356 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.1] 2023-11-28T17:40:48.356 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQADJmZlqEBYMBAArvC6HyCpUX/Qyek3hAJuBQ== 2023-11-28T17:40:48.356 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.2] 2023-11-28T17:40:48.356 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAGJmZlA0MCORAA9vWuzXhKAwpIzbXfxee2yw== 2023-11-28T17:40:48.356 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.3] 2023-11-28T17:40:48.356 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAKJmZl/L7VBxAA8hn+a+P0bvxtRVkjRIG8MQ== 2023-11-28T17:40:48.356 INFO:teuthology.orchestra.run.smithi113.stdout:[mgr.x] 2023-11-28T17:40:48.357 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQD6JWZlE2tlJhAAgrvAG8UorNaewg7pS15hmw== 2023-11-28T17:40:48.357 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.b] 2023-11-28T17:40:48.357 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQD7JWZluMWFKxAAvOwHPuX4aw9zQhs53TfAFA== 2023-11-28T17:40:48.357 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.d] 2023-11-28T17:40:48.357 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQD8JWZlefPvABAAgpfrOhsQQ2QNpAVLBWokZQ== 2023-11-28T17:40:48.357 INFO:teuthology.orchestra.run.smithi113.stdout:[mds.f] 2023-11-28T17:40:48.357 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQD8JWZlfN0WEhAAgVaUkd1xCZGnFuiH8tbS9w== 2023-11-28T17:40:48.357 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.4] 2023-11-28T17:40:48.358 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQARJmZlwoUpDhAAqGPZR6DjkkKm8X5648vpIw== 2023-11-28T17:40:48.358 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.5] 2023-11-28T17:40:48.358 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAUJmZlc6oQGRAAPqbpO+SF33lb9/GLtVpS0Q== 2023-11-28T17:40:48.358 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.6] 2023-11-28T17:40:48.358 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAXJmZlPqvBMBAAcDn8Bm3RyhhuqkMA5c/WbQ== 2023-11-28T17:40:48.358 INFO:teuthology.orchestra.run.smithi113.stdout:[osd.7] 2023-11-28T17:40:48.358 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQAbJmZlCJwlDhAA2C82u3/5O/wAKJ1S/pgtOQ== 2023-11-28T17:40:48.358 INFO:teuthology.orchestra.run.smithi113.stdout:[client.0] 2023-11-28T17:40:48.358 INFO:teuthology.orchestra.run.smithi113.stdout: key = AQD8JWZlM1lJIBAADfmbwIVsSJBfWoBEwfhc/Q== 2023-11-28T17:40:48.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T17:40:48.399 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T17:40:48.546 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:40:48.555 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:40:48.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:40:48.709 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:40:48.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:40:48.863 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:40:49.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:49.012 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:49.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:49.163 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:49.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:49.310 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:49.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:49.455 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:49.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T17:40:49.599 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T17:40:49.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:40:49.746 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:40:49.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:40:49.897 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:40:50.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:40:50.047 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:40:50.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:50.199 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:50.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:50.351 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:50.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:50.498 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:50.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:50.649 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:40:50.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T17:40:50.792 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T17:40:50.933 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-28T17:40:50.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-28T17:40:51.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T17:40:51.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-28T17:40:51.272 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-11-28T17:40:51.374 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T17:40:51.535 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-11-28T17:40:51.614 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-11-28T17:40:51.705 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T17:40:51.860 DEBUG:teuthology.orchestra.run.smithi113:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-11-28T17:40:51.942 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:40:51.951 DEBUG:teuthology.orchestra.run.smithi113:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:40:52.004 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-28T17:40:52.005 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-28T17:40:52.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-28T17:40:52.010 INFO:tasks.ceph.mon.a:Started 2023-11-28T17:40:52.010 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-28T17:40:52.010 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-28T17:40:52.018 INFO:tasks.ceph.mon.b:Started 2023-11-28T17:40:52.018 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-28T17:40:52.019 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-11-28T17:40:52.027 INFO:tasks.ceph.mon.c:Started 2023-11-28T17:40:52.027 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-28T17:40:52.027 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-11-28T17:40:52.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-11-28T17:40:52.033 INFO:tasks.ceph.mgr.y:Started 2023-11-28T17:40:52.034 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-28T17:40:52.034 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-28T17:40:52.039 INFO:tasks.ceph.mgr.x:Started 2023-11-28T17:40:52.039 INFO:tasks.ceph:Setting crush tunables to default 2023-11-28T17:40:52.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd crush tunables default 2023-11-28T17:40:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.217+0000 7efd3622e640 1 Processor -- start 2023-11-28T17:40:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.217+0000 7efd3622e640 1 -- start start 2023-11-28T17:40:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.218+0000 7efd3622e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd300c7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.218+0000 7efd3622e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 0x7efd30164440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.218+0000 7efd3622e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd300c80f0 con 0x7efd30153c80 2023-11-28T17:40:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.218+0000 7efd3622e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd3015e070 con 0x7efd300c7c60 2023-11-28T17:40:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.218+0000 7efd3622e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3015d230 con 0x7efd300c69b0 2023-11-28T17:40:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.218+0000 7efd3522c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efd30153c80 0x7efd3006b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38732/0 (socket says 172.21.15.98:38732) 2023-11-28T17:40:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.219+0000 7efd3522c640 1 -- 172.21.15.98:0/3198997221 learned_addr learned my addr 172.21.15.98:0/3198997221 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:40:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.219+0000 7efd35a2d640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 0x7efd30164440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.219+0000 7efd35a2d640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 msgr2=0x7efd30164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:40:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.219+0000 7efd35a2d640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 msgr2=0x7efd30164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:40:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.219+0000 7efd35a2d640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 0x7efd30164440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:40:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.219+0000 7efd35a2d640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 0x7efd30164440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:40:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.219+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd300c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.219+0000 7efd34a2b640 1 -- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 msgr2=0x7efd300c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:40:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.219+0000 7efd34a2b640 1 -- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 msgr2=0x7efd300c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:40:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.219+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd300c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:40:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.219+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd300c7720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:40:52.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.420+0000 7efd35a2d640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 0x7efd30164440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:52.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.420+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd300c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:52.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.421+0000 7efd35a2d640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 msgr2=0x7efd30164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:40:52.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.421+0000 7efd35a2d640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 msgr2=0x7efd30164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:40:52.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.421+0000 7efd35a2d640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 0x7efd30164440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:40:52.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.421+0000 7efd34a2b640 1 -- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 msgr2=0x7efd300c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:40:52.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.421+0000 7efd34a2b640 1 -- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 msgr2=0x7efd300c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:40:52.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.421+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd300c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:40:52.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.421+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd300c7720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:40:52.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.421+0000 7efd35a2d640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 0x7efd30164440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:40:52.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.822+0000 7efd35a2d640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 0x7efd30164440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.822+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd300c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.823+0000 7efd35a2d640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 msgr2=0x7efd30164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:40:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.823+0000 7efd35a2d640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 msgr2=0x7efd30164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:40:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.823+0000 7efd35a2d640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 0x7efd30164440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:40:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.823+0000 7efd34a2b640 1 -- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 msgr2=0x7efd300c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:40:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.823+0000 7efd34a2b640 1 -- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 msgr2=0x7efd300c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:40:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.823+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd300c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:40:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.823+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd300c7720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:40:52.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:52.823+0000 7efd35a2d640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 0x7efd30164440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:40:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:53.624+0000 7efd35a2d640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 0x7efd30164440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:53.624+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd300c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:53.625+0000 7efd35a2d640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 msgr2=0x7efd30164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:40:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:53.625+0000 7efd35a2d640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 msgr2=0x7efd30164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:40:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:53.625+0000 7efd35a2d640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 0x7efd30164440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:40:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:53.625+0000 7efd34a2b640 1 -- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 msgr2=0x7efd300c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:40:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:53.625+0000 7efd34a2b640 1 -- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 msgr2=0x7efd300c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:40:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:53.625+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd300c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:40:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:53.625+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd300c7720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:40:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:53.625+0000 7efd35a2d640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 0x7efd30164440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:40:55.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.218+0000 7efd26ffd640 1 -- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 msgr2=0x7efd300c7720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:40:55.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.218+0000 7efd26ffd640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd300c7720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:55.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.219+0000 7efd26ffd640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 msgr2=0x7efd30164440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:40:55.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.219+0000 7efd26ffd640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.98:3300/0 conn(0x7efd300c7c60 0x7efd30164440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.219+0000 7efd26ffd640 1 -- 172.21.15.98:0/3198997221 >> v1:172.21.15.98:6789/0 conn(0x7efd30153c80 legacy=0x7efd3006b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.219+0000 7efd26ffd640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 0x7efd14005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.219+0000 7efd26ffd640 1 -- 172.21.15.98:0/3198997221 --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3015d4f0 con 0x7efd14006200 2023-11-28T17:40:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.219+0000 7efd26ffd640 1 -- 172.21.15.98:0/3198997221 --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd3015e070 con 0x7efd140036a0 2023-11-28T17:40:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.220+0000 7efd3522c640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 0x7efd14005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:55.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.220+0000 7efd3522c640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 msgr2=0x7efd14005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:40:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.220+0000 7efd3522c640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 msgr2=0x7efd14005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:40:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.220+0000 7efd3522c640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 0x7efd14005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:40:55.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.220+0000 7efd3522c640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 0x7efd14005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:40:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.421+0000 7efd3522c640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 0x7efd14005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.422+0000 7efd3522c640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 msgr2=0x7efd14005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:40:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.422+0000 7efd3522c640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 msgr2=0x7efd14005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:40:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.422+0000 7efd3522c640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 0x7efd14005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:40:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.422+0000 7efd3522c640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 0x7efd14005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:40:55.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.823+0000 7efd3522c640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 0x7efd14005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:55.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.824+0000 7efd3522c640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 msgr2=0x7efd14005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:40:55.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.824+0000 7efd3522c640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 msgr2=0x7efd14005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:40:55.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.824+0000 7efd3522c640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 0x7efd14005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:40:55.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:55.824+0000 7efd3522c640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 0x7efd14005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:40:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:56.625+0000 7efd3522c640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 0x7efd14005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:56.626+0000 7efd3522c640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 msgr2=0x7efd14005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:40:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:56.626+0000 7efd3522c640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 msgr2=0x7efd14005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:40:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:56.626+0000 7efd3522c640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 0x7efd14005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:40:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:56.626+0000 7efd3522c640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 0x7efd14005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:40:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.464+0000 7efd27fff640 1 -- 172.21.15.98:0/3198997221 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700210702 0 0) 0x7efd3015d4f0 con 0x7efd14006200 2023-11-28T17:40:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.465+0000 7efd27fff640 1 -- 172.21.15.98:0/3198997221 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd1c003520 con 0x7efd14006200 2023-11-28T17:40:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.469+0000 7efd27fff640 1 -- 172.21.15.98:0/3198997221 <== mon.1 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 9873081 0 0) 0x7efd1c003520 con 0x7efd14006200 2023-11-28T17:40:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.469+0000 7efd27fff640 1 -- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 msgr2=0x7efd14005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:40:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.469+0000 7efd27fff640 1 --2- 172.21.15.98:0/3198997221 >> v2:172.21.15.113:3300/0 conn(0x7efd140036a0 0x7efd14005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.469+0000 7efd27fff640 1 -- 172.21.15.98:0/3198997221 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1400af80 con 0x7efd14006200 2023-11-28T17:40:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.469+0000 7efd27fff640 1 -- 172.21.15.98:0/3198997221 <== mon.1 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7efd2000d580 con 0x7efd14006200 2023-11-28T17:40:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.469+0000 7efd27fff640 1 -- 172.21.15.98:0/3198997221 >> v1:172.21.15.113:6789/0 conn(0x7efd14006200 legacy=0x7efd140086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.469+0000 7efd35a2d640 1 -- 172.21.15.98:0/3198997221 reap_dead start 2023-11-28T17:40:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.469+0000 7efd27fff640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd1c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.469+0000 7efd27fff640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd1c004020 0x7efd1c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.469+0000 7efd27fff640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd1c004e20 0x7efd1c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.469+0000 7efd27fff640 1 -- 172.21.15.98:0/3198997221 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3015e070 con 0x7efd1c004020 2023-11-28T17:40:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.469+0000 7efd27fff640 1 -- 172.21.15.98:0/3198997221 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c005640 con 0x7efd1c004e20 2023-11-28T17:40:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.469+0000 7efd27fff640 1 -- 172.21.15.98:0/3198997221 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c00bec0 con 0x7efd300c69b0 2023-11-28T17:40:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.470+0000 7efd3522c640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd1c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.470+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd1c004020 0x7efd1c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.470+0000 7efd35a2d640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd1c004e20 0x7efd1c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.471+0000 7efd34a2b640 1 -- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 msgr2=0x7efd1c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:40:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.471+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd1c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.471+0000 7efd34a2b640 1 -- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd1c004e20 msgr2=0x7efd1c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:40:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.471+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd1c004e20 0x7efd1c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.471+0000 7efd34a2b640 1 -- 172.21.15.98:0/3198997221 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1c003520 con 0x7efd1c004020 2023-11-28T17:40:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.473+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd1c004020 0x7efd1c0047c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efd20006fd0 tx=0x7efd1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef505a254b83711 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:40:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.474+0000 7efd27fff640 1 -- 172.21.15.98:0/3198997221 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2002d5a0 con 0x7efd1c004020 2023-11-28T17:40:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.475+0000 7efd27fff640 1 -- 172.21.15.98:0/3198997221 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd20031070 con 0x7efd1c004020 2023-11-28T17:40:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.475+0000 7efd27fff640 1 -- 172.21.15.98:0/3198997221 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2002d960 con 0x7efd1c004020 2023-11-28T17:40:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.476+0000 7efd3622e640 1 -- 172.21.15.98:0/3198997221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd1c004020 msgr2=0x7efd1c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.476+0000 7efd3622e640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd1c004020 0x7efd1c0047c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efd20006fd0 tx=0x7efd1c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:40:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.477+0000 7efd3622e640 1 -- 172.21.15.98:0/3198997221 shutdown_connections 2023-11-28T17:40:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.477+0000 7efd3622e640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd1c004020 0x7efd1c0047c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.477+0000 7efd3622e640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd1c0039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.477+0000 7efd3622e640 1 --2- 172.21.15.98:0/3198997221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd1c004e20 0x7efd1c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.477+0000 7efd3622e640 1 -- 172.21.15.98:0/3198997221 >> 172.21.15.98:0/3198997221 conn(0x7efd300cf300 msgr2=0x7efd300cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:40:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.477+0000 7efd3622e640 1 -- 172.21.15.98:0/3198997221 shutdown_connections 2023-11-28T17:40:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.477+0000 7efd3622e640 1 -- 172.21.15.98:0/3198997221 wait complete. 2023-11-28T17:40:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.480+0000 7efd3622e640 1 Processor -- start 2023-11-28T17:40:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.480+0000 7efd3622e640 1 -- start start 2023-11-28T17:40:57.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.482+0000 7efd3622e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd1c004020 0x7efd3015c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:57.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.482+0000 7efd3622e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd1c004e20 0x7efd300ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:57.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.482+0000 7efd3622e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd3014a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:57.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.482+0000 7efd3622e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd30157ce0 con 0x7efd1c004020 2023-11-28T17:40:57.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.482+0000 7efd3622e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd301f0d60 con 0x7efd1c004e20 2023-11-28T17:40:57.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.482+0000 7efd3622e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd301f1020 con 0x7efd300c69b0 2023-11-28T17:40:57.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.482+0000 7efd3522c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd1c004020 0x7efd3015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:57.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.482+0000 7efd3522c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd1c004020 0x7efd3015c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56976/0 (socket says 172.21.15.98:56976) 2023-11-28T17:40:57.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.482+0000 7efd3522c640 1 -- 172.21.15.98:0/3761394261 learned_addr learned my addr 172.21.15.98:0/3761394261 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:40:57.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.483+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3761394261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd1c004e20 0x7efd300ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:57.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.483+0000 7efd3522c640 1 -- 172.21.15.98:0/3761394261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 msgr2=0x7efd3014a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:57.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.483+0000 7efd3522c640 1 --2- 172.21.15.98:0/3761394261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd3014a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:57.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.483+0000 7efd3522c640 1 -- 172.21.15.98:0/3761394261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd1c004e20 msgr2=0x7efd300ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:57.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.483+0000 7efd3522c640 1 --2- 172.21.15.98:0/3761394261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd1c004e20 0x7efd300ce3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:57.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.483+0000 7efd3522c640 1 -- 172.21.15.98:0/3761394261 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd301f1300 con 0x7efd1c004020 2023-11-28T17:40:57.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.483+0000 7efd34a2b640 1 --2- 172.21.15.98:0/3761394261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd1c004e20 0x7efd300ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:40:57.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.483+0000 7efd3522c640 1 --2- 172.21.15.98:0/3761394261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd1c004020 0x7efd3015c310 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efd2c017680 tx=0x7efd2c017b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:40:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.484+0000 7efd25ffb640 1 -- 172.21.15.98:0/3761394261 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c005a80 con 0x7efd1c004020 2023-11-28T17:40:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.484+0000 7efd25ffb640 1 -- 172.21.15.98:0/3761394261 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2c005c00 con 0x7efd1c004020 2023-11-28T17:40:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.484+0000 7efd25ffb640 1 -- 172.21.15.98:0/3761394261 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c015930 con 0x7efd1c004020 2023-11-28T17:40:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.484+0000 7efd3622e640 1 -- 172.21.15.98:0/3761394261 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd301f15e0 con 0x7efd1c004020 2023-11-28T17:40:57.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.484+0000 7efd3622e640 1 -- 172.21.15.98:0/3761394261 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd301f1ba0 con 0x7efd1c004020 2023-11-28T17:40:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.484+0000 7efd25ffb640 1 -- 172.21.15.98:0/3761394261 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7efd2c015a90 con 0x7efd1c004020 2023-11-28T17:40:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.484+0000 7efd25ffb640 1 -- 172.21.15.98:0/3761394261 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7efd2c01d020 con 0x7efd1c004020 2023-11-28T17:40:57.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.485+0000 7efd3622e640 1 -- 172.21.15.98:0/3761394261 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd30074db0 con 0x7efd1c004020 2023-11-28T17:40:57.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.489+0000 7efd25ffb640 1 -- 172.21.15.98:0/3761394261 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7efd2c041ce0 con 0x7efd1c004020 2023-11-28T17:40:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.562+0000 7efd3622e640 1 -- 172.21.15.98:0/3761394261 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7efd300c78f0 con 0x7efd1c004020 2023-11-28T17:40:57.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.565+0000 7efd25ffb640 1 -- 172.21.15.98:0/3761394261 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7efd2c030020 con 0x7efd1c004020 2023-11-28T17:40:57.566 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2023-11-28T17:40:57.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.567+0000 7efd3622e640 1 -- 172.21.15.98:0/3761394261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd1c004020 msgr2=0x7efd3015c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.567+0000 7efd3622e640 1 --2- 172.21.15.98:0/3761394261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd1c004020 0x7efd3015c310 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efd2c017680 tx=0x7efd2c017b40 comp rx=0 tx=0).stop 2023-11-28T17:40:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.567+0000 7efd3622e640 1 -- 172.21.15.98:0/3761394261 shutdown_connections 2023-11-28T17:40:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.567+0000 7efd3622e640 1 --2- 172.21.15.98:0/3761394261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efd300c69b0 0x7efd3014a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.567+0000 7efd3622e640 1 --2- 172.21.15.98:0/3761394261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd1c004e20 0x7efd300ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.567+0000 7efd3622e640 1 --2- 172.21.15.98:0/3761394261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd1c004020 0x7efd3015c310 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.567+0000 7efd3622e640 1 -- 172.21.15.98:0/3761394261 >> 172.21.15.98:0/3761394261 conn(0x7efd300cf300 msgr2=0x7efd300ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:40:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.567+0000 7efd3622e640 1 -- 172.21.15.98:0/3761394261 shutdown_connections 2023-11-28T17:40:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:40:57.567+0000 7efd3622e640 1 -- 172.21.15.98:0/3761394261 wait complete. 2023-11-28T17:40:57.592 INFO:tasks.ceph:check_enable_crimson: False 2023-11-28T17:40:57.592 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-28T17:40:57.592 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:57.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-28T17:40:57.680 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:57.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-28T17:40:57.808 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:57.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-28T17:40:57.890 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:40:57.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-11-28T17:40:58.022 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:58.022 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-11-28T17:40:58.058 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:40:58.057+0000 7feeef248040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T17:40:58.123 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:40:58.122+0000 7feeef248040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T17:40:58.127 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:58.127 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-11-28T17:40:58.218 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:58.218 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-11-28T17:40:58.344 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2023-11-28T17:40:58.344 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-11-28T17:40:58.470 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new 189b7858-08c8-45d8-8834-fa4a3620bbf1 0 2023-11-28T17:40:58.700 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.700+0000 7f7b0b615640 1 Processor -- start 2023-11-28T17:40:58.701 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.700+0000 7f7b0b615640 1 -- start start 2023-11-28T17:40:58.703 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.701+0000 7f7b0b615640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7b041540d0 0x7f7b0415dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:58.703 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.701+0000 7f7b0b615640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7b040ca280 0x7f7b040ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:58.703 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.701+0000 7f7b0b615640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b0406c060 con 0x7f7b040cb070 2023-11-28T17:40:58.703 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.701+0000 7f7b0b615640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b040cbdd0 con 0x7f7b041540d0 2023-11-28T17:40:58.703 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.701+0000 7f7b0b615640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b041544b0 con 0x7f7b040ca280 2023-11-28T17:40:58.703 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.701+0000 7f7b08bb0640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f7b040cb070 0x7f7b0406b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:40126/0 (socket says 172.21.15.113:40126) 2023-11-28T17:40:58.703 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.701+0000 7f7b08bb0640 1 -- 172.21.15.113:0/2317550600 learned_addr learned my addr 172.21.15.113:0/2317550600 (peer_addr_for_me v1:172.21.15.113:0/0) 2023-11-28T17:40:58.703 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.701+0000 7f7b09bb2640 1 --2- 172.21.15.113:0/2317550600 >> v2:172.21.15.113:3300/0 conn(0x7f7b040ca280 0x7f7b040ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:58.703 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.702+0000 7f7b093b1640 1 --2- 172.21.15.113:0/2317550600 >> v2:172.21.15.98:3300/0 conn(0x7f7b041540d0 0x7f7b0415dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:58.703 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.702+0000 7f7afbfff640 1 -- 172.21.15.113:0/2317550600 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709170654 0 0) 0x7f7af4001110 con 0x7f7b040cb070 2023-11-28T17:40:58.704 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.702+0000 7f7afbfff640 1 -- 172.21.15.113:0/2317550600 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7af0003520 con 0x7f7b040cb070 2023-11-28T17:40:58.704 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.702+0000 7f7b09bb2640 1 -- 172.21.15.113:0/2317550600 >> v2:172.21.15.98:3300/0 conn(0x7f7b041540d0 msgr2=0x7f7b0415dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:40:58.704 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.702+0000 7f7b09bb2640 1 --2- 172.21.15.113:0/2317550600 >> v2:172.21.15.98:3300/0 conn(0x7f7b041540d0 0x7f7b0415dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:58.704 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.702+0000 7f7b09bb2640 1 -- 172.21.15.113:0/2317550600 >> v1:172.21.15.113:6789/0 conn(0x7f7b040cb070 legacy=0x7f7b0406b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:58.704 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.702+0000 7f7b09bb2640 1 -- 172.21.15.113:0/2317550600 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0406a660 con 0x7f7b040ca280 2023-11-28T17:40:58.705 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.703+0000 7f7b09bb2640 1 --2- 172.21.15.113:0/2317550600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b040ca280 0x7f7b040ca6d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7b0415aca0 tx=0x7f7afc02bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1847d66dc7b5c2d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:40:58.705 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.703+0000 7f7afbfff640 1 -- 172.21.15.113:0/2317550600 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc02cc70 con 0x7f7b040ca280 2023-11-28T17:40:58.705 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.703+0000 7f7afbfff640 1 -- 172.21.15.113:0/2317550600 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7afc02cdd0 con 0x7f7b040ca280 2023-11-28T17:40:58.705 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.703+0000 7f7afbfff640 1 -- 172.21.15.113:0/2317550600 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc034660 con 0x7f7b040ca280 2023-11-28T17:40:58.705 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.704+0000 7f7b0b615640 1 -- 172.21.15.113:0/2317550600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b040ca280 msgr2=0x7f7b040ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:58.705 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.704+0000 7f7b0b615640 1 --2- 172.21.15.113:0/2317550600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b040ca280 0x7f7b040ca6d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7b0415aca0 tx=0x7f7afc02bec0 comp rx=0 tx=0).stop 2023-11-28T17:40:58.705 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.704+0000 7f7b0b615640 1 -- 172.21.15.113:0/2317550600 shutdown_connections 2023-11-28T17:40:58.706 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.704+0000 7f7b0b615640 1 --2- 172.21.15.113:0/2317550600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b040ca280 0x7f7b040ca6d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:58.706 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.704+0000 7f7b0b615640 1 --2- 172.21.15.113:0/2317550600 >> v2:172.21.15.98:3300/0 conn(0x7f7b041540d0 0x7f7b0415dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:58.706 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.704+0000 7f7b0b615640 1 -- 172.21.15.113:0/2317550600 >> 172.21.15.113:0/2317550600 conn(0x7f7b040cf490 msgr2=0x7f7b040cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:40:58.706 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.704+0000 7f7b0b615640 1 -- 172.21.15.113:0/2317550600 shutdown_connections 2023-11-28T17:40:58.706 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.704+0000 7f7b0b615640 1 -- 172.21.15.113:0/2317550600 wait complete. 2023-11-28T17:40:58.706 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.706+0000 7f7b0b615640 1 Processor -- start 2023-11-28T17:40:58.707 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.706+0000 7f7b0b615640 1 -- start start 2023-11-28T17:40:58.708 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.707+0000 7f7b0b615640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b040ca280 0x7f7b040c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:58.708 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.707+0000 7f7b0b615640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b040cb070 0x7f7b040c37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:58.708 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.707+0000 7f7b0b615640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b041540d0 0x7f7b0414f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:58.708 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.707+0000 7f7b0b615640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b041f2b40 con 0x7f7b040ca280 2023-11-28T17:40:58.708 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.707+0000 7f7b0b615640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b040c60e0 con 0x7f7b040cb070 2023-11-28T17:40:58.709 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.707+0000 7f7b0b615640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b040c5b40 con 0x7f7b041540d0 2023-11-28T17:40:58.709 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.707+0000 7f7b08bb0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b040cb070 0x7f7b040c37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:58.709 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.707+0000 7f7b09bb2640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b041540d0 0x7f7b0414f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:58.709 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.707+0000 7f7b08bb0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b040cb070 0x7f7b040c37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43392/0 (socket says 172.21.15.113:43392) 2023-11-28T17:40:58.709 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.707+0000 7f7b08bb0640 1 -- 172.21.15.113:0/3183027630 learned_addr learned my addr 172.21.15.113:0/3183027630 (peer_addr_for_me v2:172.21.15.113:0/0) 2023-11-28T17:40:58.709 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.707+0000 7f7b093b1640 1 --2- 172.21.15.113:0/3183027630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b040ca280 0x7f7b040c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:58.709 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.708+0000 7f7b08bb0640 1 -- 172.21.15.113:0/3183027630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b041540d0 msgr2=0x7f7b0414f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:58.709 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.708+0000 7f7b08bb0640 1 --2- 172.21.15.113:0/3183027630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b041540d0 0x7f7b0414f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:58.709 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.708+0000 7f7b08bb0640 1 -- 172.21.15.113:0/3183027630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b040ca280 msgr2=0x7f7b040c4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:58.709 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.708+0000 7f7b08bb0640 1 --2- 172.21.15.113:0/3183027630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b040ca280 0x7f7b040c4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:58.709 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.708+0000 7f7b08bb0640 1 -- 172.21.15.113:0/3183027630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc009a70 con 0x7f7b040cb070 2023-11-28T17:40:58.710 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.708+0000 7f7b08bb0640 1 --2- 172.21.15.113:0/3183027630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b040cb070 0x7f7b040c37b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7af400c700 tx=0x7f7af400cb00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:40:58.711 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.708+0000 7f7af9ffb640 1 -- 172.21.15.113:0/3183027630 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af4005ce0 con 0x7f7b040cb070 2023-11-28T17:40:58.711 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.708+0000 7f7af9ffb640 1 -- 172.21.15.113:0/3183027630 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af4005e40 con 0x7f7b040cb070 2023-11-28T17:40:58.711 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.708+0000 7f7af9ffb640 1 -- 172.21.15.113:0/3183027630 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af4007290 con 0x7f7b040cb070 2023-11-28T17:40:58.711 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.708+0000 7f7b0b615640 1 -- 172.21.15.113:0/3183027630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b040c5e00 con 0x7f7b040cb070 2023-11-28T17:40:58.711 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.709+0000 7f7b0b615640 1 -- 172.21.15.113:0/3183027630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b041549f0 con 0x7f7b040cb070 2023-11-28T17:40:58.711 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.709+0000 7f7af9ffb640 1 -- 172.21.15.113:0/3183027630 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7af401c440 con 0x7f7b040cb070 2023-11-28T17:40:58.711 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.709+0000 7f7af9ffb640 1 -- 172.21.15.113:0/3183027630 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7af4012810 con 0x7f7b040cb070 2023-11-28T17:40:58.711 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.710+0000 7f7b0b615640 1 -- 172.21.15.113:0/3183027630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0406ab70 con 0x7f7b040cb070 2023-11-28T17:40:58.712 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:40:58.711+0000 7feeef248040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T17:40:58.713 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.712+0000 7f7af9ffb640 1 -- 172.21.15.113:0/3183027630 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f7b0406ab70 con 0x7f7b040cb070 2023-11-28T17:40:58.772 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:40:58.772+0000 7feeef248040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T17:40:58.791 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.790+0000 7f7b0b615640 1 -- 172.21.15.113:0/3183027630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "189b7858-08c8-45d8-8834-fa4a3620bbf1", "id": 0} v 0) v1 -- 0x7f7b04074f60 con 0x7f7b040cb070 2023-11-28T17:40:58.791 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:40:58.791+0000 7f44faa12040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T17:40:58.798 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.798+0000 7f7af9ffb640 1 -- 172.21.15.113:0/3183027630 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "189b7858-08c8-45d8-8834-fa4a3620bbf1", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7af4017070 con 0x7f7b040cb070 2023-11-28T17:40:58.799 INFO:teuthology.orchestra.run.smithi113.stdout:0 2023-11-28T17:40:58.800 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.799+0000 7f7b0b615640 1 -- 172.21.15.113:0/3183027630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b040cb070 msgr2=0x7f7b040c37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:58.801 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.799+0000 7f7b0b615640 1 --2- 172.21.15.113:0/3183027630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b040cb070 0x7f7b040c37b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7af400c700 tx=0x7f7af400cb00 comp rx=0 tx=0).stop 2023-11-28T17:40:58.801 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.799+0000 7f7b0b615640 1 -- 172.21.15.113:0/3183027630 shutdown_connections 2023-11-28T17:40:58.801 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.799+0000 7f7b0b615640 1 --2- 172.21.15.113:0/3183027630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7b041540d0 0x7f7b0414f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:58.801 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.799+0000 7f7b0b615640 1 --2- 172.21.15.113:0/3183027630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7b040cb070 0x7f7b040c37b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:58.801 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.799+0000 7f7b0b615640 1 --2- 172.21.15.113:0/3183027630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b040ca280 0x7f7b040c4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:58.801 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.799+0000 7f7b0b615640 1 -- 172.21.15.113:0/3183027630 >> 172.21.15.113:0/3183027630 conn(0x7f7b040cf490 msgr2=0x7f7b041529c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:40:58.801 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.800+0000 7f7b0b615640 1 -- 172.21.15.113:0/3183027630 shutdown_connections 2023-11-28T17:40:58.801 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:58.800+0000 7f7b0b615640 1 -- 172.21.15.113:0/3183027630 wait complete. 2023-11-28T17:40:58.817 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new f5a771a0-ab90-45ff-80f9-f62f0ef1a954 1 2023-11-28T17:40:58.824 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:40:58.823+0000 7feeef248040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T17:40:58.858 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:40:58.858+0000 7f44faa12040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T17:40:58.886 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:40:58.885+0000 7feeef248040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T17:40:58.934 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:40:58.934+0000 7feeef248040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T17:40:59.014 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.013+0000 7f41d8a13640 1 Processor -- start 2023-11-28T17:40:59.014 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.013+0000 7f41d8a13640 1 -- start start 2023-11-28T17:40:59.016 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.015+0000 7f41d8a13640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f41d4070260 0x7f41d4154d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.016 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.015+0000 7f41d8a13640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d4070640 0x7f41d41552c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.016 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.015+0000 7f41d8a13640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d40c28c0 con 0x7f41d40c23f0 2023-11-28T17:40:59.016 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.015+0000 7f41d8a13640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d40cd3e0 con 0x7f41d4070260 2023-11-28T17:40:59.016 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.015+0000 7f41d8a13640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d40cd6c0 con 0x7f41d4070640 2023-11-28T17:40:59.016 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.015+0000 7f41d37fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f41d4070260 0x7f41d4154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.016 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.015+0000 7f41d37fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f41d4070260 0x7f41d4154d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43402/0 (socket says 172.21.15.113:43402) 2023-11-28T17:40:59.016 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.015+0000 7f41d37fe640 1 -- 172.21.15.113:0/926900975 learned_addr learned my addr 172.21.15.113:0/926900975 (peer_addr_for_me v2:172.21.15.113:0/0) 2023-11-28T17:40:59.017 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.015+0000 7f41d3fff640 1 --2- 172.21.15.113:0/926900975 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d4070640 0x7f41d41552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.017 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.016+0000 7f41d37fe640 1 -- 172.21.15.113:0/926900975 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d4070640 msgr2=0x7f41d41552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:40:59.017 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.016+0000 7f41d37fe640 1 --2- 172.21.15.113:0/926900975 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d4070640 0x7f41d41552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.017 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.016+0000 7f41d37fe640 1 -- 172.21.15.113:0/926900975 >> v1:172.21.15.113:6789/0 conn(0x7f41d40c23f0 legacy=0x7f41d4157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:59.018 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.016+0000 7f41d37fe640 1 -- 172.21.15.113:0/926900975 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d40cd9a0 con 0x7f41d4070260 2023-11-28T17:40:59.018 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.017+0000 7f41d37fe640 1 --2- 172.21.15.113:0/926900975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d4070260 0x7f41d4154d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f41c4009870 tx=0x7f41c402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1386db57a7607e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:40:59.019 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.017+0000 7f41d2ffd640 1 -- 172.21.15.113:0/926900975 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c4030840 con 0x7f41d4070260 2023-11-28T17:40:59.019 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.017+0000 7f41d2ffd640 1 -- 172.21.15.113:0/926900975 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c40309a0 con 0x7f41d4070260 2023-11-28T17:40:59.019 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.017+0000 7f41d2ffd640 1 -- 172.21.15.113:0/926900975 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c4030d30 con 0x7f41d4070260 2023-11-28T17:40:59.019 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.018+0000 7f41d8a13640 1 -- 172.21.15.113:0/926900975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d4070260 msgr2=0x7f41d4154d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:59.019 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.018+0000 7f41d8a13640 1 --2- 172.21.15.113:0/926900975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d4070260 0x7f41d4154d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f41c4009870 tx=0x7f41c402b980 comp rx=0 tx=0).stop 2023-11-28T17:40:59.019 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.018+0000 7f41d8a13640 1 -- 172.21.15.113:0/926900975 shutdown_connections 2023-11-28T17:40:59.019 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.018+0000 7f41d8a13640 1 --2- 172.21.15.113:0/926900975 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d4070640 0x7f41d41552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.020 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.018+0000 7f41d8a13640 1 --2- 172.21.15.113:0/926900975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d4070260 0x7f41d4154d80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.020 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.018+0000 7f41d8a13640 1 -- 172.21.15.113:0/926900975 >> 172.21.15.113:0/926900975 conn(0x7f41d406a4d0 msgr2=0x7f41d406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:40:59.020 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.018+0000 7f41d8a13640 1 -- 172.21.15.113:0/926900975 shutdown_connections 2023-11-28T17:40:59.020 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.018+0000 7f41d8a13640 1 -- 172.21.15.113:0/926900975 wait complete. 2023-11-28T17:40:59.020 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.020+0000 7f41d8a13640 1 Processor -- start 2023-11-28T17:40:59.020 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.020+0000 7f41d8a13640 1 -- start start 2023-11-28T17:40:59.023 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41d8a13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d4070260 0x7f41d4155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.023 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41d8a13640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d4070640 0x7f41d40c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.023 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41d8a13640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d40c23f0 0x7f41d41ef4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.023 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41d8a13640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d4158500 con 0x7f41d4070260 2023-11-28T17:40:59.024 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41d8a13640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d4156150 con 0x7f41d4070640 2023-11-28T17:40:59.024 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41d8a13640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d4157880 con 0x7f41d40c23f0 2023-11-28T17:40:59.024 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41cbfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d4070640 0x7f41d40c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.024 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41d3fff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d40c23f0 0x7f41d41ef4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.024 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41cbfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d4070640 0x7f41d40c8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43406/0 (socket says 172.21.15.113:43406) 2023-11-28T17:40:59.024 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41cbfff640 1 -- 172.21.15.113:0/2891704407 learned_addr learned my addr 172.21.15.113:0/2891704407 (peer_addr_for_me v2:172.21.15.113:0/0) 2023-11-28T17:40:59.024 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41d37fe640 1 --2- 172.21.15.113:0/2891704407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d4070260 0x7f41d4155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.024 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41d3fff640 1 -- 172.21.15.113:0/2891704407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d4070640 msgr2=0x7f41d40c8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:59.024 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41d3fff640 1 --2- 172.21.15.113:0/2891704407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d4070640 0x7f41d40c8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.024 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41d3fff640 1 -- 172.21.15.113:0/2891704407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d4070260 msgr2=0x7f41d4155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:59.024 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41d3fff640 1 --2- 172.21.15.113:0/2891704407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d4070260 0x7f41d4155a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.024 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41d3fff640 1 -- 172.21.15.113:0/2891704407 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c40094d0 con 0x7f41d40c23f0 2023-11-28T17:40:59.024 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.021+0000 7f41cbfff640 1 --2- 172.21.15.113:0/2891704407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d4070640 0x7f41d40c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:40:59.025 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.022+0000 7f41d3fff640 1 --2- 172.21.15.113:0/2891704407 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d40c23f0 0x7f41d41ef4b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00ef90 tx=0x7f41cc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:40:59.025 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.022+0000 7f41d0ff9640 1 -- 172.21.15.113:0/2891704407 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc009c00 con 0x7f41d40c23f0 2023-11-28T17:40:59.025 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.022+0000 7f41d0ff9640 1 -- 172.21.15.113:0/2891704407 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc00f040 con 0x7f41d40c23f0 2023-11-28T17:40:59.025 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.022+0000 7f41d0ff9640 1 -- 172.21.15.113:0/2891704407 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0146c0 con 0x7f41d40c23f0 2023-11-28T17:40:59.025 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.022+0000 7f41d8a13640 1 -- 172.21.15.113:0/2891704407 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d4157b60 con 0x7f41d40c23f0 2023-11-28T17:40:59.025 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.022+0000 7f41d8a13640 1 -- 172.21.15.113:0/2891704407 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d4158120 con 0x7f41d40c23f0 2023-11-28T17:40:59.025 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.022+0000 7f41d0ff9640 1 -- 172.21.15.113:0/2891704407 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f41cc014b20 con 0x7f41d40c23f0 2023-11-28T17:40:59.025 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.022+0000 7f41d0ff9640 1 -- 172.21.15.113:0/2891704407 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f41cc013790 con 0x7f41d40c23f0 2023-11-28T17:40:59.025 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.023+0000 7f41d8a13640 1 -- 172.21.15.113:0/2891704407 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d40cab80 con 0x7f41d40c23f0 2023-11-28T17:40:59.027 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.026+0000 7f41d0ff9640 1 -- 172.21.15.113:0/2891704407 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f41cc03bce0 con 0x7f41d40c23f0 2023-11-28T17:40:59.048 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:40:59.047+0000 7feeef248040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T17:40:59.105 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.104+0000 7f41d8a13640 1 -- 172.21.15.113:0/2891704407 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f5a771a0-ab90-45ff-80f9-f62f0ef1a954", "id": 1} v 0) v1 -- 0x7f41d4074db0 con 0x7f41d40c23f0 2023-11-28T17:40:59.112 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.111+0000 7f41d0ff9640 1 -- 172.21.15.113:0/2891704407 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f5a771a0-ab90-45ff-80f9-f62f0ef1a954", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f41cc02a070 con 0x7f41d40c23f0 2023-11-28T17:40:59.112 INFO:teuthology.orchestra.run.smithi113.stdout:1 2023-11-28T17:40:59.114 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.112+0000 7f41d8a13640 1 -- 172.21.15.113:0/2891704407 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d40c23f0 msgr2=0x7f41d41ef4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:59.115 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.112+0000 7f41d8a13640 1 --2- 172.21.15.113:0/2891704407 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d40c23f0 0x7f41d41ef4b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00ef90 tx=0x7f41cc00c490 comp rx=0 tx=0).stop 2023-11-28T17:40:59.115 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.113+0000 7f41d8a13640 1 -- 172.21.15.113:0/2891704407 shutdown_connections 2023-11-28T17:40:59.115 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.113+0000 7f41d8a13640 1 --2- 172.21.15.113:0/2891704407 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d40c23f0 0x7f41d41ef4b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.115 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.113+0000 7f41d8a13640 1 --2- 172.21.15.113:0/2891704407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d4070640 0x7f41d40c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.116 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.113+0000 7f41d8a13640 1 --2- 172.21.15.113:0/2891704407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d4070260 0x7f41d4155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.116 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.113+0000 7f41d8a13640 1 -- 172.21.15.113:0/2891704407 >> 172.21.15.113:0/2891704407 conn(0x7f41d406a4d0 msgr2=0x7f41d4153f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:40:59.116 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.113+0000 7f41d8a13640 1 -- 172.21.15.113:0/2891704407 shutdown_connections 2023-11-28T17:40:59.116 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.113+0000 7f41d8a13640 1 -- 172.21.15.113:0/2891704407 wait complete. 2023-11-28T17:40:59.128 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:40:59.128+0000 7feeef248040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T17:40:59.132 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new ea38def8-88ca-44ee-8e2e-b90920eec720 2 2023-11-28T17:40:59.182 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:40:59.181+0000 7feeef248040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T17:40:59.247 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:40:59.240+0000 7feeef248040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T17:40:59.298 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:40:59.295+0000 7feeef248040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T17:40:59.365 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.364+0000 7ff626fdb640 1 Processor -- start 2023-11-28T17:40:59.365 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.365+0000 7ff626fdb640 1 -- start start 2023-11-28T17:40:59.367 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.366+0000 7ff626fdb640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff620070520 0x7ff6200c2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.367 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.366+0000 7ff626fdb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff6200caae0 0x7ff620154c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.367 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.366+0000 7ff626fdb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62015b190 con 0x7ff620070900 2023-11-28T17:40:59.367 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.366+0000 7ff626fdb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6200c37c0 con 0x7ff6200caae0 2023-11-28T17:40:59.367 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.366+0000 7ff626fdb640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff62015b3e0 con 0x7ff620070520 2023-11-28T17:40:59.368 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.366+0000 7ff625fd9640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff620070520 0x7ff6200c2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.368 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.366+0000 7ff625fd9640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff620070520 0x7ff6200c2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43420/0 (socket says 172.21.15.113:43420) 2023-11-28T17:40:59.368 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.366+0000 7ff625fd9640 1 -- 172.21.15.113:0/1016765773 learned_addr learned my addr 172.21.15.113:0/1016765773 (peer_addr_for_me v2:172.21.15.113:0/0) 2023-11-28T17:40:59.368 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.366+0000 7ff6267da640 1 --2- 172.21.15.113:0/1016765773 >> v2:172.21.15.98:3300/0 conn(0x7ff6200caae0 0x7ff620154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.368 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.367+0000 7ff625fd9640 1 -- 172.21.15.113:0/1016765773 >> v2:172.21.15.98:3300/0 conn(0x7ff6200caae0 msgr2=0x7ff620154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:40:59.368 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.367+0000 7ff625fd9640 1 --2- 172.21.15.113:0/1016765773 >> v2:172.21.15.98:3300/0 conn(0x7ff6200caae0 0x7ff620154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.368 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.367+0000 7ff625fd9640 1 -- 172.21.15.113:0/1016765773 >> v1:172.21.15.98:6789/0 conn(0x7ff620070900 legacy=0x7ff620157e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:40:59.368 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.367+0000 7ff625fd9640 1 -- 172.21.15.113:0/1016765773 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62015b6c0 con 0x7ff620070520 2023-11-28T17:40:59.369 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.368+0000 7ff625fd9640 1 --2- 172.21.15.113:0/1016765773 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff620070520 0x7ff6200c2310 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff620072010 tx=0x7ff61c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0c935d410b7cdcf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:40:59.369 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.368+0000 7ff624fd7640 1 -- 172.21.15.113:0/1016765773 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c031870 con 0x7ff620070520 2023-11-28T17:40:59.369 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.368+0000 7ff624fd7640 1 -- 172.21.15.113:0/1016765773 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61c0319d0 con 0x7ff620070520 2023-11-28T17:40:59.369 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.368+0000 7ff624fd7640 1 -- 172.21.15.113:0/1016765773 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c031d60 con 0x7ff620070520 2023-11-28T17:40:59.370 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.369+0000 7ff626fdb640 1 -- 172.21.15.113:0/1016765773 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff620070520 msgr2=0x7ff6200c2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:59.370 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.369+0000 7ff626fdb640 1 --2- 172.21.15.113:0/1016765773 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff620070520 0x7ff6200c2310 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff620072010 tx=0x7ff61c02b910 comp rx=0 tx=0).stop 2023-11-28T17:40:59.370 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.369+0000 7ff626fdb640 1 -- 172.21.15.113:0/1016765773 shutdown_connections 2023-11-28T17:40:59.370 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.369+0000 7ff626fdb640 1 --2- 172.21.15.113:0/1016765773 >> v2:172.21.15.98:3300/0 conn(0x7ff6200caae0 0x7ff620154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.370 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.369+0000 7ff626fdb640 1 --2- 172.21.15.113:0/1016765773 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff620070520 0x7ff6200c2310 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.370 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.369+0000 7ff626fdb640 1 -- 172.21.15.113:0/1016765773 >> 172.21.15.113:0/1016765773 conn(0x7ff62006a4d0 msgr2=0x7ff62006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:40:59.370 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.369+0000 7ff626fdb640 1 -- 172.21.15.113:0/1016765773 shutdown_connections 2023-11-28T17:40:59.370 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.369+0000 7ff626fdb640 1 -- 172.21.15.113:0/1016765773 wait complete. 2023-11-28T17:40:59.371 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.371+0000 7ff626fdb640 1 Processor -- start 2023-11-28T17:40:59.371 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.371+0000 7ff626fdb640 1 -- start start 2023-11-28T17:40:59.374 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff626fdb640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff620070520 0x7ff6201ef590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.374 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff626fdb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff620070900 0x7ff62014d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.374 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff626fdb640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6200caae0 0x7ff62014abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.374 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff626fdb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6200ae800 con 0x7ff620070900 2023-11-28T17:40:59.374 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff626fdb640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6200ae2d0 con 0x7ff6200caae0 2023-11-28T17:40:59.374 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff626fdb640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6201f32f0 con 0x7ff620070520 2023-11-28T17:40:59.375 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff625fd9640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff620070520 0x7ff6201ef590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.375 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff625fd9640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff620070520 0x7ff6201ef590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.113:40210/0 (socket says 172.21.15.113:40210) 2023-11-28T17:40:59.375 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff625fd9640 1 -- 172.21.15.113:0/2818835510 learned_addr learned my addr 172.21.15.113:0/2818835510 (peer_addr_for_me v2:172.21.15.113:0/0) 2023-11-28T17:40:59.375 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff6267da640 1 --2- 172.21.15.113:0/2818835510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6200caae0 0x7ff62014abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.375 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff6257d8640 1 --2- 172.21.15.113:0/2818835510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff620070900 0x7ff62014d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.375 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff625fd9640 1 -- 172.21.15.113:0/2818835510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6200caae0 msgr2=0x7ff62014abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:59.375 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff625fd9640 1 --2- 172.21.15.113:0/2818835510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6200caae0 0x7ff62014abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.375 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff625fd9640 1 -- 172.21.15.113:0/2818835510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff620070900 msgr2=0x7ff62014d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:59.375 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff625fd9640 1 --2- 172.21.15.113:0/2818835510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff620070900 0x7ff62014d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.375 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.372+0000 7ff625fd9640 1 -- 172.21.15.113:0/2818835510 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c0094d0 con 0x7ff620070520 2023-11-28T17:40:59.375 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.373+0000 7ff625fd9640 1 --2- 172.21.15.113:0/2818835510 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff620070520 0x7ff6201ef590 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff61c000c00 tx=0x7ff61c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:40:59.375 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.373+0000 7ff6167fc640 1 -- 172.21.15.113:0/2818835510 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c03abe0 con 0x7ff620070520 2023-11-28T17:40:59.376 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.373+0000 7ff6167fc640 1 -- 172.21.15.113:0/2818835510 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61c03ad40 con 0x7ff620070520 2023-11-28T17:40:59.376 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.373+0000 7ff6167fc640 1 -- 172.21.15.113:0/2818835510 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c044610 con 0x7ff620070520 2023-11-28T17:40:59.376 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.373+0000 7ff626fdb640 1 -- 172.21.15.113:0/2818835510 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6200ae520 con 0x7ff620070520 2023-11-28T17:40:59.376 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.373+0000 7ff626fdb640 1 -- 172.21.15.113:0/2818835510 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6201f2f80 con 0x7ff620070520 2023-11-28T17:40:59.376 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.373+0000 7ff6167fc640 1 -- 172.21.15.113:0/2818835510 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff61c044a70 con 0x7ff620070520 2023-11-28T17:40:59.376 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.374+0000 7ff6167fc640 1 -- 172.21.15.113:0/2818835510 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7ff61c02d040 con 0x7ff620070520 2023-11-28T17:40:59.376 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.374+0000 7ff626fdb640 1 -- 172.21.15.113:0/2818835510 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6201f2b00 con 0x7ff620070520 2023-11-28T17:40:59.377 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.377+0000 7ff6167fc640 1 -- 172.21.15.113:0/2818835510 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff6201f2b00 con 0x7ff620070520 2023-11-28T17:40:59.457 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.456+0000 7ff626fdb640 1 -- 172.21.15.113:0/2818835510 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ea38def8-88ca-44ee-8e2e-b90920eec720", "id": 2} v 0) v1 -- 0x7ff6201f3110 con 0x7ff620070520 2023-11-28T17:40:59.465 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.464+0000 7ff6167fc640 1 -- 172.21.15.113:0/2818835510 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ea38def8-88ca-44ee-8e2e-b90920eec720", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff61c035020 con 0x7ff620070520 2023-11-28T17:40:59.466 INFO:teuthology.orchestra.run.smithi113.stdout:2 2023-11-28T17:40:59.467 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.466+0000 7ff626fdb640 1 -- 172.21.15.113:0/2818835510 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff620070520 msgr2=0x7ff6201ef590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:59.467 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.466+0000 7ff626fdb640 1 --2- 172.21.15.113:0/2818835510 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff620070520 0x7ff6201ef590 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff61c000c00 tx=0x7ff61c0046c0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.467 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.466+0000 7ff626fdb640 1 -- 172.21.15.113:0/2818835510 shutdown_connections 2023-11-28T17:40:59.468 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.466+0000 7ff626fdb640 1 --2- 172.21.15.113:0/2818835510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6200caae0 0x7ff62014abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.468 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.466+0000 7ff626fdb640 1 --2- 172.21.15.113:0/2818835510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff620070900 0x7ff62014d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.468 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.466+0000 7ff626fdb640 1 --2- 172.21.15.113:0/2818835510 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff620070520 0x7ff6201ef590 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.468 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.466+0000 7ff626fdb640 1 -- 172.21.15.113:0/2818835510 >> 172.21.15.113:0/2818835510 conn(0x7ff62006a4d0 msgr2=0x7ff620157e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:40:59.468 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.466+0000 7ff626fdb640 1 -- 172.21.15.113:0/2818835510 shutdown_connections 2023-11-28T17:40:59.468 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.466+0000 7ff626fdb640 1 -- 172.21.15.113:0/2818835510 wait complete. 2023-11-28T17:40:59.484 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new a8373586-43f6-4c83-b0c5-d26889c25963 3 2023-11-28T17:40:59.525 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:40:59.524+0000 7f44faa12040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T17:40:59.590 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:40:59.589+0000 7f44faa12040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T17:40:59.643 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:40:59.643+0000 7f44faa12040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T17:40:59.672 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.671+0000 7ff6e54ee640 1 Processor -- start 2023-11-28T17:40:59.672 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.671+0000 7ff6e54ee640 1 -- start start 2023-11-28T17:40:59.674 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.673+0000 7ff6e54ee640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff6e00ca280 0x7ff6e00ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.674 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.673+0000 7ff6e54ee640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e006c060 con 0x7ff6e00cb070 2023-11-28T17:40:59.674 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.673+0000 7ff6e54ee640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e006a660 con 0x7ff6e01540d0 2023-11-28T17:40:59.674 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.673+0000 7ff6e54ee640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e00cbdd0 con 0x7ff6e00ca280 2023-11-28T17:40:59.674 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.673+0000 7ff6df7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff6e00ca280 0x7ff6e00ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.674 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.673+0000 7ff6deffd640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7ff6e01540d0 0x7ff6e015dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:40130/0 (socket says 172.21.15.113:40130) 2023-11-28T17:40:59.674 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.673+0000 7ff6deffd640 1 -- 172.21.15.113:0/3537428273 learned_addr learned my addr 172.21.15.113:0/3537428273 (peer_addr_for_me v1:172.21.15.113:0/0) 2023-11-28T17:40:59.675 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.674+0000 7ff6de7fc640 1 -- 172.21.15.113:0/3537428273 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637122121 0 0) 0x7ff6e006a660 con 0x7ff6e01540d0 2023-11-28T17:40:59.675 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.674+0000 7ff6df7fe640 1 -- 172.21.15.113:0/3537428273 >> v1:172.21.15.113:6789/0 conn(0x7ff6e01540d0 legacy=0x7ff6e015dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:59.675 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.674+0000 7ff6df7fe640 1 -- 172.21.15.113:0/3537428273 >> v1:172.21.15.98:6789/0 conn(0x7ff6e00cb070 legacy=0x7ff6e006b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:40:59.675 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.674+0000 7ff6df7fe640 1 -- 172.21.15.113:0/3537428273 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e006a8b0 con 0x7ff6e00ca280 2023-11-28T17:40:59.676 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.675+0000 7ff6df7fe640 1 --2- 172.21.15.113:0/3537428273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e00ca280 0x7ff6e00ca6d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8009d90 tx=0x7ff6d802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7beb7ca352db791e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:40:59.676 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.675+0000 7ff6de7fc640 1 -- 172.21.15.113:0/3537428273 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d802cc60 con 0x7ff6e00ca280 2023-11-28T17:40:59.676 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.675+0000 7ff6de7fc640 1 -- 172.21.15.113:0/3537428273 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d802cdc0 con 0x7ff6e00ca280 2023-11-28T17:40:59.676 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.675+0000 7ff6de7fc640 1 -- 172.21.15.113:0/3537428273 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8034660 con 0x7ff6e00ca280 2023-11-28T17:40:59.677 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.676+0000 7ff6e54ee640 1 -- 172.21.15.113:0/3537428273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e00ca280 msgr2=0x7ff6e00ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:59.678 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.676+0000 7ff6e54ee640 1 --2- 172.21.15.113:0/3537428273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e00ca280 0x7ff6e00ca6d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8009d90 tx=0x7ff6d802beb0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.678 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.676+0000 7ff6e54ee640 1 -- 172.21.15.113:0/3537428273 shutdown_connections 2023-11-28T17:40:59.678 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.676+0000 7ff6e54ee640 1 --2- 172.21.15.113:0/3537428273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e00ca280 0x7ff6e00ca6d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.678 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.676+0000 7ff6e54ee640 1 -- 172.21.15.113:0/3537428273 >> 172.21.15.113:0/3537428273 conn(0x7ff6e00cf490 msgr2=0x7ff6e00cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:40:59.678 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.676+0000 7ff6e54ee640 1 -- 172.21.15.113:0/3537428273 shutdown_connections 2023-11-28T17:40:59.678 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.676+0000 7ff6e54ee640 1 -- 172.21.15.113:0/3537428273 wait complete. 2023-11-28T17:40:59.679 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.679+0000 7ff6e54ee640 1 Processor -- start 2023-11-28T17:40:59.679 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.679+0000 7ff6e54ee640 1 -- start start 2023-11-28T17:40:59.682 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.680+0000 7ff6e54ee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6e00ca280 0x7ff6e01f0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.682 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.680+0000 7ff6e54ee640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6e00cb070 0x7ff6e0157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.682 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.681+0000 7ff6e54ee640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e01540d0 0x7ff6e01f3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.682 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.681+0000 7ff6e54ee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e01f3d60 con 0x7ff6e00ca280 2023-11-28T17:40:59.682 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.681+0000 7ff6e54ee640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e01f0000 con 0x7ff6e01540d0 2023-11-28T17:40:59.682 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.681+0000 7ff6e54ee640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e0156a50 con 0x7ff6e00cb070 2023-11-28T17:40:59.683 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.681+0000 7ff6df7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e01540d0 0x7ff6e01f3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.683 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.681+0000 7ff6d7fff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6e00cb070 0x7ff6e0157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.683 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.681+0000 7ff6df7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e01540d0 0x7ff6e01f3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43440/0 (socket says 172.21.15.113:43440) 2023-11-28T17:40:59.683 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.681+0000 7ff6df7fe640 1 -- 172.21.15.113:0/2491607473 learned_addr learned my addr 172.21.15.113:0/2491607473 (peer_addr_for_me v2:172.21.15.113:0/0) 2023-11-28T17:40:59.683 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.681+0000 7ff6deffd640 1 --2- 172.21.15.113:0/2491607473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6e00ca280 0x7ff6e01f0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.683 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.681+0000 7ff6df7fe640 1 -- 172.21.15.113:0/2491607473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6e00cb070 msgr2=0x7ff6e0157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:59.683 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.681+0000 7ff6df7fe640 1 --2- 172.21.15.113:0/2491607473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6e00cb070 0x7ff6e0157fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.683 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.681+0000 7ff6df7fe640 1 -- 172.21.15.113:0/2491607473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6e00ca280 msgr2=0x7ff6e01f0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:59.683 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.681+0000 7ff6df7fe640 1 --2- 172.21.15.113:0/2491607473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6e00ca280 0x7ff6e01f0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.683 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.681+0000 7ff6df7fe640 1 -- 172.21.15.113:0/2491607473 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d8009a70 con 0x7ff6e01540d0 2023-11-28T17:40:59.683 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.682+0000 7ff6df7fe640 1 --2- 172.21.15.113:0/2491607473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e01540d0 0x7ff6e01f3820 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8009d90 tx=0x7ff6d8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:40:59.684 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.682+0000 7ff6d77fe640 1 -- 172.21.15.113:0/2491607473 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d803e610 con 0x7ff6e01540d0 2023-11-28T17:40:59.684 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.682+0000 7ff6d77fe640 1 -- 172.21.15.113:0/2491607473 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d803e770 con 0x7ff6e01540d0 2023-11-28T17:40:59.684 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.682+0000 7ff6d77fe640 1 -- 172.21.15.113:0/2491607473 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d803eb20 con 0x7ff6e01540d0 2023-11-28T17:40:59.684 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.682+0000 7ff6e54ee640 1 -- 172.21.15.113:0/2491607473 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e0156d30 con 0x7ff6e01540d0 2023-11-28T17:40:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.682+0000 7ff6e54ee640 1 -- 172.21.15.113:0/2491607473 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e01ef9b0 con 0x7ff6e01540d0 2023-11-28T17:40:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.682+0000 7ff6d77fe640 1 -- 172.21.15.113:0/2491607473 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff6d803b050 con 0x7ff6e01540d0 2023-11-28T17:40:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.682+0000 7ff6d77fe640 1 -- 172.21.15.113:0/2491607473 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7ff6d8003e20 con 0x7ff6e01540d0 2023-11-28T17:40:59.685 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.683+0000 7ff6e54ee640 1 -- 172.21.15.113:0/2491607473 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e0074f60 con 0x7ff6e01540d0 2023-11-28T17:40:59.686 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.686+0000 7ff6d77fe640 1 -- 172.21.15.113:0/2491607473 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff6d8044070 con 0x7ff6e01540d0 2023-11-28T17:40:59.741 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:40:59.740+0000 7f44faa12040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T17:40:59.750 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:40:59.749+0000 7feeef248040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T17:40:59.763 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.763+0000 7ff6e54ee640 1 -- 172.21.15.113:0/2491607473 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a8373586-43f6-4c83-b0c5-d26889c25963", "id": 3} v 0) v1 -- 0x7ff6e01f0140 con 0x7ff6e01540d0 2023-11-28T17:40:59.771 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.770+0000 7ff6d77fe640 1 -- 172.21.15.113:0/2491607473 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a8373586-43f6-4c83-b0c5-d26889c25963", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff6d8039070 con 0x7ff6e01540d0 2023-11-28T17:40:59.771 INFO:teuthology.orchestra.run.smithi113.stdout:3 2023-11-28T17:40:59.772 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.771+0000 7ff6e54ee640 1 -- 172.21.15.113:0/2491607473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e01540d0 msgr2=0x7ff6e01f3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:59.772 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.771+0000 7ff6e54ee640 1 --2- 172.21.15.113:0/2491607473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e01540d0 0x7ff6e01f3820 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8009d90 tx=0x7ff6d8004a40 comp rx=0 tx=0).stop 2023-11-28T17:40:59.773 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.772+0000 7ff6e54ee640 1 -- 172.21.15.113:0/2491607473 shutdown_connections 2023-11-28T17:40:59.773 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.772+0000 7ff6e54ee640 1 --2- 172.21.15.113:0/2491607473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6e01540d0 0x7ff6e01f3820 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.773 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.772+0000 7ff6e54ee640 1 --2- 172.21.15.113:0/2491607473 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6e00cb070 0x7ff6e0157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.773 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.772+0000 7ff6e54ee640 1 --2- 172.21.15.113:0/2491607473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6e00ca280 0x7ff6e01f0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.773 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.772+0000 7ff6e54ee640 1 -- 172.21.15.113:0/2491607473 >> 172.21.15.113:0/2491607473 conn(0x7ff6e00cf490 msgr2=0x7ff6e00cde80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:40:59.773 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.772+0000 7ff6e54ee640 1 -- 172.21.15.113:0/2491607473 shutdown_connections 2023-11-28T17:40:59.773 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.772+0000 7ff6e54ee640 1 -- 172.21.15.113:0/2491607473 wait complete. 2023-11-28T17:40:59.789 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new fcf4f283-d0e0-4860-94f2-20301fd75204 4 2023-11-28T17:40:59.794 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:40:59.794+0000 7f44faa12040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T17:40:59.908 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:40:59.907+0000 7feeef248040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T17:40:59.913 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:40:59.913+0000 7f44faa12040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T17:40:59.990 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.989+0000 7fba6f59e640 1 Processor -- start 2023-11-28T17:40:59.990 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.989+0000 7fba6f59e640 1 -- start start 2023-11-28T17:40:59.992 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.991+0000 7fba6f59e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba70150dc0 0x7fba700c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.992 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.991+0000 7fba6f59e640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fba700c7670 0x7fba7006b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:40:59.992 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.991+0000 7fba6f59e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba700c8490 con 0x7fba701509e0 2023-11-28T17:40:59.992 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.991+0000 7fba6f59e640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba700c7bd0 con 0x7fba700c7670 2023-11-28T17:40:59.992 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.991+0000 7fba6f59e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba700c7030 con 0x7fba70150dc0 2023-11-28T17:40:59.992 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.991+0000 7fba6ed9d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fba700c7670 0x7fba7006b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.992 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.991+0000 7fba6dd9b640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba70150dc0 0x7fba700c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:40:59.992 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.991+0000 7fba6ed9d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fba700c7670 0x7fba7006b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43450/0 (socket says 172.21.15.113:43450) 2023-11-28T17:40:59.993 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.991+0000 7fba6ed9d640 1 -- 172.21.15.113:0/2580177261 learned_addr learned my addr 172.21.15.113:0/2580177261 (peer_addr_for_me v2:172.21.15.113:0/0) 2023-11-28T17:40:59.993 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.992+0000 7fba6dd9b640 1 -- 172.21.15.113:0/2580177261 >> v2:172.21.15.113:3300/0 conn(0x7fba700c7670 msgr2=0x7fba7006b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:40:59.993 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.992+0000 7fba6dd9b640 1 --2- 172.21.15.113:0/2580177261 >> v2:172.21.15.113:3300/0 conn(0x7fba700c7670 0x7fba7006b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.993 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.992+0000 7fba6dd9b640 1 -- 172.21.15.113:0/2580177261 >> v1:172.21.15.98:6789/0 conn(0x7fba701509e0 legacy=0x7fba700c3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:40:59.993 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.992+0000 7fba6dd9b640 1 -- 172.21.15.113:0/2580177261 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba700c7e40 con 0x7fba70150dc0 2023-11-28T17:40:59.994 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.992+0000 7fba6ed9d640 1 --2- 172.21.15.113:0/2580177261 >> v2:172.21.15.113:3300/0 conn(0x7fba700c7670 0x7fba7006b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:40:59.994 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.993+0000 7fba6dd9b640 1 --2- 172.21.15.113:0/2580177261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba70150dc0 0x7fba700c6a10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fba640097a0 tx=0x7fba6402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e45233d8904e826b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:40:59.995 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.993+0000 7fba6d59a640 1 -- 172.21.15.113:0/2580177261 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba64030800 con 0x7fba70150dc0 2023-11-28T17:40:59.995 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.993+0000 7fba6d59a640 1 -- 172.21.15.113:0/2580177261 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba64030960 con 0x7fba70150dc0 2023-11-28T17:40:59.995 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.993+0000 7fba6d59a640 1 -- 172.21.15.113:0/2580177261 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba64030cf0 con 0x7fba70150dc0 2023-11-28T17:40:59.995 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.994+0000 7fba6f59e640 1 -- 172.21.15.113:0/2580177261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba70150dc0 msgr2=0x7fba700c6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:40:59.995 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.994+0000 7fba6f59e640 1 --2- 172.21.15.113:0/2580177261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba70150dc0 0x7fba700c6a10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fba640097a0 tx=0x7fba6402b940 comp rx=0 tx=0).stop 2023-11-28T17:40:59.995 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.994+0000 7fba6f59e640 1 -- 172.21.15.113:0/2580177261 shutdown_connections 2023-11-28T17:40:59.995 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.994+0000 7fba6f59e640 1 --2- 172.21.15.113:0/2580177261 >> v2:172.21.15.113:3300/0 conn(0x7fba700c7670 0x7fba7006b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.995 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.994+0000 7fba6f59e640 1 --2- 172.21.15.113:0/2580177261 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba70150dc0 0x7fba700c6a10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:40:59.995 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.994+0000 7fba6f59e640 1 -- 172.21.15.113:0/2580177261 >> 172.21.15.113:0/2580177261 conn(0x7fba70071290 msgr2=0x7fba70071670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:40:59.996 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.994+0000 7fba6f59e640 1 -- 172.21.15.113:0/2580177261 shutdown_connections 2023-11-28T17:40:59.996 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.994+0000 7fba6f59e640 1 -- 172.21.15.113:0/2580177261 wait complete. 2023-11-28T17:40:59.996 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.996+0000 7fba6f59e640 1 Processor -- start 2023-11-28T17:40:59.996 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.996+0000 7fba6f59e640 1 -- start start 2023-11-28T17:40:59.997 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:40:59.996+0000 7f44faa12040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T17:41:00.000 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6f59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba700c7670 0x7fba70156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.000 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6f59e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba701509e0 0x7fba7014bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.000 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6f59e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba70150dc0 0x7fba701f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.000 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6f59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba700cbfe0 con 0x7fba700c7670 2023-11-28T17:41:00.000 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6f59e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba70154830 con 0x7fba701509e0 2023-11-28T17:41:00.000 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6f59e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba701553b0 con 0x7fba70150dc0 2023-11-28T17:41:00.000 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6ed9d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba70150dc0 0x7fba701f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.000 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6dd9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba701509e0 0x7fba7014bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.000 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6ed9d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba70150dc0 0x7fba701f2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.113:40256/0 (socket says 172.21.15.113:40256) 2023-11-28T17:41:00.000 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6ed9d640 1 -- 172.21.15.113:0/2232800665 learned_addr learned my addr 172.21.15.113:0/2232800665 (peer_addr_for_me v2:172.21.15.113:0/0) 2023-11-28T17:41:00.000 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6e59c640 1 --2- 172.21.15.113:0/2232800665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba700c7670 0x7fba70156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.001 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6ed9d640 1 -- 172.21.15.113:0/2232800665 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba701509e0 msgr2=0x7fba7014bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.001 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6ed9d640 1 --2- 172.21.15.113:0/2232800665 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba701509e0 0x7fba7014bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.001 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6ed9d640 1 -- 172.21.15.113:0/2232800665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba700c7670 msgr2=0x7fba70156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.001 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6ed9d640 1 --2- 172.21.15.113:0/2232800665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba700c7670 0x7fba70156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.001 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6ed9d640 1 -- 172.21.15.113:0/2232800665 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba68009b30 con 0x7fba70150dc0 2023-11-28T17:41:00.001 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.997+0000 7fba6e59c640 1 --2- 172.21.15.113:0/2232800665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba700c7670 0x7fba70156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:00.001 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.998+0000 7fba6ed9d640 1 --2- 172.21.15.113:0/2232800665 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba70150dc0 0x7fba701f2c30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fba68009e50 tx=0x7fba6800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:00.001 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.998+0000 7fba5effd640 1 -- 172.21.15.113:0/2232800665 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68018070 con 0x7fba70150dc0 2023-11-28T17:41:00.001 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.998+0000 7fba5effd640 1 -- 172.21.15.113:0/2232800665 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6800f040 con 0x7fba70150dc0 2023-11-28T17:41:00.001 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.998+0000 7fba5effd640 1 -- 172.21.15.113:0/2232800665 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6800cda0 con 0x7fba70150dc0 2023-11-28T17:41:00.001 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.998+0000 7fba6f59e640 1 -- 172.21.15.113:0/2232800665 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba64009400 con 0x7fba70150dc0 2023-11-28T17:41:00.001 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.998+0000 7fba6f59e640 1 -- 172.21.15.113:0/2232800665 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba701566f0 con 0x7fba70150dc0 2023-11-28T17:41:00.002 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.998+0000 7fba5effd640 1 -- 172.21.15.113:0/2232800665 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fba68003c90 con 0x7fba70150dc0 2023-11-28T17:41:00.002 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.998+0000 7fba5effd640 1 -- 172.21.15.113:0/2232800665 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fba680047f0 con 0x7fba70150dc0 2023-11-28T17:41:00.002 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:40:59.999+0000 7fba6f59e640 1 -- 172.21.15.113:0/2232800665 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba70157860 con 0x7fba70150dc0 2023-11-28T17:41:00.006 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.004+0000 7fba5effd640 1 -- 172.21.15.113:0/2232800665 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fba680092e0 con 0x7fba70150dc0 2023-11-28T17:41:00.054 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:00.053+0000 7f44faa12040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T17:41:00.056 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:41:00.055+0000 7feeef248040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T17:41:00.081 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.081+0000 7fba6f59e640 1 -- 172.21.15.113:0/2232800665 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd new", "uuid": "fcf4f283-d0e0-4860-94f2-20301fd75204", "id": 4} v 0) v1 -- 0x7fba70074db0 con 0x7fba70150dc0 2023-11-28T17:41:00.088 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.087+0000 7fba5effd640 1 -- 172.21.15.113:0/2232800665 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fcf4f283-d0e0-4860-94f2-20301fd75204", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fba6802b0b0 con 0x7fba70150dc0 2023-11-28T17:41:00.088 INFO:teuthology.orchestra.run.smithi113.stdout:4 2023-11-28T17:41:00.089 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.088+0000 7fba6f59e640 1 -- 172.21.15.113:0/2232800665 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba70150dc0 msgr2=0x7fba701f2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.089 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.088+0000 7fba6f59e640 1 --2- 172.21.15.113:0/2232800665 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba70150dc0 0x7fba701f2c30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fba68009e50 tx=0x7fba6800c490 comp rx=0 tx=0).stop 2023-11-28T17:41:00.090 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.088+0000 7fba6f59e640 1 -- 172.21.15.113:0/2232800665 shutdown_connections 2023-11-28T17:41:00.090 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.089+0000 7fba6f59e640 1 --2- 172.21.15.113:0/2232800665 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba70150dc0 0x7fba701f2c30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.090 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.089+0000 7fba6f59e640 1 --2- 172.21.15.113:0/2232800665 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba701509e0 0x7fba7014bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.090 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.089+0000 7fba6f59e640 1 --2- 172.21.15.113:0/2232800665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba700c7670 0x7fba70156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.090 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.089+0000 7fba6f59e640 1 -- 172.21.15.113:0/2232800665 >> 172.21.15.113:0/2232800665 conn(0x7fba70071290 msgr2=0x7fba70073880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:00.090 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.089+0000 7fba6f59e640 1 -- 172.21.15.113:0/2232800665 shutdown_connections 2023-11-28T17:41:00.090 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.089+0000 7fba6f59e640 1 -- 172.21.15.113:0/2232800665 wait complete. 2023-11-28T17:41:00.105 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new b8b1086a-77a2-459a-b541-154e955258c4 5 2023-11-28T17:41:00.110 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:41:00.109+0000 7feeef248040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T17:41:00.117 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:00.116+0000 7f44faa12040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T17:41:00.174 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:00.174+0000 7f44faa12040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T17:41:00.219 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:41:00.218+0000 7feeef248040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T17:41:00.291 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.290+0000 7fc8fd267640 1 Processor -- start 2023-11-28T17:41:00.291 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.290+0000 7fc8fd267640 1 -- start start 2023-11-28T17:41:00.294 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.292+0000 7fc8fd267640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc8f8153c80 0x7fc8f806b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.294 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.292+0000 7fc8fd267640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8f80c69b0 0x7fc8f806be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.294 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.292+0000 7fc8fd267640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f815d230 con 0x7fc8f80c7720 2023-11-28T17:41:00.294 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.292+0000 7fc8fd267640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f815e070 con 0x7fc8f8153c80 2023-11-28T17:41:00.294 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.292+0000 7fc8fd267640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f815d4f0 con 0x7fc8f80c69b0 2023-11-28T17:41:00.294 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.293+0000 7fc8effff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8f80c69b0 0x7fc8f806be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.294 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.293+0000 7fc8fca66640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fc8f80c7720 0x7fc8f80c7bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:40134/0 (socket says 172.21.15.113:40134) 2023-11-28T17:41:00.294 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.293+0000 7fc8effff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8f80c69b0 0x7fc8f806be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.113:40270/0 (socket says 172.21.15.113:40270) 2023-11-28T17:41:00.294 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.293+0000 7fc8fca66640 1 -- 172.21.15.113:0/1633567216 learned_addr learned my addr 172.21.15.113:0/1633567216 (peer_addr_for_me v1:172.21.15.113:0/0) 2023-11-28T17:41:00.294 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.293+0000 7fc8f7fff640 1 --2- 172.21.15.113:0/1633567216 >> v2:172.21.15.98:3300/0 conn(0x7fc8f8153c80 0x7fc8f806b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.295 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.293+0000 7fc8effff640 1 -- 172.21.15.113:0/1633567216 >> v2:172.21.15.98:3300/0 conn(0x7fc8f8153c80 msgr2=0x7fc8f806b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:00.295 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.293+0000 7fc8effff640 1 --2- 172.21.15.113:0/1633567216 >> v2:172.21.15.98:3300/0 conn(0x7fc8f8153c80 0x7fc8f806b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.295 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.293+0000 7fc8effff640 1 -- 172.21.15.113:0/1633567216 >> v1:172.21.15.113:6789/0 conn(0x7fc8f80c7720 legacy=0x7fc8f80c7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.295 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.293+0000 7fc8effff640 1 -- 172.21.15.113:0/1633567216 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f815d7d0 con 0x7fc8f80c69b0 2023-11-28T17:41:00.296 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.294+0000 7fc8effff640 1 --2- 172.21.15.113:0/1633567216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8f80c69b0 0x7fc8f806be80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc8e00097a0 tx=0x7fc8e002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f727b508c332edc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:00.296 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.295+0000 7fc8f77fe640 1 -- 172.21.15.113:0/1633567216 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0030800 con 0x7fc8f80c69b0 2023-11-28T17:41:00.296 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.295+0000 7fc8f77fe640 1 -- 172.21.15.113:0/1633567216 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e0030960 con 0x7fc8f80c69b0 2023-11-28T17:41:00.296 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.295+0000 7fc8f77fe640 1 -- 172.21.15.113:0/1633567216 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e0030cf0 con 0x7fc8f80c69b0 2023-11-28T17:41:00.296 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.295+0000 7fc8fd267640 1 -- 172.21.15.113:0/1633567216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8f80c69b0 msgr2=0x7fc8f806be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.297 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.295+0000 7fc8fd267640 1 --2- 172.21.15.113:0/1633567216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8f80c69b0 0x7fc8f806be80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc8e00097a0 tx=0x7fc8e002b940 comp rx=0 tx=0).stop 2023-11-28T17:41:00.297 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.295+0000 7fc8fd267640 1 -- 172.21.15.113:0/1633567216 shutdown_connections 2023-11-28T17:41:00.297 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.295+0000 7fc8fd267640 1 --2- 172.21.15.113:0/1633567216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8f80c69b0 0x7fc8f806be80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.297 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.295+0000 7fc8fd267640 1 --2- 172.21.15.113:0/1633567216 >> v2:172.21.15.98:3300/0 conn(0x7fc8f8153c80 0x7fc8f806b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.297 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.295+0000 7fc8fd267640 1 -- 172.21.15.113:0/1633567216 >> 172.21.15.113:0/1633567216 conn(0x7fc8f80cf300 msgr2=0x7fc8f80cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:00.297 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.295+0000 7fc8fd267640 1 -- 172.21.15.113:0/1633567216 shutdown_connections 2023-11-28T17:41:00.297 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.296+0000 7fc8fd267640 1 -- 172.21.15.113:0/1633567216 wait complete. 2023-11-28T17:41:00.297 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.297+0000 7fc8fd267640 1 Processor -- start 2023-11-28T17:41:00.298 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.297+0000 7fc8fd267640 1 -- start start 2023-11-28T17:41:00.300 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.298+0000 7fc8fd267640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f80c69b0 0x7fc8f8152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.300 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.298+0000 7fc8fd267640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f80c7720 0x7fc8f814b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.300 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.298+0000 7fc8fd267640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8f8153c80 0x7fc8f814bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.301 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.298+0000 7fc8fd267640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f80c2cb0 con 0x7fc8f80c7720 2023-11-28T17:41:00.301 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.298+0000 7fc8fd267640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f8156a30 con 0x7fc8f80c69b0 2023-11-28T17:41:00.301 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.298+0000 7fc8fd267640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f8154400 con 0x7fc8f8153c80 2023-11-28T17:41:00.301 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.298+0000 7fc8f7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f80c69b0 0x7fc8f8152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.301 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.298+0000 7fc8fca66640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8f8153c80 0x7fc8f814bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.301 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.298+0000 7fc8f7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f80c69b0 0x7fc8f8152d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43464/0 (socket says 172.21.15.113:43464) 2023-11-28T17:41:00.301 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.298+0000 7fc8f7fff640 1 -- 172.21.15.113:0/2183712167 learned_addr learned my addr 172.21.15.113:0/2183712167 (peer_addr_for_me v2:172.21.15.113:0/0) 2023-11-28T17:41:00.301 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.299+0000 7fc8effff640 1 --2- 172.21.15.113:0/2183712167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f80c7720 0x7fc8f814b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.301 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.299+0000 7fc8f7fff640 1 -- 172.21.15.113:0/2183712167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8f8153c80 msgr2=0x7fc8f814bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.301 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.299+0000 7fc8f7fff640 1 --2- 172.21.15.113:0/2183712167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8f8153c80 0x7fc8f814bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.301 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.299+0000 7fc8f7fff640 1 -- 172.21.15.113:0/2183712167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f80c7720 msgr2=0x7fc8f814b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.302 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.299+0000 7fc8f7fff640 1 --2- 172.21.15.113:0/2183712167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f80c7720 0x7fc8f814b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.302 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.299+0000 7fc8f7fff640 1 -- 172.21.15.113:0/2183712167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e0009400 con 0x7fc8f80c69b0 2023-11-28T17:41:00.302 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.299+0000 7fc8fca66640 1 --2- 172.21.15.113:0/2183712167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8f8153c80 0x7fc8f814bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:41:00.302 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.299+0000 7fc8f7fff640 1 --2- 172.21.15.113:0/2183712167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f80c69b0 0x7fc8f8152d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc8e800d620 tx=0x7fc8e800dae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:00.302 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.299+0000 7fc8f57fa640 1 -- 172.21.15.113:0/2183712167 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e800b620 con 0x7fc8f80c69b0 2023-11-28T17:41:00.302 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.299+0000 7fc8f57fa640 1 -- 172.21.15.113:0/2183712167 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e800b780 con 0x7fc8f80c69b0 2023-11-28T17:41:00.302 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.299+0000 7fc8f57fa640 1 -- 172.21.15.113:0/2183712167 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e800bb30 con 0x7fc8f80c69b0 2023-11-28T17:41:00.302 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.299+0000 7fc8fd267640 1 -- 172.21.15.113:0/2183712167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f81546c0 con 0x7fc8f80c69b0 2023-11-28T17:41:00.302 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.300+0000 7fc8fd267640 1 -- 172.21.15.113:0/2183712167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f81563e0 con 0x7fc8f80c69b0 2023-11-28T17:41:00.302 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.300+0000 7fc8f57fa640 1 -- 172.21.15.113:0/2183712167 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc8e801e460 con 0x7fc8f80c69b0 2023-11-28T17:41:00.302 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.300+0000 7fc8f57fa640 1 -- 172.21.15.113:0/2183712167 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fc8e801ec90 con 0x7fc8f80c69b0 2023-11-28T17:41:00.303 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.301+0000 7fc8fd267640 1 -- 172.21.15.113:0/2183712167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f8074db0 con 0x7fc8f80c69b0 2023-11-28T17:41:00.303 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.303+0000 7fc8f57fa640 1 -- 172.21.15.113:0/2183712167 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fc8e801e760 con 0x7fc8f80c69b0 2023-11-28T17:41:00.329 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:41:00.328+0000 7feeef248040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T17:41:00.381 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.381+0000 7fc8fd267640 1 -- 172.21.15.113:0/2183712167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b8b1086a-77a2-459a-b541-154e955258c4", "id": 5} v 0) v1 -- 0x7fc8f81525a0 con 0x7fc8f80c69b0 2023-11-28T17:41:00.388 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.387+0000 7fc8f57fa640 1 -- 172.21.15.113:0/2183712167 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b8b1086a-77a2-459a-b541-154e955258c4", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc8e801e9d0 con 0x7fc8f80c69b0 2023-11-28T17:41:00.388 INFO:teuthology.orchestra.run.smithi113.stdout:5 2023-11-28T17:41:00.389 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.389+0000 7fc8fd267640 1 -- 172.21.15.113:0/2183712167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f80c69b0 msgr2=0x7fc8f8152d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.389 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.389+0000 7fc8fd267640 1 --2- 172.21.15.113:0/2183712167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f80c69b0 0x7fc8f8152d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc8e800d620 tx=0x7fc8e800dae0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.390 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.389+0000 7fc8fd267640 1 -- 172.21.15.113:0/2183712167 shutdown_connections 2023-11-28T17:41:00.390 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.389+0000 7fc8fd267640 1 --2- 172.21.15.113:0/2183712167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc8f8153c80 0x7fc8f814bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.390 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.389+0000 7fc8fd267640 1 --2- 172.21.15.113:0/2183712167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f80c7720 0x7fc8f814b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.390 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.389+0000 7fc8fd267640 1 --2- 172.21.15.113:0/2183712167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8f80c69b0 0x7fc8f8152d30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.390 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.389+0000 7fc8fd267640 1 -- 172.21.15.113:0/2183712167 >> 172.21.15.113:0/2183712167 conn(0x7fc8f80cf300 msgr2=0x7fc8f80cb040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:00.390 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.389+0000 7fc8fd267640 1 -- 172.21.15.113:0/2183712167 shutdown_connections 2023-11-28T17:41:00.390 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.389+0000 7fc8fd267640 1 -- 172.21.15.113:0/2183712167 wait complete. 2023-11-28T17:41:00.405 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new a6a41999-a583-4c92-8675-5c5336e9319a 6 2023-11-28T17:41:00.491 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:41:00.489+0000 7feeef248040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T17:41:00.540 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:41:00.539+0000 7feeef248040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T17:41:00.637 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.637+0000 7f1c02483640 1 Processor -- start 2023-11-28T17:41:00.638 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.637+0000 7f1c02483640 1 -- start start 2023-11-28T17:41:00.639 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.638+0000 7f1c02483640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1bfc153fb0 0x7f1bfc0c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.639 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.638+0000 7f1c02483640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1bfc0c76f0 0x7f1bfc0c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.639 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.638+0000 7f1c02483640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bfc15db80 con 0x7f1bfc0b5270 2023-11-28T17:41:00.640 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.638+0000 7f1c02483640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bfc0cb730 con 0x7f1bfc153fb0 2023-11-28T17:41:00.640 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.638+0000 7f1c02483640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bfc15e0c0 con 0x7f1bfc0c76f0 2023-11-28T17:41:00.640 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.639+0000 7f1c00a20640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1bfc0c76f0 0x7f1bfc0c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.640 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.639+0000 7f1c00a20640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1bfc0c76f0 0x7f1bfc0c7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43472/0 (socket says 172.21.15.113:43472) 2023-11-28T17:41:00.640 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.639+0000 7f1bfbfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1bfc153fb0 0x7f1bfc0c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.640 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.639+0000 7f1c00a20640 1 -- 172.21.15.113:0/259281886 learned_addr learned my addr 172.21.15.113:0/259281886 (peer_addr_for_me v2:172.21.15.113:0/0) 2023-11-28T17:41:00.641 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.639+0000 7f1c00a20640 1 -- 172.21.15.113:0/259281886 >> v2:172.21.15.98:3300/0 conn(0x7f1bfc153fb0 msgr2=0x7f1bfc0c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:00.641 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.639+0000 7f1c00a20640 1 --2- 172.21.15.113:0/259281886 >> v2:172.21.15.98:3300/0 conn(0x7f1bfc153fb0 0x7f1bfc0c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.641 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.639+0000 7f1c00a20640 1 -- 172.21.15.113:0/259281886 >> v1:172.21.15.113:6789/0 conn(0x7f1bfc0b5270 legacy=0x7f1bfc0ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.641 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.639+0000 7f1c00a20640 1 -- 172.21.15.113:0/259281886 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc0d00a0 con 0x7f1bfc0c76f0 2023-11-28T17:41:00.641 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.640+0000 7f1c00a20640 1 --2- 172.21.15.113:0/259281886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bfc0c76f0 0x7f1bfc0c7b20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc06c730 tx=0x7f1bf402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9816e0ff2bc22fe8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.641+0000 7f1bfb7fe640 1 -- 172.21.15.113:0/259281886 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf402ca80 con 0x7f1bfc0c76f0 2023-11-28T17:41:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.641+0000 7f1bfb7fe640 1 -- 172.21.15.113:0/259281886 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf402cbe0 con 0x7f1bfc0c76f0 2023-11-28T17:41:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.641+0000 7f1bfb7fe640 1 -- 172.21.15.113:0/259281886 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40343f0 con 0x7f1bfc0c76f0 2023-11-28T17:41:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.641+0000 7f1c02483640 1 -- 172.21.15.113:0/259281886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bfc0c76f0 msgr2=0x7f1bfc0c7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.642 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.641+0000 7f1c02483640 1 --2- 172.21.15.113:0/259281886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bfc0c76f0 0x7f1bfc0c7b20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc06c730 tx=0x7f1bf402bee0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.643 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.642+0000 7f1c02483640 1 -- 172.21.15.113:0/259281886 shutdown_connections 2023-11-28T17:41:00.643 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.642+0000 7f1c02483640 1 --2- 172.21.15.113:0/259281886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bfc0c76f0 0x7f1bfc0c7b20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.643 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.642+0000 7f1c02483640 1 --2- 172.21.15.113:0/259281886 >> v2:172.21.15.98:3300/0 conn(0x7f1bfc153fb0 0x7f1bfc0c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.643 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.642+0000 7f1c02483640 1 -- 172.21.15.113:0/259281886 >> 172.21.15.113:0/259281886 conn(0x7f1bfc0cf4a0 msgr2=0x7f1bfc0cf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:00.643 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.642+0000 7f1c02483640 1 -- 172.21.15.113:0/259281886 shutdown_connections 2023-11-28T17:41:00.643 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.642+0000 7f1c02483640 1 -- 172.21.15.113:0/259281886 wait complete. 2023-11-28T17:41:00.644 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:00.643+0000 7f44faa12040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T17:41:00.644 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.643+0000 7f1c02483640 1 Processor -- start 2023-11-28T17:41:00.644 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.644+0000 7f1c02483640 1 -- start start 2023-11-28T17:41:00.645 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1c02483640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bfc0c76f0 0x7f1bfc0ce670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.645 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1c02483640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bfc153fb0 0x7f1bfc1496c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.646 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1c02483640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bfc15cbf0 0x7f1bfc149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.646 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1c02483640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc1ef3e0 con 0x7f1bfc0c76f0 2023-11-28T17:41:00.646 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1c02483640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc1f3520 con 0x7f1bfc15cbf0 2023-11-28T17:41:00.646 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1c02483640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc1f3800 con 0x7f1bfc153fb0 2023-11-28T17:41:00.646 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1c00a20640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bfc15cbf0 0x7f1bfc149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.646 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1bf3fff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bfc153fb0 0x7f1bfc1496c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.646 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1c00a20640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bfc15cbf0 0x7f1bfc149c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43486/0 (socket says 172.21.15.113:43486) 2023-11-28T17:41:00.646 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1c00a20640 1 -- 172.21.15.113:0/2694577779 learned_addr learned my addr 172.21.15.113:0/2694577779 (peer_addr_for_me v2:172.21.15.113:0/0) 2023-11-28T17:41:00.646 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1bfbfff640 1 --2- 172.21.15.113:0/2694577779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bfc0c76f0 0x7f1bfc0ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.647 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1c00a20640 1 -- 172.21.15.113:0/2694577779 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bfc153fb0 msgr2=0x7f1bfc1496c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.647 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1c00a20640 1 --2- 172.21.15.113:0/2694577779 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bfc153fb0 0x7f1bfc1496c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.648 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1c00a20640 1 -- 172.21.15.113:0/2694577779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bfc0c76f0 msgr2=0x7f1bfc0ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.648 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1c00a20640 1 --2- 172.21.15.113:0/2694577779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bfc0c76f0 0x7f1bfc0ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.648 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1c00a20640 1 -- 172.21.15.113:0/2694577779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf4009b30 con 0x7f1bfc15cbf0 2023-11-28T17:41:00.648 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.645+0000 7f1bf3fff640 1 --2- 172.21.15.113:0/2694577779 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bfc153fb0 0x7f1bfc1496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:41:00.648 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.646+0000 7f1c00a20640 1 --2- 172.21.15.113:0/2694577779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bfc15cbf0 0x7f1bfc149c00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009f40 tx=0x7f1bf4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:00.648 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.646+0000 7f1bf97fa640 1 -- 172.21.15.113:0/2694577779 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4009510 con 0x7f1bfc15cbf0 2023-11-28T17:41:00.648 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.646+0000 7f1bf97fa640 1 -- 172.21.15.113:0/2694577779 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf403e450 con 0x7f1bfc15cbf0 2023-11-28T17:41:00.648 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.646+0000 7f1c02483640 1 -- 172.21.15.113:0/2694577779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bfc15a480 con 0x7f1bfc15cbf0 2023-11-28T17:41:00.648 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.646+0000 7f1bf97fa640 1 -- 172.21.15.113:0/2694577779 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf403e800 con 0x7f1bfc15cbf0 2023-11-28T17:41:00.648 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.646+0000 7f1c02483640 1 -- 172.21.15.113:0/2694577779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bfc159c70 con 0x7f1bfc15cbf0 2023-11-28T17:41:00.648 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.646+0000 7f1bf97fa640 1 -- 172.21.15.113:0/2694577779 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1bf402cd50 con 0x7f1bfc15cbf0 2023-11-28T17:41:00.648 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.646+0000 7f1bf97fa640 1 -- 172.21.15.113:0/2694577779 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f1bf403f400 con 0x7f1bfc15cbf0 2023-11-28T17:41:00.649 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.647+0000 7f1c02483640 1 -- 172.21.15.113:0/2694577779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bfc074f60 con 0x7f1bfc15cbf0 2023-11-28T17:41:00.650 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.650+0000 7f1bf97fa640 1 -- 172.21.15.113:0/2694577779 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f1bf4043020 con 0x7f1bfc15cbf0 2023-11-28T17:41:00.698 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:41:00.697+0000 7feeef248040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T17:41:00.728 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.727+0000 7f1c02483640 1 -- 172.21.15.113:0/2694577779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a6a41999-a583-4c92-8675-5c5336e9319a", "id": 6} v 0) v1 -- 0x7f1bfc1ef550 con 0x7f1bfc15cbf0 2023-11-28T17:41:00.737 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.737+0000 7f1bf97fa640 1 -- 172.21.15.113:0/2694577779 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a6a41999-a583-4c92-8675-5c5336e9319a", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1bf40560b0 con 0x7f1bfc15cbf0 2023-11-28T17:41:00.737 INFO:teuthology.orchestra.run.smithi113.stdout:6 2023-11-28T17:41:00.738 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.738+0000 7f1c02483640 1 -- 172.21.15.113:0/2694577779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bfc15cbf0 msgr2=0x7f1bfc149c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.738 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.738+0000 7f1c02483640 1 --2- 172.21.15.113:0/2694577779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bfc15cbf0 0x7f1bfc149c00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009f40 tx=0x7f1bf4004a40 comp rx=0 tx=0).stop 2023-11-28T17:41:00.739 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.738+0000 7f1c02483640 1 -- 172.21.15.113:0/2694577779 shutdown_connections 2023-11-28T17:41:00.739 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.738+0000 7f1c02483640 1 --2- 172.21.15.113:0/2694577779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1bfc15cbf0 0x7f1bfc149c00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.739 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.738+0000 7f1c02483640 1 --2- 172.21.15.113:0/2694577779 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1bfc153fb0 0x7f1bfc1496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.739 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.738+0000 7f1c02483640 1 --2- 172.21.15.113:0/2694577779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bfc0c76f0 0x7f1bfc0ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.739 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.738+0000 7f1c02483640 1 -- 172.21.15.113:0/2694577779 >> 172.21.15.113:0/2694577779 conn(0x7f1bfc0cf4a0 msgr2=0x7f1bfc1528d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:00.739 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.738+0000 7f1c02483640 1 -- 172.21.15.113:0/2694577779 shutdown_connections 2023-11-28T17:41:00.739 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.738+0000 7f1c02483640 1 -- 172.21.15.113:0/2694577779 wait complete. 2023-11-28T17:41:00.755 DEBUG:teuthology.orchestra.run.smithi113:> sudo ceph --cluster ceph osd new 7ab8ecaf-289c-444c-a54d-ffa2cacfa663 7 2023-11-28T17:41:00.812 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:00.811+0000 7f44faa12040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T17:41:00.942 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.941+0000 7f0628c9f640 1 Processor -- start 2023-11-28T17:41:00.942 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.941+0000 7f0628c9f640 1 -- start start 2023-11-28T17:41:00.948 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.942+0000 7f0628c9f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0624071da0 0x7f0624072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.948 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.942+0000 7f0628c9f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06240709f0 con 0x7f0624150c30 2023-11-28T17:41:00.948 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.942+0000 7f0628c9f640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06240cc830 con 0x7f0624066600 2023-11-28T17:41:00.948 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.942+0000 7f0628c9f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0624071b50 con 0x7f0624071da0 2023-11-28T17:41:00.948 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.942+0000 7f061bfff640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f0624066600 0x7f0624071440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.113:40148/0 (socket says 172.21.15.113:40148) 2023-11-28T17:41:00.948 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f061bfff640 1 -- 172.21.15.113:0/3725244297 learned_addr learned my addr 172.21.15.113:0/3725244297 (peer_addr_for_me v1:172.21.15.113:0/0) 2023-11-28T17:41:00.948 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0623fff640 1 --2- 172.21.15.113:0/3725244297 >> v2:172.21.15.98:3300/0 conn(0x7f0624071da0 0x7f0624072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.948 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0622ffd640 1 -- 172.21.15.113:0/3725244297 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144060811 0 0) 0x7f06240cc830 con 0x7f0624066600 2023-11-28T17:41:00.948 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0622ffd640 1 -- 172.21.15.113:0/3725244297 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0610003520 con 0x7f0624066600 2023-11-28T17:41:00.948 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0622ffd640 1 -- 172.21.15.113:0/3725244297 <== mon.1 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2017051970 0 0) 0x7f0610003520 con 0x7f0624066600 2023-11-28T17:41:00.949 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0622ffd640 1 -- 172.21.15.113:0/3725244297 >> v2:172.21.15.98:3300/0 conn(0x7f0624071da0 msgr2=0x7f0624072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:00.949 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0622ffd640 1 --2- 172.21.15.113:0/3725244297 >> v2:172.21.15.98:3300/0 conn(0x7f0624071da0 0x7f0624072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.949 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0622ffd640 1 -- 172.21.15.113:0/3725244297 >> v1:172.21.15.98:6789/0 conn(0x7f0624150c30 legacy=0x7f06240c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:00.949 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0622ffd640 1 -- 172.21.15.113:0/3725244297 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06240c2ee0 con 0x7f0624066600 2023-11-28T17:41:00.949 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0622ffd640 1 -- 172.21.15.113:0/3725244297 <== mon.1 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f060c001670 con 0x7f0624066600 2023-11-28T17:41:00.949 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0622ffd640 1 -- 172.21.15.113:0/3725244297 >> v1:172.21.15.113:6789/0 conn(0x7f0624066600 legacy=0x7f0624071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.949 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0622ffd640 1 --2- 172.21.15.113:0/3725244297 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06100039c0 0x7f0610003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.949 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0622ffd640 1 --2- 172.21.15.113:0/3725244297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06100047c0 0x7f0610004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.949 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0622ffd640 1 --2- 172.21.15.113:0/3725244297 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0610005180 0x7f061000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.949 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0622ffd640 1 -- 172.21.15.113:0/3725244297 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0624071b50 con 0x7f06100047c0 2023-11-28T17:41:00.949 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0622ffd640 1 -- 172.21.15.113:0/3725244297 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061000c050 con 0x7f0610005180 2023-11-28T17:41:00.949 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.943+0000 7f0622ffd640 1 -- 172.21.15.113:0/3725244297 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061000c210 con 0x7f06100039c0 2023-11-28T17:41:00.949 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.944+0000 7f06237fe640 1 --2- 172.21.15.113:0/3725244297 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06100039c0 0x7f0610003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.950 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.944+0000 7f0623fff640 1 --2- 172.21.15.113:0/3725244297 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0610005180 0x7f061000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.950 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.944+0000 7f061bfff640 1 --2- 172.21.15.113:0/3725244297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06100047c0 0x7f0610004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.950 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.944+0000 7f0623fff640 1 -- 172.21.15.113:0/3725244297 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06100039c0 msgr2=0x7f0610003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:00.950 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.944+0000 7f0623fff640 1 --2- 172.21.15.113:0/3725244297 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06100039c0 0x7f0610003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.950 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.944+0000 7f0623fff640 1 -- 172.21.15.113:0/3725244297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06100047c0 msgr2=0x7f0610004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:00.950 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.944+0000 7f0623fff640 1 --2- 172.21.15.113:0/3725244297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06100047c0 0x7f0610004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.950 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.944+0000 7f0623fff640 1 -- 172.21.15.113:0/3725244297 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0610003520 con 0x7f0610005180 2023-11-28T17:41:00.950 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.945+0000 7f0623fff640 1 -- 172.21.15.113:0/3725244297 reap_dead start 2023-11-28T17:41:00.950 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.945+0000 7f0623fff640 1 --2- 172.21.15.113:0/3725244297 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0610005180 0x7f061000b980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f061c009fc0 tx=0x7f06100057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33c1c955200b69f5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:00.950 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.945+0000 7f0622ffd640 1 -- 172.21.15.113:0/3725244297 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c02cc60 con 0x7f0610005180 2023-11-28T17:41:00.950 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.945+0000 7f0622ffd640 1 -- 172.21.15.113:0/3725244297 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061c00f040 con 0x7f0610005180 2023-11-28T17:41:00.950 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.945+0000 7f0622ffd640 1 -- 172.21.15.113:0/3725244297 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c003990 con 0x7f0610005180 2023-11-28T17:41:00.950 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.946+0000 7f0628c9f640 1 -- 172.21.15.113:0/3725244297 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0610005180 msgr2=0x7f061000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.951 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.946+0000 7f0628c9f640 1 --2- 172.21.15.113:0/3725244297 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0610005180 0x7f061000b980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f061c009fc0 tx=0x7f06100057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.951 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.946+0000 7f0628c9f640 1 -- 172.21.15.113:0/3725244297 shutdown_connections 2023-11-28T17:41:00.951 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.946+0000 7f0628c9f640 1 --2- 172.21.15.113:0/3725244297 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0610005180 0x7f061000b980 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.951 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.946+0000 7f0628c9f640 1 -- 172.21.15.113:0/3725244297 >> 172.21.15.113:0/3725244297 conn(0x7f06240cb8f0 msgr2=0x7f06240cbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:00.951 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.946+0000 7f0628c9f640 1 -- 172.21.15.113:0/3725244297 shutdown_connections 2023-11-28T17:41:00.951 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.946+0000 7f0628c9f640 1 -- 172.21.15.113:0/3725244297 wait complete. 2023-11-28T17:41:00.953 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.948+0000 7f0628c9f640 1 Processor -- start 2023-11-28T17:41:00.953 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.948+0000 7f0628c9f640 1 -- start start 2023-11-28T17:41:00.953 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f0628c9f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0610005180 0x7f0624152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.953 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f0628c9f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06241f03b0 0x7f0624155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.953 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f0628c9f640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06241534d0 0x7f06241557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:00.954 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f0628c9f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0624157b60 con 0x7f06241f03b0 2023-11-28T17:41:00.954 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f0628c9f640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06241f0880 con 0x7f0610005180 2023-11-28T17:41:00.954 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f0628c9f640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0624153df0 con 0x7f06241534d0 2023-11-28T17:41:00.954 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f06237fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0610005180 0x7f0624152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.954 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f06237fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0610005180 0x7f0624152a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:43514/0 (socket says 172.21.15.113:43514) 2023-11-28T17:41:00.954 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f06237fe640 1 -- 172.21.15.113:0/2673729834 learned_addr learned my addr 172.21.15.113:0/2673729834 (peer_addr_for_me v2:172.21.15.113:0/0) 2023-11-28T17:41:00.954 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f0623fff640 1 --2- 172.21.15.113:0/2673729834 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06241534d0 0x7f06241557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:00.954 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f06237fe640 1 -- 172.21.15.113:0/2673729834 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06241534d0 msgr2=0x7f06241557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.954 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f06237fe640 1 --2- 172.21.15.113:0/2673729834 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06241534d0 0x7f06241557a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.954 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f06237fe640 1 -- 172.21.15.113:0/2673729834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06241f03b0 msgr2=0x7f0624155260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:00.954 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f06237fe640 1 --2- 172.21.15.113:0/2673729834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06241f03b0 0x7f0624155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:00.954 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f06237fe640 1 -- 172.21.15.113:0/2673729834 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06241540d0 con 0x7f0610005180 2023-11-28T17:41:00.954 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.949+0000 7f06237fe640 1 --2- 172.21.15.113:0/2673729834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0610005180 0x7f0624152a30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f061400f5f0 tx=0x7f061400fab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:00.955 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.950+0000 7f0620ff9640 1 -- 172.21.15.113:0/2673729834 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0614003a70 con 0x7f0610005180 2023-11-28T17:41:00.955 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.950+0000 7f0620ff9640 1 -- 172.21.15.113:0/2673729834 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0614003bd0 con 0x7f0610005180 2023-11-28T17:41:00.955 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.950+0000 7f0620ff9640 1 -- 172.21.15.113:0/2673729834 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0614004050 con 0x7f0610005180 2023-11-28T17:41:00.955 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.950+0000 7f0628c9f640 1 -- 172.21.15.113:0/2673729834 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0624157510 con 0x7f0610005180 2023-11-28T17:41:00.955 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.950+0000 7f0628c9f640 1 -- 172.21.15.113:0/2673729834 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0624152f70 con 0x7f0610005180 2023-11-28T17:41:00.955 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.950+0000 7f0620ff9640 1 -- 172.21.15.113:0/2673729834 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f06140044b0 con 0x7f0610005180 2023-11-28T17:41:00.955 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.950+0000 7f0620ff9640 1 -- 172.21.15.113:0/2673729834 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f061400dcc0 con 0x7f0610005180 2023-11-28T17:41:00.955 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.951+0000 7f0628c9f640 1 -- 172.21.15.113:0/2673729834 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06240c5820 con 0x7f0610005180 2023-11-28T17:41:00.955 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:00.951+0000 7f44faa12040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T17:41:00.956 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:00.955+0000 7f0620ff9640 1 -- 172.21.15.113:0/2673729834 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f061403bce0 con 0x7f0610005180 2023-11-28T17:41:01.007 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:01.006+0000 7f44faa12040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T17:41:01.033 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:01.033+0000 7f0628c9f640 1 -- 172.21.15.113:0/2673729834 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7ab8ecaf-289c-444c-a54d-ffa2cacfa663", "id": 7} v 0) v1 -- 0x7f062406ff10 con 0x7f0610005180 2023-11-28T17:41:01.040 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:01.039+0000 7f0620ff9640 1 -- 172.21.15.113:0/2673729834 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7ab8ecaf-289c-444c-a54d-ffa2cacfa663", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f061402a070 con 0x7f0610005180 2023-11-28T17:41:01.040 INFO:teuthology.orchestra.run.smithi113.stdout:7 2023-11-28T17:41:01.041 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:01.041+0000 7f0628c9f640 1 -- 172.21.15.113:0/2673729834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0610005180 msgr2=0x7f0624152a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:01.041 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:01.041+0000 7f0628c9f640 1 --2- 172.21.15.113:0/2673729834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0610005180 0x7f0624152a30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f061400f5f0 tx=0x7f061400fab0 comp rx=0 tx=0).stop 2023-11-28T17:41:01.042 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:01.041+0000 7f0628c9f640 1 -- 172.21.15.113:0/2673729834 shutdown_connections 2023-11-28T17:41:01.042 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:01.041+0000 7f0628c9f640 1 --2- 172.21.15.113:0/2673729834 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f06241534d0 0x7f06241557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:01.042 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:01.041+0000 7f0628c9f640 1 --2- 172.21.15.113:0/2673729834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06241f03b0 0x7f0624155260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:01.042 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:01.041+0000 7f0628c9f640 1 --2- 172.21.15.113:0/2673729834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0610005180 0x7f0624152a30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:01.042 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:01.041+0000 7f0628c9f640 1 -- 172.21.15.113:0/2673729834 >> 172.21.15.113:0/2673729834 conn(0x7f06240cb8f0 msgr2=0x7f06240cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:01.042 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:01.041+0000 7f0628c9f640 1 -- 172.21.15.113:0/2673729834 shutdown_connections 2023-11-28T17:41:01.042 INFO:teuthology.orchestra.run.smithi113.stderr:2023-11-28T17:41:01.041+0000 7f0628c9f640 1 -- 172.21.15.113:0/2673729834 wait complete. 2023-11-28T17:41:01.057 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-28T17:41:01.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-11-28T17:41:01.067 INFO:tasks.ceph.osd.0:Started 2023-11-28T17:41:01.067 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-28T17:41:01.067 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-11-28T17:41:01.077 INFO:tasks.ceph.osd.1:Started 2023-11-28T17:41:01.077 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-28T17:41:01.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-11-28T17:41:01.086 INFO:tasks.ceph.osd.2:Started 2023-11-28T17:41:01.086 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-28T17:41:01.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-11-28T17:41:01.093 INFO:tasks.ceph.osd.3:Started 2023-11-28T17:41:01.093 INFO:tasks.ceph.osd.4:Restarting daemon 2023-11-28T17:41:01.094 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-11-28T17:41:01.102 INFO:tasks.ceph.osd.4:Started 2023-11-28T17:41:01.103 INFO:tasks.ceph.osd.5:Restarting daemon 2023-11-28T17:41:01.103 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-11-28T17:41:01.111 INFO:tasks.ceph.osd.5:Started 2023-11-28T17:41:01.111 INFO:tasks.ceph.osd.6:Restarting daemon 2023-11-28T17:41:01.111 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-11-28T17:41:01.116 INFO:tasks.ceph.osd.6:Started 2023-11-28T17:41:01.116 INFO:tasks.ceph.osd.7:Restarting daemon 2023-11-28T17:41:01.116 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-11-28T17:41:01.122 INFO:tasks.ceph.osd.7:Started 2023-11-28T17:41:01.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:41:01.125 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:01.124+0000 7f44faa12040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T17:41:01.273 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:01.272+0000 7f44faa12040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T17:41:01.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.339+0000 7fa9d91fa640 1 Processor -- start 2023-11-28T17:41:01.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.339+0000 7fa9d91fa640 1 -- start start 2023-11-28T17:41:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.341+0000 7fa9d91fa640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9d4153bb0 0x7fa9d415a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.341+0000 7fa9d91fa640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9d40ca390 0x7fa9d40ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.341+0000 7fa9d91fa640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d415d330 con 0x7fa9d40cf280 2023-11-28T17:41:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.341+0000 7fa9d91fa640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9d415d120 con 0x7fa9d4153bb0 2023-11-28T17:41:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.341+0000 7fa9d91fa640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d415d610 con 0x7fa9d40ca390 2023-11-28T17:41:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.341+0000 7fa9d3fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9d4153bb0 0x7fa9d415a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.341+0000 7fa9d3fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9d4153bb0 0x7fa9d415a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39004/0 (socket says 172.21.15.98:39004) 2023-11-28T17:41:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.341+0000 7fa9d3fff640 1 -- 172.21.15.98:0/3745394756 learned_addr learned my addr 172.21.15.98:0/3745394756 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.341+0000 7fa9cbfff640 1 --2- 172.21.15.98:0/3745394756 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9d40ca390 0x7fa9d40ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.341+0000 7fa9d3fff640 1 -- 172.21.15.98:0/3745394756 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9d40ca390 msgr2=0x7fa9d40ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.341+0000 7fa9d3fff640 1 --2- 172.21.15.98:0/3745394756 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9d40ca390 0x7fa9d40ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.341+0000 7fa9d3fff640 1 -- 172.21.15.98:0/3745394756 >> v1:172.21.15.113:6789/0 conn(0x7fa9d40cf280 legacy=0x7fa9d40b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.341+0000 7fa9d3fff640 1 -- 172.21.15.98:0/3745394756 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d415d8f0 con 0x7fa9d4153bb0 2023-11-28T17:41:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.342+0000 7fa9d3fff640 1 --2- 172.21.15.98:0/3745394756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d4153bb0 0x7fa9d415a6d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4009870 tx=0x7fa9c402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9cb723f3047fe91 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.343+0000 7fa9d37fe640 1 -- 172.21.15.98:0/3745394756 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c402f840 con 0x7fa9d4153bb0 2023-11-28T17:41:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.343+0000 7fa9d37fe640 1 -- 172.21.15.98:0/3745394756 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c402f9a0 con 0x7fa9d4153bb0 2023-11-28T17:41:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.343+0000 7fa9d37fe640 1 -- 172.21.15.98:0/3745394756 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c402fd30 con 0x7fa9d4153bb0 2023-11-28T17:41:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.343+0000 7fa9d91fa640 1 -- 172.21.15.98:0/3745394756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d4153bb0 msgr2=0x7fa9d415a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.343+0000 7fa9d91fa640 1 --2- 172.21.15.98:0/3745394756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d4153bb0 0x7fa9d415a6d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4009870 tx=0x7fa9c402ba20 comp rx=0 tx=0).stop 2023-11-28T17:41:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.343+0000 7fa9d91fa640 1 -- 172.21.15.98:0/3745394756 shutdown_connections 2023-11-28T17:41:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.343+0000 7fa9d91fa640 1 --2- 172.21.15.98:0/3745394756 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9d40ca390 0x7fa9d40ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.343+0000 7fa9d91fa640 1 --2- 172.21.15.98:0/3745394756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d4153bb0 0x7fa9d415a6d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.343+0000 7fa9d91fa640 1 -- 172.21.15.98:0/3745394756 >> 172.21.15.98:0/3745394756 conn(0x7fa9d4071c00 msgr2=0x7fa9d4072010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.343+0000 7fa9d91fa640 1 -- 172.21.15.98:0/3745394756 shutdown_connections 2023-11-28T17:41:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.343+0000 7fa9d91fa640 1 -- 172.21.15.98:0/3745394756 wait complete. 2023-11-28T17:41:01.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.345+0000 7fa9d91fa640 1 Processor -- start 2023-11-28T17:41:01.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.345+0000 7fa9d91fa640 1 -- start start 2023-11-28T17:41:01.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.346+0000 7fa9d91fa640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9d40ca390 0x7fa9d41ef870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:01.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.346+0000 7fa9d91fa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d40cf280 0x7fa9d41e64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.346+0000 7fa9d91fa640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9d4153bb0 0x7fa9d4149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.346+0000 7fa9d91fa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d41f3560 con 0x7fa9d40cf280 2023-11-28T17:41:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.346+0000 7fa9d91fa640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d40b1fd0 con 0x7fa9d4153bb0 2023-11-28T17:41:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.346+0000 7fa9d91fa640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d40b1cf0 con 0x7fa9d40ca390 2023-11-28T17:41:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.346+0000 7fa9cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d40cf280 0x7fa9d41e64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.346+0000 7fa9cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d40cf280 0x7fa9d41e64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39010/0 (socket says 172.21.15.98:39010) 2023-11-28T17:41:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.346+0000 7fa9cbfff640 1 -- 172.21.15.98:0/1763805482 learned_addr learned my addr 172.21.15.98:0/1763805482 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.346+0000 7fa9d3fff640 1 --2- 172.21.15.98:0/1763805482 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9d40ca390 0x7fa9d41ef870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.346+0000 7fa9d89f9640 1 --2- 172.21.15.98:0/1763805482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9d4153bb0 0x7fa9d4149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.347+0000 7fa9cbfff640 1 -- 172.21.15.98:0/1763805482 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9d40ca390 msgr2=0x7fa9d41ef870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:01.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.347+0000 7fa9cbfff640 1 --2- 172.21.15.98:0/1763805482 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9d40ca390 0x7fa9d41ef870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:01.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.347+0000 7fa9cbfff640 1 -- 172.21.15.98:0/1763805482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9d4153bb0 msgr2=0x7fa9d4149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:01.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.347+0000 7fa9cbfff640 1 --2- 172.21.15.98:0/1763805482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9d4153bb0 0x7fa9d4149c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:01.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.347+0000 7fa9cbfff640 1 -- 172.21.15.98:0/1763805482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c40094d0 con 0x7fa9d40cf280 2023-11-28T17:41:01.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.347+0000 7fa9d3fff640 1 --2- 172.21.15.98:0/1763805482 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9d40ca390 0x7fa9d41ef870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:41:01.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.347+0000 7fa9cbfff640 1 --2- 172.21.15.98:0/1763805482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d40cf280 0x7fa9d41e64e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00e9b0 tx=0x7fa9bc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:01.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.347+0000 7fa9d17fa640 1 -- 172.21.15.98:0/1763805482 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc009670 con 0x7fa9d40cf280 2023-11-28T17:41:01.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.347+0000 7fa9d17fa640 1 -- 172.21.15.98:0/1763805482 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9bc00c9a0 con 0x7fa9d40cf280 2023-11-28T17:41:01.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.347+0000 7fa9d91fa640 1 -- 172.21.15.98:0/1763805482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d41f2f10 con 0x7fa9d40cf280 2023-11-28T17:41:01.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.347+0000 7fa9d17fa640 1 -- 172.21.15.98:0/1763805482 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc00cd50 con 0x7fa9d40cf280 2023-11-28T17:41:01.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.347+0000 7fa9d91fa640 1 -- 172.21.15.98:0/1763805482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d41efdb0 con 0x7fa9d40cf280 2023-11-28T17:41:01.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.348+0000 7fa9d17fa640 1 -- 172.21.15.98:0/1763805482 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa9bc013770 con 0x7fa9d40cf280 2023-11-28T17:41:01.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.348+0000 7fa9d17fa640 1 -- 172.21.15.98:0/1763805482 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fa9bc01c450 con 0x7fa9d40cf280 2023-11-28T17:41:01.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.348+0000 7fa9d91fa640 1 -- 172.21.15.98:0/1763805482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d40ca100 con 0x7fa9d40cf280 2023-11-28T17:41:01.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.351+0000 7fa9d17fa640 1 -- 172.21.15.98:0/1763805482 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fa9bc021060 con 0x7fa9d40cf280 2023-11-28T17:41:01.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.416+0000 7fa9d91fa640 1 -- 172.21.15.98:0/1763805482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9d41f36d0 con 0x7fa9d40cf280 2023-11-28T17:41:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.416+0000 7fa9d17fa640 1 -- 172.21.15.98:0/1763805482 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fa9bc02a020 con 0x7fa9d40cf280 2023-11-28T17:41:01.417 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:01.417 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":10,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:41:01.035090+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:41:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.417+0000 7fa9d91fa640 1 -- 172.21.15.98:0/1763805482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d40cf280 msgr2=0x7fa9d41e64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.417+0000 7fa9d91fa640 1 --2- 172.21.15.98:0/1763805482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d40cf280 0x7fa9d41e64e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00e9b0 tx=0x7fa9bc00ee70 comp rx=0 tx=0).stop 2023-11-28T17:41:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.417+0000 7fa9d91fa640 1 -- 172.21.15.98:0/1763805482 shutdown_connections 2023-11-28T17:41:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.417+0000 7fa9d91fa640 1 --2- 172.21.15.98:0/1763805482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9d4153bb0 0x7fa9d4149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.417+0000 7fa9d91fa640 1 --2- 172.21.15.98:0/1763805482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9d40cf280 0x7fa9d41e64e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.417+0000 7fa9d91fa640 1 --2- 172.21.15.98:0/1763805482 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9d40ca390 0x7fa9d41ef870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.417+0000 7fa9d91fa640 1 -- 172.21.15.98:0/1763805482 >> 172.21.15.98:0/1763805482 conn(0x7fa9d4071c00 msgr2=0x7fa9d4151b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.417+0000 7fa9d91fa640 1 -- 172.21.15.98:0/1763805482 shutdown_connections 2023-11-28T17:41:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:01.418+0000 7fa9d91fa640 1 -- 172.21.15.98:0/1763805482 wait complete. 2023-11-28T17:41:01.435 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-28T17:41:01.435 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-28T17:41:01.446 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:01.444+0000 7f44faa12040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T17:41:01.495 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:01.494+0000 7f44faa12040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T17:41:01.653 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:01.653+0000 7f44faa12040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T17:41:01.726 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:41:01.724+0000 7feeef248040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T17:41:01.785 INFO:tasks.ceph.osd.1.smithi098.stderr:2023-11-28T17:41:01.784+0000 7f3f77020500 -1 Falling back to public interface 2023-11-28T17:41:01.787 INFO:tasks.ceph.osd.4.smithi113.stderr:2023-11-28T17:41:01.787+0000 7fd182223500 -1 Falling back to public interface 2023-11-28T17:41:01.793 INFO:tasks.ceph.osd.0.smithi098.stderr:2023-11-28T17:41:01.793+0000 7ff52521b500 -1 Falling back to public interface 2023-11-28T17:41:01.794 INFO:tasks.ceph.osd.2.smithi098.stderr:2023-11-28T17:41:01.793+0000 7f3ece02a500 -1 Falling back to public interface 2023-11-28T17:41:01.804 INFO:tasks.ceph.osd.3.smithi098.stderr:2023-11-28T17:41:01.803+0000 7f964c8d2500 -1 Falling back to public interface 2023-11-28T17:41:01.810 INFO:tasks.ceph.osd.6.smithi113.stderr:2023-11-28T17:41:01.809+0000 7fb5e0d1e500 -1 Falling back to public interface 2023-11-28T17:41:01.836 INFO:tasks.ceph.osd.5.smithi113.stderr:2023-11-28T17:41:01.835+0000 7fb863614500 -1 Falling back to public interface 2023-11-28T17:41:01.840 INFO:tasks.ceph.osd.7.smithi113.stderr:2023-11-28T17:41:01.840+0000 7f5184212500 -1 Falling back to public interface 2023-11-28T17:41:02.033 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:41:02.032+0000 7feeef248040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T17:41:02.094 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:41:02.093+0000 7feeef248040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T17:41:02.210 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:41:02.210+0000 7feeef248040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T17:41:02.239 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T17:41:02.381 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.380+0000 7fb2ed4c2640 1 Processor -- start 2023-11-28T17:41:02.381 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.380+0000 7fb2ed4c2640 1 -- start start 2023-11-28T17:41:02.382 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.381+0000 7fb2ed4c2640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2e80cc2f0 0x7fb2e80c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:02.382 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.381+0000 7fb2ed4c2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e815ad90 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.382 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.381+0000 7fb2ed4c2640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e8159f30 con 0x7fb2e8153c10 2023-11-28T17:41:02.382 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.381+0000 7fb2ed4c2640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e80ca530 con 0x7fb2e80cc2f0 2023-11-28T17:41:02.383 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.382+0000 7fb2e6ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2e80cc2f0 0x7fb2e80c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:02.383 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.382+0000 7fb2e77fe640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb2e8153c10 0x7fb2e806b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.98:51148/0 (socket says 172.21.15.98:51148) 2023-11-28T17:41:02.383 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.382+0000 7fb2e77fe640 1 -- 172.21.15.98:0/1385190579 learned_addr learned my addr 172.21.15.98:0/1385190579 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:02.383 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.382+0000 7fb2e5ffb640 1 -- 172.21.15.98:0/1385190579 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993981809 0 0) 0x7fb2e815ad90 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.383 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.382+0000 7fb2e5ffb640 1 -- 172.21.15.98:0/1385190579 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d0003520 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.384 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.382+0000 7fb2e5ffb640 1 -- 172.21.15.98:0/1385190579 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581387261 0 0) 0x7fb2e8159f30 con 0x7fb2e8153c10 2023-11-28T17:41:02.384 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.382+0000 7fb2e5ffb640 1 -- 172.21.15.98:0/1385190579 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e815ad90 con 0x7fb2e8153c10 2023-11-28T17:41:02.384 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.382+0000 7fb2e6ffd640 1 -- 172.21.15.98:0/1385190579 >> v1:172.21.15.113:6789/0 conn(0x7fb2e8153c10 legacy=0x7fb2e806b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:02.384 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.382+0000 7fb2e6ffd640 1 -- 172.21.15.98:0/1385190579 >> v1:172.21.15.98:6789/0 conn(0x7fb2e80cc6d0 legacy=0x7fb2e80cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:02.384 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.382+0000 7fb2e6ffd640 1 -- 172.21.15.98:0/1385190579 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e815a210 con 0x7fb2e80cc2f0 2023-11-28T17:41:02.384 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.384+0000 7fb2e6ffd640 1 --2- 172.21.15.98:0/1385190579 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2e80cc2f0 0x7fb2e80c9ff0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80097d0 tx=0x7fb2d802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=690c57da2e87a703 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:02.385 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.384+0000 7fb2e5ffb640 1 -- 172.21.15.98:0/1385190579 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8031800 con 0x7fb2e80cc2f0 2023-11-28T17:41:02.385 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.384+0000 7fb2e5ffb640 1 -- 172.21.15.98:0/1385190579 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d8031960 con 0x7fb2e80cc2f0 2023-11-28T17:41:02.385 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.384+0000 7fb2e5ffb640 1 -- 172.21.15.98:0/1385190579 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8031cf0 con 0x7fb2e80cc2f0 2023-11-28T17:41:02.385 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.384+0000 7fb2ed4c2640 1 -- 172.21.15.98:0/1385190579 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2e80cc2f0 msgr2=0x7fb2e80c9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:02.385 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.384+0000 7fb2ed4c2640 1 --2- 172.21.15.98:0/1385190579 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2e80cc2f0 0x7fb2e80c9ff0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80097d0 tx=0x7fb2d802b8a0 comp rx=0 tx=0).stop 2023-11-28T17:41:02.386 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.384+0000 7fb2ed4c2640 1 -- 172.21.15.98:0/1385190579 shutdown_connections 2023-11-28T17:41:02.386 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.384+0000 7fb2ed4c2640 1 --2- 172.21.15.98:0/1385190579 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2e80cc2f0 0x7fb2e80c9ff0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:02.386 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.384+0000 7fb2ed4c2640 1 -- 172.21.15.98:0/1385190579 >> 172.21.15.98:0/1385190579 conn(0x7fb2e806cd30 msgr2=0x7fb2e80663a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:02.386 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.385+0000 7fb2ed4c2640 1 -- 172.21.15.98:0/1385190579 shutdown_connections 2023-11-28T17:41:02.386 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.385+0000 7fb2ed4c2640 1 -- 172.21.15.98:0/1385190579 wait complete. 2023-11-28T17:41:02.387 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.386+0000 7fb2ed4c2640 1 Processor -- start 2023-11-28T17:41:02.387 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.386+0000 7fb2ed4c2640 1 -- start start 2023-11-28T17:41:02.389 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.387+0000 7fb2ed4c2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2e80cc2f0 0x7fb2e81f3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:02.389 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.387+0000 7fb2ed4c2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e80cc6d0 0x7fb2e814f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:02.389 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.387+0000 7fb2ed4c2640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2e8153c10 0x7fb2e8158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:02.389 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.387+0000 7fb2ed4c2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e81f2a60 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.390 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.387+0000 7fb2ed4c2640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e81f39b0 con 0x7fb2e80cc2f0 2023-11-28T17:41:02.390 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.387+0000 7fb2ed4c2640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e81f2fe0 con 0x7fb2e8153c10 2023-11-28T17:41:02.390 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.387+0000 7fb2e67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e80cc6d0 0x7fb2e814f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:02.390 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.388+0000 7fb2e67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e80cc6d0 0x7fb2e814f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39084/0 (socket says 172.21.15.98:39084) 2023-11-28T17:41:02.390 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.388+0000 7fb2e67fc640 1 -- 172.21.15.98:0/2509002033 learned_addr learned my addr 172.21.15.98:0/2509002033 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:02.390 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.388+0000 7fb2e77fe640 1 --2- 172.21.15.98:0/2509002033 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2e8153c10 0x7fb2e8158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:02.390 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.388+0000 7fb2e6ffd640 1 --2- 172.21.15.98:0/2509002033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2e80cc2f0 0x7fb2e81f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:02.390 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.388+0000 7fb2e67fc640 1 -- 172.21.15.98:0/2509002033 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2e8153c10 msgr2=0x7fb2e8158ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:02.390 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.388+0000 7fb2e67fc640 1 --2- 172.21.15.98:0/2509002033 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2e8153c10 0x7fb2e8158ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:02.390 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.388+0000 7fb2e67fc640 1 -- 172.21.15.98:0/2509002033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2e80cc2f0 msgr2=0x7fb2e81f3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:02.391 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.388+0000 7fb2e67fc640 1 --2- 172.21.15.98:0/2509002033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2e80cc2f0 0x7fb2e81f3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:02.391 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.388+0000 7fb2e67fc640 1 -- 172.21.15.98:0/2509002033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d8009400 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.391 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.388+0000 7fb2e67fc640 1 --2- 172.21.15.98:0/2509002033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e80cc6d0 0x7fb2e814f940 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc00c500 tx=0x7fb2dc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:02.391 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.388+0000 7fb2cf7fe640 1 -- 172.21.15.98:0/2509002033 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc0052d0 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.391 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.388+0000 7fb2cf7fe640 1 -- 172.21.15.98:0/2509002033 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2dc005430 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.391 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.388+0000 7fb2ed4c2640 1 -- 172.21.15.98:0/2509002033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e8154d50 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.391 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.388+0000 7fb2cf7fe640 1 -- 172.21.15.98:0/2509002033 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc012870 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.391 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.388+0000 7fb2ed4c2640 1 -- 172.21.15.98:0/2509002033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e8155310 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.392 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.389+0000 7fb2cf7fe640 1 -- 172.21.15.98:0/2509002033 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 2) v1 ==== 44726+0+0 (secure 0 0 0) 0x7fb2dc0129d0 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.392 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.389+0000 7fb2ed4c2640 1 -- 172.21.15.98:0/2509002033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e8067bf0 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.392 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.390+0000 7fb2cf7fe640 1 -- 172.21.15.98:0/2509002033 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb2dc0495b0 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.393 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.393+0000 7fb2cf7fe640 1 -- 172.21.15.98:0/2509002033 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fb2dc0102e0 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.455 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.454+0000 7fb2ed4c2640 1 -- 172.21.15.98:0/2509002033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2e8074e50 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.456 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.455+0000 7fb2cf7fe640 1 -- 172.21.15.98:0/2509002033 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fb2dc019af0 con 0x7fb2e80cc6d0 2023-11-28T17:41:02.456 INFO:teuthology.misc.health.smithi098.stdout: 2023-11-28T17:41:02.456 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:41:01.035090+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:41:02.457 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.456+0000 7fb2ed4c2640 1 -- 172.21.15.98:0/2509002033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e80cc6d0 msgr2=0x7fb2e814f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:02.457 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.456+0000 7fb2ed4c2640 1 --2- 172.21.15.98:0/2509002033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e80cc6d0 0x7fb2e814f940 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc00c500 tx=0x7fb2dc00c9c0 comp rx=0 tx=0).stop 2023-11-28T17:41:02.457 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.456+0000 7fb2ed4c2640 1 -- 172.21.15.98:0/2509002033 shutdown_connections 2023-11-28T17:41:02.457 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.456+0000 7fb2ed4c2640 1 --2- 172.21.15.98:0/2509002033 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2e8153c10 0x7fb2e8158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:02.457 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.456+0000 7fb2ed4c2640 1 --2- 172.21.15.98:0/2509002033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e80cc6d0 0x7fb2e814f940 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:02.458 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.456+0000 7fb2ed4c2640 1 --2- 172.21.15.98:0/2509002033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2e80cc2f0 0x7fb2e81f3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:02.458 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.456+0000 7fb2ed4c2640 1 -- 172.21.15.98:0/2509002033 >> 172.21.15.98:0/2509002033 conn(0x7fb2e806cd30 msgr2=0x7fb2e80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:02.458 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.457+0000 7fb2ed4c2640 1 -- 172.21.15.98:0/2509002033 shutdown_connections 2023-11-28T17:41:02.458 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:02.457+0000 7fb2ed4c2640 1 -- 172.21.15.98:0/2509002033 wait complete. 2023-11-28T17:41:02.469 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-28T17:41:02.680 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:02.679+0000 7f44faa12040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T17:41:02.993 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:02.993+0000 7f44faa12040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T17:41:03.052 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:03.051+0000 7f44faa12040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T17:41:03.166 INFO:tasks.ceph.mgr.x.smithi113.stderr:2023-11-28T17:41:03.165+0000 7f44faa12040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T17:41:03.246 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:41:03.245+0000 7fee905d1640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-28T17:41:03.247 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:41:03.246+0000 7fee905d1640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-28T17:41:05.121 INFO:tasks.ceph.osd.2.smithi098.stderr:2023-11-28T17:41:05.120+0000 7f3ece02a500 -1 osd.2 0 log_to_monitors true 2023-11-28T17:41:05.295 INFO:tasks.ceph.osd.1.smithi098.stderr:2023-11-28T17:41:05.294+0000 7f3f77020500 -1 osd.1 0 log_to_monitors true 2023-11-28T17:41:05.338 INFO:tasks.ceph.osd.0.smithi098.stderr:2023-11-28T17:41:05.337+0000 7ff52521b500 -1 osd.0 0 log_to_monitors true 2023-11-28T17:41:05.355 INFO:tasks.ceph.osd.3.smithi098.stderr:2023-11-28T17:41:05.354+0000 7f964c8d2500 -1 osd.3 0 log_to_monitors true 2023-11-28T17:41:05.361 INFO:tasks.ceph.osd.4.smithi113.stderr:2023-11-28T17:41:05.361+0000 7fd182223500 -1 osd.4 0 log_to_monitors true 2023-11-28T17:41:05.363 INFO:tasks.ceph.osd.6.smithi113.stderr:2023-11-28T17:41:05.363+0000 7fb5e0d1e500 -1 osd.6 0 log_to_monitors true 2023-11-28T17:41:05.418 INFO:tasks.ceph.osd.5.smithi113.stderr:2023-11-28T17:41:05.418+0000 7fb863614500 -1 osd.5 0 log_to_monitors true 2023-11-28T17:41:05.428 INFO:tasks.ceph.osd.7.smithi113.stderr:2023-11-28T17:41:05.427+0000 7f5184212500 -1 osd.7 0 log_to_monitors true 2023-11-28T17:41:06.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:41:06.734 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:41:07.213 INFO:tasks.ceph.osd.2.smithi098.stderr:2023-11-28T17:41:07.213+0000 7f3ec95fe640 -1 osd.2 0 waiting for initial osdmap 2023-11-28T17:41:07.224 INFO:tasks.ceph.osd.2.smithi098.stderr:2023-11-28T17:41:07.224+0000 7f3ec1425640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:41:08.554 INFO:tasks.ceph.osd.0.smithi098.stderr:2023-11-28T17:41:08.553+0000 7ff520b7e640 -1 osd.0 0 waiting for initial osdmap 2023-11-28T17:41:08.564 INFO:tasks.ceph.osd.0.smithi098.stderr:2023-11-28T17:41:08.563+0000 7ff5189a5640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:41:08.576 INFO:tasks.ceph.osd.3.smithi098.stderr:2023-11-28T17:41:08.576+0000 7f9648860640 -1 osd.3 0 waiting for initial osdmap 2023-11-28T17:41:08.577 INFO:tasks.ceph.osd.1.smithi098.stderr:2023-11-28T17:41:08.576+0000 7f3f73204640 -1 osd.1 0 waiting for initial osdmap 2023-11-28T17:41:08.587 INFO:tasks.ceph.osd.3.smithi098.stderr:2023-11-28T17:41:08.587+0000 7f9640687640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:41:08.588 INFO:tasks.ceph.osd.1.smithi098.stderr:2023-11-28T17:41:08.588+0000 7f3f6a817640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:41:08.701 INFO:tasks.ceph.osd.5.smithi113.stderr:2023-11-28T17:41:08.700+0000 7fb85f5a2640 -1 osd.5 0 waiting for initial osdmap 2023-11-28T17:41:08.706 INFO:tasks.ceph.osd.5.smithi113.stderr:2023-11-28T17:41:08.706+0000 7fb8573c9640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:41:08.772 INFO:tasks.ceph.osd.7.smithi113.stderr:2023-11-28T17:41:08.772+0000 7f5180356640 -1 osd.7 0 waiting for initial osdmap 2023-11-28T17:41:08.774 INFO:tasks.ceph.osd.6.smithi113.stderr:2023-11-28T17:41:08.773+0000 7fb5dccac640 -1 osd.6 0 waiting for initial osdmap 2023-11-28T17:41:08.780 INFO:tasks.ceph.osd.7.smithi113.stderr:2023-11-28T17:41:08.779+0000 7f5177969640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:41:08.783 INFO:tasks.ceph.osd.4.smithi113.stderr:2023-11-28T17:41:08.783+0000 7fd17dc13640 -1 osd.4 0 waiting for initial osdmap 2023-11-28T17:41:08.790 INFO:tasks.ceph.osd.6.smithi113.stderr:2023-11-28T17:41:08.790+0000 7fb5d4ad3640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:41:08.794 INFO:tasks.ceph.osd.4.smithi113.stderr:2023-11-28T17:41:08.794+0000 7fd175a3a640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:41:09.275 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T17:41:09.457 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.456+0000 7f138edce640 1 Processor -- start 2023-11-28T17:41:09.457 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.457+0000 7f138edce640 1 -- start start 2023-11-28T17:41:09.459 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.458+0000 7f138edce640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f13880c38e0 0x7f13880c3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:09.459 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.458+0000 7f138edce640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13880c4270 con 0x7f1388151ce0 2023-11-28T17:41:09.459 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.458+0000 7f138edce640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13880c3450 con 0x7f1388158720 2023-11-28T17:41:09.459 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.458+0000 7f138edce640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13880c3020 con 0x7f13880c38e0 2023-11-28T17:41:09.459 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.458+0000 7f138cb6a640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1388151ce0 0x7f13880c2910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42424/0 (socket says 172.21.15.98:42424) 2023-11-28T17:41:09.459 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.458+0000 7f138cb6a640 1 -- 172.21.15.98:0/1331528364 learned_addr learned my addr 172.21.15.98:0/1331528364 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:09.461 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.459+0000 7f138d36b640 1 --2- 172.21.15.98:0/1331528364 >> v2:172.21.15.113:3300/0 conn(0x7f13880c38e0 0x7f13880c3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:09.461 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.459+0000 7f13877fe640 1 -- 172.21.15.98:0/1331528364 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937493400 0 0) 0x7f137c001110 con 0x7f1388151ce0 2023-11-28T17:41:09.461 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.459+0000 7f13877fe640 1 -- 172.21.15.98:0/1331528364 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1378003520 con 0x7f1388151ce0 2023-11-28T17:41:09.462 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.459+0000 7f13877fe640 1 -- 172.21.15.98:0/1331528364 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 858626188 0 0) 0x7f1378003520 con 0x7f1388151ce0 2023-11-28T17:41:09.462 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.459+0000 7f13877fe640 1 -- 172.21.15.98:0/1331528364 >> v2:172.21.15.113:3300/0 conn(0x7f13880c38e0 msgr2=0x7f13880c3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:09.462 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.459+0000 7f13877fe640 1 --2- 172.21.15.98:0/1331528364 >> v2:172.21.15.113:3300/0 conn(0x7f13880c38e0 0x7f13880c3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.462 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.459+0000 7f13877fe640 1 -- 172.21.15.98:0/1331528364 >> v1:172.21.15.113:6789/0 conn(0x7f1388158720 legacy=0x7f13880c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:09.462 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.459+0000 7f13877fe640 1 -- 172.21.15.98:0/1331528364 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138815b390 con 0x7f1388151ce0 2023-11-28T17:41:09.462 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.459+0000 7f13877fe640 1 -- 172.21.15.98:0/1331528364 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f137c001800 con 0x7f1388151ce0 2023-11-28T17:41:09.462 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.459+0000 7f13877fe640 1 -- 172.21.15.98:0/1331528364 >> v1:172.21.15.98:6789/0 conn(0x7f1388151ce0 legacy=0x7f13880c2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:09.462 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.460+0000 7f13877fe640 1 --2- 172.21.15.98:0/1331528364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1378003970 0x7f1378003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:09.462 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.460+0000 7f13877fe640 1 --2- 172.21.15.98:0/1331528364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13780047c0 0x7f1378004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:09.462 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.460+0000 7f13877fe640 1 --2- 172.21.15.98:0/1331528364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1378005180 0x7f137800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:09.462 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.460+0000 7f13877fe640 1 -- 172.21.15.98:0/1331528364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13880c3020 con 0x7f1378003970 2023-11-28T17:41:09.462 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.460+0000 7f13877fe640 1 -- 172.21.15.98:0/1331528364 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137800c030 con 0x7f1378005180 2023-11-28T17:41:09.463 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.460+0000 7f13877fe640 1 -- 172.21.15.98:0/1331528364 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137800c1f0 con 0x7f13780047c0 2023-11-28T17:41:09.463 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.460+0000 7f138cb6a640 1 --2- 172.21.15.98:0/1331528364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1378003970 0x7f1378003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:09.463 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.460+0000 7f138cb6a640 1 -- 172.21.15.98:0/1331528364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13780047c0 msgr2=0x7f1378004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:09.463 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.460+0000 7f138cb6a640 1 --2- 172.21.15.98:0/1331528364 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13780047c0 0x7f1378004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.463 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.460+0000 7f138cb6a640 1 -- 172.21.15.98:0/1331528364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1378005180 msgr2=0x7f137800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:09.463 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.460+0000 7f138cb6a640 1 --2- 172.21.15.98:0/1331528364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1378005180 0x7f137800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.463 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.460+0000 7f138cb6a640 1 -- 172.21.15.98:0/1331528364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1378003520 con 0x7f1378003970 2023-11-28T17:41:09.463 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.460+0000 7f138d36b640 1 -- 172.21.15.98:0/1331528364 reap_dead start 2023-11-28T17:41:09.464 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.461+0000 7f138cb6a640 1 --2- 172.21.15.98:0/1331528364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1378003970 0x7f1378003d90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f137c00c190 tx=0x7f13780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd47be65ad32461 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:09.464 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.461+0000 7f13877fe640 1 -- 172.21.15.98:0/1331528364 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c005ad0 con 0x7f1378003970 2023-11-28T17:41:09.464 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.461+0000 7f13877fe640 1 -- 172.21.15.98:0/1331528364 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137c005c50 con 0x7f1378003970 2023-11-28T17:41:09.465 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.461+0000 7f13877fe640 1 -- 172.21.15.98:0/1331528364 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c02c7e0 con 0x7f1378003970 2023-11-28T17:41:09.465 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.462+0000 7f138edce640 1 -- 172.21.15.98:0/1331528364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1378003970 msgr2=0x7f1378003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:09.465 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.462+0000 7f138edce640 1 --2- 172.21.15.98:0/1331528364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1378003970 0x7f1378003d90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f137c00c190 tx=0x7f13780057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.465 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.462+0000 7f138edce640 1 -- 172.21.15.98:0/1331528364 shutdown_connections 2023-11-28T17:41:09.465 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.462+0000 7f138edce640 1 --2- 172.21.15.98:0/1331528364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1378003970 0x7f1378003d90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.465 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.462+0000 7f138edce640 1 -- 172.21.15.98:0/1331528364 >> 172.21.15.98:0/1331528364 conn(0x7f138806dfa0 msgr2=0x7f138806e380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:09.465 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.462+0000 7f138edce640 1 -- 172.21.15.98:0/1331528364 shutdown_connections 2023-11-28T17:41:09.465 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.462+0000 7f138edce640 1 -- 172.21.15.98:0/1331528364 wait complete. 2023-11-28T17:41:09.465 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.464+0000 7f138edce640 1 Processor -- start 2023-11-28T17:41:09.465 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.464+0000 7f138edce640 1 -- start start 2023-11-28T17:41:09.467 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.465+0000 7f138edce640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1378003970 0x7f13881f3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:09.467 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.465+0000 7f138edce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13880ca050 0x7f13881f0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:09.467 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.465+0000 7f138edce640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1388071ca0 0x7f1388156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:09.467 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.465+0000 7f138edce640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13880ca4b0 con 0x7f13880ca050 2023-11-28T17:41:09.467 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.465+0000 7f138edce640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13881f3970 con 0x7f1378003970 2023-11-28T17:41:09.467 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.465+0000 7f138edce640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13880716e0 con 0x7f1388071ca0 2023-11-28T17:41:09.467 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.465+0000 7f1387fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13880ca050 0x7f13881f0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:09.468 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.465+0000 7f1387fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13880ca050 0x7f13881f0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39142/0 (socket says 172.21.15.98:39142) 2023-11-28T17:41:09.468 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.465+0000 7f1387fff640 1 -- 172.21.15.98:0/4205661077 learned_addr learned my addr 172.21.15.98:0/4205661077 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:09.468 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.465+0000 7f138cb6a640 1 --2- 172.21.15.98:0/4205661077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1378003970 0x7f13881f3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:09.468 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.465+0000 7f138d36b640 1 --2- 172.21.15.98:0/4205661077 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1388071ca0 0x7f1388156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:09.468 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.466+0000 7f1387fff640 1 -- 172.21.15.98:0/4205661077 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1388071ca0 msgr2=0x7f1388156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:09.468 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.466+0000 7f1387fff640 1 --2- 172.21.15.98:0/4205661077 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1388071ca0 0x7f1388156660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.468 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.466+0000 7f1387fff640 1 -- 172.21.15.98:0/4205661077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1378003970 msgr2=0x7f13881f3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:09.468 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.466+0000 7f1387fff640 1 --2- 172.21.15.98:0/4205661077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1378003970 0x7f13881f3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.468 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.466+0000 7f1387fff640 1 -- 172.21.15.98:0/4205661077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13880719c0 con 0x7f13880ca050 2023-11-28T17:41:09.468 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.466+0000 7f1387fff640 1 --2- 172.21.15.98:0/4205661077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13880ca050 0x7f13881f0740 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f137400c500 tx=0x7f137400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:09.468 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.466+0000 7f13857fa640 1 -- 172.21.15.98:0/4205661077 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13740052d0 con 0x7f13880ca050 2023-11-28T17:41:09.468 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.466+0000 7f13857fa640 1 -- 172.21.15.98:0/4205661077 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1374005430 con 0x7f13880ca050 2023-11-28T17:41:09.469 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.466+0000 7f13857fa640 1 -- 172.21.15.98:0/4205661077 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1374012870 con 0x7f13880ca050 2023-11-28T17:41:09.469 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.466+0000 7f138edce640 1 -- 172.21.15.98:0/4205661077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13881f3320 con 0x7f13880ca050 2023-11-28T17:41:09.469 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.466+0000 7f138edce640 1 -- 172.21.15.98:0/4205661077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1388156ba0 con 0x7f13880ca050 2023-11-28T17:41:09.469 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.467+0000 7f138edce640 1 -- 172.21.15.98:0/4205661077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13880c4ba0 con 0x7f13880ca050 2023-11-28T17:41:09.470 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.468+0000 7f13857fa640 1 -- 172.21.15.98:0/4205661077 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) v1 ==== 89493+0+0 (secure 0 0 0) 0x7f13740129d0 con 0x7f13880ca050 2023-11-28T17:41:09.470 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.469+0000 7f13857fa640 1 --2- 172.21.15.98:0/4205661077 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f136c06b240 0x7f136c06d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:09.470 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.469+0000 7f13857fa640 1 -- 172.21.15.98:0/4205661077 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2511+0+0 (secure 0 0 0) 0x7f1374088a50 con 0x7f13880ca050 2023-11-28T17:41:09.470 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.469+0000 7f138cb6a640 1 --2- 172.21.15.98:0/4205661077 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f136c06b240 0x7f136c06d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:09.470 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.470+0000 7f138cb6a640 1 --2- 172.21.15.98:0/4205661077 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f136c06b240 0x7f136c06d700 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f137c034000 tx=0x7f137c009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:09.474 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.474+0000 7f13857fa640 1 -- 172.21.15.98:0/4205661077 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1374054610 con 0x7f13880ca050 2023-11-28T17:41:09.666 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.665+0000 7f138edce640 1 -- 172.21.15.98:0/4205661077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f138815bed0 con 0x7f13880ca050 2023-11-28T17:41:09.667 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.667+0000 7f13857fa640 1 -- 172.21.15.98:0/4205661077 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+10943 (secure 0 0 0) 0x7f1374057c30 con 0x7f13880ca050 2023-11-28T17:41:09.668 INFO:teuthology.misc.health.smithi098.stdout: 2023-11-28T17:41:09.668 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":15,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:41:09.556993+0000","last_up_change":"2023-11-28T17:41:09.556993+0000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6810","nonce":2188704385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6813","nonce":2188704385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6819","nonce":2188704385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6816","nonce":2188704385}]},"public_addr":"172.21.15.98:6810/2188704385","cluster_addr":"172.21.15.98:6813/2188704385","heartbeat_back_addr":"172.21.15.98:6819/2188704385","heartbeat_front_addr":"172.21.15.98:6816/2188704385","state":["exists","up"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6801","nonce":3918226860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6803","nonce":3918226860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6807","nonce":3918226860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6805","nonce":3918226860}]},"public_addr":"172.21.15.98:6801/3918226860","cluster_addr":"172.21.15.98:6803/3918226860","heartbeat_back_addr":"172.21.15.98:6807/3918226860","heartbeat_front_addr":"172.21.15.98:6805/3918226860","state":["exists","up"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6812","nonce":2691532400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6815","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6818","nonce":2691532400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6823","nonce":2691532400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6821","nonce":2691532400}]},"public_addr":"172.21.15.98:6812/2691532400","cluster_addr":"172.21.15.98:6818/2691532400","heartbeat_back_addr":"172.21.15.98:6823/2691532400","heartbeat_front_addr":"172.21.15.98:6821/2691532400","state":["exists","up"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6825","nonce":1905487008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6827","nonce":1905487008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6831","nonce":1905487008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6829","nonce":1905487008}]},"public_addr":"172.21.15.98:6825/1905487008","cluster_addr":"172.21.15.98:6827/1905487008","heartbeat_back_addr":"172.21.15.98:6831/1905487008","heartbeat_front_addr":"172.21.15.98:6829/1905487008","state":["exists","up"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6801","nonce":2974677215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6803","nonce":2974677215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6807","nonce":2974677215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6805","nonce":2974677215}]},"public_addr":"172.21.15.113:6801/2974677215","cluster_addr":"172.21.15.113:6803/2974677215","heartbeat_back_addr":"172.21.15.113:6807/2974677215","heartbeat_front_addr":"172.21.15.113:6805/2974677215","state":["exists","up"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6817","nonce":2685000355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6821","nonce":2685000355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6829","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6831","nonce":2685000355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6823","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6826","nonce":2685000355}]},"public_addr":"172.21.15.113:6817/2685000355","cluster_addr":"172.21.15.113:6821/2685000355","heartbeat_back_addr":"172.21.15.113:6831/2685000355","heartbeat_front_addr":"172.21.15.113:6826/2685000355","state":["exists","up"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6809","nonce":2764300831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6811","nonce":2764300831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6815","nonce":2764300831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6813","nonce":2764300831}]},"public_addr":"172.21.15.113:6809/2764300831","cluster_addr":"172.21.15.113:6811/2764300831","heartbeat_back_addr":"172.21.15.113:6815/2764300831","heartbeat_front_addr":"172.21.15.113:6813/2764300831","state":["exists","up"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6819","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6820","nonce":1656248400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6824","nonce":1656248400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6830","nonce":1656248400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6825","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6827","nonce":1656248400}]},"public_addr":"172.21.15.113:6820/1656248400","cluster_addr":"172.21.15.113:6824/1656248400","heartbeat_back_addr":"172.21.15.113:6830/1656248400","heartbeat_front_addr":"172.21.15.113:6827/1656248400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.117494+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:41:09.671 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.669+0000 7f138edce640 1 -- 172.21.15.98:0/4205661077 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f136c06b240 msgr2=0x7f136c06d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:09.671 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.669+0000 7f138edce640 1 --2- 172.21.15.98:0/4205661077 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f136c06b240 0x7f136c06d700 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f137c034000 tx=0x7f137c009030 comp rx=0 tx=0).stop 2023-11-28T17:41:09.671 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.670+0000 7f138edce640 1 -- 172.21.15.98:0/4205661077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13880ca050 msgr2=0x7f13881f0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:09.671 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.670+0000 7f138edce640 1 --2- 172.21.15.98:0/4205661077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13880ca050 0x7f13881f0740 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f137400c500 tx=0x7f137400c9c0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.671 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.670+0000 7f138edce640 1 -- 172.21.15.98:0/4205661077 shutdown_connections 2023-11-28T17:41:09.671 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.670+0000 7f138edce640 1 --2- 172.21.15.98:0/4205661077 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1388071ca0 0x7f1388156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.671 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.670+0000 7f138edce640 1 --2- 172.21.15.98:0/4205661077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13880ca050 0x7f13881f0740 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.671 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.670+0000 7f138edce640 1 --2- 172.21.15.98:0/4205661077 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f136c06b240 0x7f136c06d700 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.671 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.670+0000 7f138edce640 1 --2- 172.21.15.98:0/4205661077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1378003970 0x7f13881f3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.671 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.670+0000 7f138edce640 1 -- 172.21.15.98:0/4205661077 >> 172.21.15.98:0/4205661077 conn(0x7f138806dfa0 msgr2=0x7f138814f500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:09.672 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.670+0000 7f138edce640 1 -- 172.21.15.98:0/4205661077 shutdown_connections 2023-11-28T17:41:09.672 INFO:teuthology.misc.health.smithi098.stderr:2023-11-28T17:41:09.670+0000 7f138edce640 1 -- 172.21.15.98:0/4205661077 wait complete. 2023-11-28T17:41:09.684 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-11-28T17:41:09.684 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-28T17:41:09.685 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T17:41:09.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T17:41:09.693 INFO:tasks.ceph.mds.a:Started 2023-11-28T17:41:09.693 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T17:41:09.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T17:41:09.701 INFO:tasks.ceph.mds.c:Started 2023-11-28T17:41:09.701 INFO:tasks.ceph.mds.e:Restarting daemon 2023-11-28T17:41:09.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-11-28T17:41:09.706 INFO:tasks.ceph.mds.e:Started 2023-11-28T17:41:09.706 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T17:41:09.707 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T17:41:09.715 INFO:tasks.ceph.mds.b:Started 2023-11-28T17:41:09.716 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T17:41:09.716 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T17:41:09.724 INFO:tasks.ceph.mds.d:Started 2023-11-28T17:41:09.725 INFO:tasks.ceph.mds.f:Restarting daemon 2023-11-28T17:41:09.725 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-11-28T17:41:09.732 INFO:tasks.ceph.mds.f:Started 2023-11-28T17:41:09.732 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-11-28T17:41:09.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:41:09.842 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2023-11-28T17:41:09.844 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-11-28T17:41:09.853 INFO:tasks.ceph.mds.e.smithi098.stdout:starting mds.e at 2023-11-28T17:41:09.880 INFO:tasks.ceph.mds.b.smithi113.stdout:starting mds.b at 2023-11-28T17:41:09.880 INFO:tasks.ceph.mds.d.smithi113.stdout:starting mds.d at 2023-11-28T17:41:09.884 INFO:tasks.ceph.mds.f.smithi113.stdout:starting mds.f at 2023-11-28T17:41:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.916+0000 7fa9e71db640 1 Processor -- start 2023-11-28T17:41:09.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.917+0000 7fa9e71db640 1 -- start start 2023-11-28T17:41:09.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.918+0000 7fa9e71db640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9e0153bb0 0x7fa9e015a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:09.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.918+0000 7fa9e71db640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9e00ca7d0 0x7fa9e00b3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:09.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.918+0000 7fa9e71db640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e015d330 con 0x7fa9e00ca390 2023-11-28T17:41:09.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.918+0000 7fa9e71db640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e015d120 con 0x7fa9e00ca7d0 2023-11-28T17:41:09.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.918+0000 7fa9e71db640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00cac60 con 0x7fa9e0153bb0 2023-11-28T17:41:09.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.918+0000 7fa9e69da640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9e00ca7d0 0x7fa9e00b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:09.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.918+0000 7fa9e69da640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9e00ca7d0 0x7fa9e00b3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39182/0 (socket says 172.21.15.98:39182) 2023-11-28T17:41:09.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.918+0000 7fa9e69da640 1 -- 172.21.15.98:0/586917828 learned_addr learned my addr 172.21.15.98:0/586917828 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:09.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.918+0000 7fa9e61d9640 1 --2- 172.21.15.98:0/586917828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9e0153bb0 0x7fa9e015a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:09.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.919+0000 7fa9e69da640 1 -- 172.21.15.98:0/586917828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9e0153bb0 msgr2=0x7fa9e015a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:09.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.919+0000 7fa9e69da640 1 --2- 172.21.15.98:0/586917828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9e0153bb0 0x7fa9e015a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.919+0000 7fa9e69da640 1 -- 172.21.15.98:0/586917828 >> v1:172.21.15.113:6789/0 conn(0x7fa9e00ca390 legacy=0x7fa9e00cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:09.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.919+0000 7fa9e69da640 1 -- 172.21.15.98:0/586917828 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e015d610 con 0x7fa9e00ca7d0 2023-11-28T17:41:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.920+0000 7fa9e69da640 1 --2- 172.21.15.98:0/586917828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e00ca7d0 0x7fa9e00b3760 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009e50 tx=0x7fa9d802bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8924d0987b64adcd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.920+0000 7fa9e51d7640 1 -- 172.21.15.98:0/586917828 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d802cb90 con 0x7fa9e00ca7d0 2023-11-28T17:41:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.920+0000 7fa9e51d7640 1 -- 172.21.15.98:0/586917828 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d802ccf0 con 0x7fa9e00ca7d0 2023-11-28T17:41:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.920+0000 7fa9e51d7640 1 -- 172.21.15.98:0/586917828 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8034610 con 0x7fa9e00ca7d0 2023-11-28T17:41:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.920+0000 7fa9e71db640 1 -- 172.21.15.98:0/586917828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e00ca7d0 msgr2=0x7fa9e00b3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.920+0000 7fa9e71db640 1 --2- 172.21.15.98:0/586917828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e00ca7d0 0x7fa9e00b3760 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009e50 tx=0x7fa9d802bff0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.920+0000 7fa9e71db640 1 -- 172.21.15.98:0/586917828 shutdown_connections 2023-11-28T17:41:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.920+0000 7fa9e71db640 1 --2- 172.21.15.98:0/586917828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e00ca7d0 0x7fa9e00b3760 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.920+0000 7fa9e71db640 1 --2- 172.21.15.98:0/586917828 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9e0153bb0 0x7fa9e015a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.920+0000 7fa9e71db640 1 -- 172.21.15.98:0/586917828 >> 172.21.15.98:0/586917828 conn(0x7fa9e0071c00 msgr2=0x7fa9e0072010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.921+0000 7fa9e71db640 1 -- 172.21.15.98:0/586917828 shutdown_connections 2023-11-28T17:41:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.921+0000 7fa9e71db640 1 -- 172.21.15.98:0/586917828 wait complete. 2023-11-28T17:41:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.922+0000 7fa9e71db640 1 Processor -- start 2023-11-28T17:41:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.922+0000 7fa9e71db640 1 -- start start 2023-11-28T17:41:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.923+0000 7fa9e71db640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e00ca390 0x7fa9e01594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.923+0000 7fa9e71db640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9e00ca7d0 0x7fa9e015c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.923+0000 7fa9e71db640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9e0153bb0 0x7fa9e01f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.923+0000 7fa9e71db640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e0159a20 con 0x7fa9e00ca390 2023-11-28T17:41:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.923+0000 7fa9e71db640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e014c170 con 0x7fa9e0153bb0 2023-11-28T17:41:09.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.923+0000 7fa9e71db640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e0159f20 con 0x7fa9e00ca7d0 2023-11-28T17:41:09.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.923+0000 7fa9e61d9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e00ca390 0x7fa9e01594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:09.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.923+0000 7fa9e59d8640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9e00ca7d0 0x7fa9e015c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:09.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.923+0000 7fa9e61d9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e00ca390 0x7fa9e01594e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39190/0 (socket says 172.21.15.98:39190) 2023-11-28T17:41:09.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.923+0000 7fa9e61d9640 1 -- 172.21.15.98:0/4282763501 learned_addr learned my addr 172.21.15.98:0/4282763501 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:09.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.923+0000 7fa9e69da640 1 --2- 172.21.15.98:0/4282763501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9e0153bb0 0x7fa9e01f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:09.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.924+0000 7fa9e61d9640 1 -- 172.21.15.98:0/4282763501 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9e00ca7d0 msgr2=0x7fa9e015c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:09.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.924+0000 7fa9e61d9640 1 --2- 172.21.15.98:0/4282763501 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9e00ca7d0 0x7fa9e015c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.924+0000 7fa9e61d9640 1 -- 172.21.15.98:0/4282763501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9e0153bb0 msgr2=0x7fa9e01f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.924+0000 7fa9e61d9640 1 --2- 172.21.15.98:0/4282763501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9e0153bb0 0x7fa9e01f2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.924+0000 7fa9e61d9640 1 -- 172.21.15.98:0/4282763501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d8009b30 con 0x7fa9e00ca390 2023-11-28T17:41:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.924+0000 7fa9e61d9640 1 --2- 172.21.15.98:0/4282763501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e00ca390 0x7fa9e01594e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00e870 tx=0x7fa9dc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.924+0000 7fa9d6ffd640 1 -- 172.21.15.98:0/4282763501 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc0094f0 con 0x7fa9e00ca390 2023-11-28T17:41:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.924+0000 7fa9d6ffd640 1 -- 172.21.15.98:0/4282763501 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9dc00c940 con 0x7fa9e00ca390 2023-11-28T17:41:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.924+0000 7fa9e71db640 1 -- 172.21.15.98:0/4282763501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e0159230 con 0x7fa9e00ca390 2023-11-28T17:41:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.924+0000 7fa9d6ffd640 1 -- 172.21.15.98:0/4282763501 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc00ccf0 con 0x7fa9e00ca390 2023-11-28T17:41:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.924+0000 7fa9e71db640 1 -- 172.21.15.98:0/4282763501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e0158470 con 0x7fa9e00ca390 2023-11-28T17:41:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.925+0000 7fa9e71db640 1 -- 172.21.15.98:0/4282763501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e0074db0 con 0x7fa9e00ca390 2023-11-28T17:41:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.926+0000 7fa9d6ffd640 1 -- 172.21.15.98:0/4282763501 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) v1 ==== 89493+0+0 (secure 0 0 0) 0x7fa9dc013470 con 0x7fa9e00ca390 2023-11-28T17:41:09.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.927+0000 7fa9d6ffd640 1 --2- 172.21.15.98:0/4282763501 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa9c406b210 0x7fa9c406d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:09.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.927+0000 7fa9d6ffd640 1 -- 172.21.15.98:0/4282763501 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fa9dc0899e0 con 0x7fa9e00ca390 2023-11-28T17:41:09.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.928+0000 7fa9e59d8640 1 --2- 172.21.15.98:0/4282763501 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa9c406b210 0x7fa9c406d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:09.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.928+0000 7fa9e59d8640 1 --2- 172.21.15.98:0/4282763501 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa9c406b210 0x7fa9c406d6d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0157af0 tx=0x7fa9d0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:09.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:09.931+0000 7fa9d6ffd640 1 -- 172.21.15.98:0/4282763501 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9dc054e10 con 0x7fa9e00ca390 2023-11-28T17:41:10.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.121+0000 7fa9e71db640 1 -- 172.21.15.98:0/4282763501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9e0075d90 con 0x7fa9e00ca390 2023-11-28T17:41:10.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.124+0000 7fa9d6ffd640 1 -- 172.21.15.98:0/4282763501 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+10943 (secure 0 0 0) 0x7fa9dc017090 con 0x7fa9e00ca390 2023-11-28T17:41:10.125 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:10.125 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":15,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:41:09.556993+0000","last_up_change":"2023-11-28T17:41:09.556993+0000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6810","nonce":2188704385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6813","nonce":2188704385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6819","nonce":2188704385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6816","nonce":2188704385}]},"public_addr":"172.21.15.98:6810/2188704385","cluster_addr":"172.21.15.98:6813/2188704385","heartbeat_back_addr":"172.21.15.98:6819/2188704385","heartbeat_front_addr":"172.21.15.98:6816/2188704385","state":["exists","up"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6801","nonce":3918226860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6803","nonce":3918226860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6807","nonce":3918226860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6805","nonce":3918226860}]},"public_addr":"172.21.15.98:6801/3918226860","cluster_addr":"172.21.15.98:6803/3918226860","heartbeat_back_addr":"172.21.15.98:6807/3918226860","heartbeat_front_addr":"172.21.15.98:6805/3918226860","state":["exists","up"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6812","nonce":2691532400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6815","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6818","nonce":2691532400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6823","nonce":2691532400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6821","nonce":2691532400}]},"public_addr":"172.21.15.98:6812/2691532400","cluster_addr":"172.21.15.98:6818/2691532400","heartbeat_back_addr":"172.21.15.98:6823/2691532400","heartbeat_front_addr":"172.21.15.98:6821/2691532400","state":["exists","up"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6825","nonce":1905487008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6827","nonce":1905487008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6831","nonce":1905487008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6829","nonce":1905487008}]},"public_addr":"172.21.15.98:6825/1905487008","cluster_addr":"172.21.15.98:6827/1905487008","heartbeat_back_addr":"172.21.15.98:6831/1905487008","heartbeat_front_addr":"172.21.15.98:6829/1905487008","state":["exists","up"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6801","nonce":2974677215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6803","nonce":2974677215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6807","nonce":2974677215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6805","nonce":2974677215}]},"public_addr":"172.21.15.113:6801/2974677215","cluster_addr":"172.21.15.113:6803/2974677215","heartbeat_back_addr":"172.21.15.113:6807/2974677215","heartbeat_front_addr":"172.21.15.113:6805/2974677215","state":["exists","up"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6817","nonce":2685000355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6821","nonce":2685000355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6829","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6831","nonce":2685000355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6823","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6826","nonce":2685000355}]},"public_addr":"172.21.15.113:6817/2685000355","cluster_addr":"172.21.15.113:6821/2685000355","heartbeat_back_addr":"172.21.15.113:6831/2685000355","heartbeat_front_addr":"172.21.15.113:6826/2685000355","state":["exists","up"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6809","nonce":2764300831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6811","nonce":2764300831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6815","nonce":2764300831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6813","nonce":2764300831}]},"public_addr":"172.21.15.113:6809/2764300831","cluster_addr":"172.21.15.113:6811/2764300831","heartbeat_back_addr":"172.21.15.113:6815/2764300831","heartbeat_front_addr":"172.21.15.113:6813/2764300831","state":["exists","up"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6819","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6820","nonce":1656248400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6824","nonce":1656248400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6830","nonce":1656248400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6825","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6827","nonce":1656248400}]},"public_addr":"172.21.15.113:6820/1656248400","cluster_addr":"172.21.15.113:6824/1656248400","heartbeat_back_addr":"172.21.15.113:6830/1656248400","heartbeat_front_addr":"172.21.15.113:6827/1656248400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.117494+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:41:10.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.127+0000 7fa9e71db640 1 -- 172.21.15.98:0/4282763501 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa9c406b210 msgr2=0x7fa9c406d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:10.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.127+0000 7fa9e71db640 1 --2- 172.21.15.98:0/4282763501 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa9c406b210 0x7fa9c406d6d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0157af0 tx=0x7fa9d0009040 comp rx=0 tx=0).stop 2023-11-28T17:41:10.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.127+0000 7fa9e71db640 1 -- 172.21.15.98:0/4282763501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e00ca390 msgr2=0x7fa9e01594e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:10.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.127+0000 7fa9e71db640 1 --2- 172.21.15.98:0/4282763501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e00ca390 0x7fa9e01594e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00e870 tx=0x7fa9dc00ed30 comp rx=0 tx=0).stop 2023-11-28T17:41:10.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.127+0000 7fa9e71db640 1 -- 172.21.15.98:0/4282763501 shutdown_connections 2023-11-28T17:41:10.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.127+0000 7fa9e71db640 1 --2- 172.21.15.98:0/4282763501 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa9c406b210 0x7fa9c406d6d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:10.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.127+0000 7fa9e71db640 1 --2- 172.21.15.98:0/4282763501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9e0153bb0 0x7fa9e01f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:10.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.127+0000 7fa9e71db640 1 --2- 172.21.15.98:0/4282763501 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa9e00ca7d0 0x7fa9e015c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:10.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.127+0000 7fa9e71db640 1 --2- 172.21.15.98:0/4282763501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e00ca390 0x7fa9e01594e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:10.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.127+0000 7fa9e71db640 1 -- 172.21.15.98:0/4282763501 >> 172.21.15.98:0/4282763501 conn(0x7fa9e0071c00 msgr2=0x7fa9e0152e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:10.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.127+0000 7fa9e71db640 1 -- 172.21.15.98:0/4282763501 shutdown_connections 2023-11-28T17:41:10.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.127+0000 7fa9e71db640 1 -- 172.21.15.98:0/4282763501 wait complete. 2023-11-28T17:41:10.144 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2023-11-28T17:41:10.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:41:10.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.354+0000 7f9b0135a640 1 Processor -- start 2023-11-28T17:41:10.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.354+0000 7f9b0135a640 1 -- start start 2023-11-28T17:41:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.356+0000 7f9b0135a640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9afc0c5c00 0x7f9afc0c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.356+0000 7f9b0135a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9afc157850 con 0x7f9afc150c00 2023-11-28T17:41:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.356+0000 7f9b0135a640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9afc156980 con 0x7f9afc0c4a20 2023-11-28T17:41:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.356+0000 7f9b0135a640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc0c2e10 con 0x7f9afc0c5c00 2023-11-28T17:41:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.356+0000 7f9afaffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9afc150c00 0x7f9afc0c2700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48134/0 (socket says 172.21.15.98:48134) 2023-11-28T17:41:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.356+0000 7f9afaffd640 1 -- 172.21.15.98:0/127573862 learned_addr learned my addr 172.21.15.98:0/127573862 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.356+0000 7f9afa7fc640 1 -- 172.21.15.98:0/127573862 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334871381 0 0) 0x7f9afc157850 con 0x7f9afc150c00 2023-11-28T17:41:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9afa7fc640 1 -- 172.21.15.98:0/127573862 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ae8003520 con 0x7f9afc150c00 2023-11-28T17:41:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9af27fc640 1 --2- 172.21.15.98:0/127573862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9afc0c5c00 0x7f9afc0c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9afa7fc640 1 -- 172.21.15.98:0/127573862 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3811427953 0 0) 0x7f9ae8003520 con 0x7f9afc150c00 2023-11-28T17:41:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9afa7fc640 1 -- 172.21.15.98:0/127573862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9afc0c5c00 msgr2=0x7f9afc0c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9afa7fc640 1 --2- 172.21.15.98:0/127573862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9afc0c5c00 0x7f9afc0c44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9afa7fc640 1 -- 172.21.15.98:0/127573862 >> v1:172.21.15.113:6789/0 conn(0x7f9afc0c4a20 legacy=0x7f9afc0c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9afa7fc640 1 -- 172.21.15.98:0/127573862 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc156c40 con 0x7f9afc150c00 2023-11-28T17:41:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9afa7fc640 1 -- 172.21.15.98:0/127573862 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f9aec001670 con 0x7f9afc150c00 2023-11-28T17:41:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9afa7fc640 1 -- 172.21.15.98:0/127573862 >> v1:172.21.15.98:6789/0 conn(0x7f9afc150c00 legacy=0x7f9afc0c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9afa7fc640 1 --2- 172.21.15.98:0/127573862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9afc0c5c00 0x7f9ae80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9afa7fc640 1 --2- 172.21.15.98:0/127573862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8003f70 0x7f9ae80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9afa7fc640 1 --2- 172.21.15.98:0/127573862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ae8004d00 0x7f9ae8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9afa7fc640 1 -- 172.21.15.98:0/127573862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae800ba90 con 0x7f9ae8003f70 2023-11-28T17:41:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9afa7fc640 1 -- 172.21.15.98:0/127573862 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae800bc00 con 0x7f9ae8004d00 2023-11-28T17:41:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9afa7fc640 1 -- 172.21.15.98:0/127573862 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae800bdc0 con 0x7f9afc0c5c00 2023-11-28T17:41:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.357+0000 7f9af27fc640 1 --2- 172.21.15.98:0/127573862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8003f70 0x7f9ae80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.358+0000 7f9afaffd640 1 --2- 172.21.15.98:0/127573862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9afc0c5c00 0x7f9ae80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.358+0000 7f9af27fc640 1 -- 172.21.15.98:0/127573862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9afc0c5c00 msgr2=0x7f9ae80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.358+0000 7f9af27fc640 1 --2- 172.21.15.98:0/127573862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9afc0c5c00 0x7f9ae80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.358+0000 7f9af27fc640 1 -- 172.21.15.98:0/127573862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ae8004d00 msgr2=0x7f9ae8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.358+0000 7f9af27fc640 1 --2- 172.21.15.98:0/127573862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ae8004d00 0x7f9ae8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.358+0000 7f9af27fc640 1 -- 172.21.15.98:0/127573862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae8003520 con 0x7f9ae8003f70 2023-11-28T17:41:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.358+0000 7f9afaffd640 1 --2- 172.21.15.98:0/127573862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9afc0c5c00 0x7f9ae80039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.359+0000 7f9af27fc640 1 --2- 172.21.15.98:0/127573862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8003f70 0x7f9ae80047c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4009ac0 tx=0x7f9ae80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10611f4fef194f94 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.359+0000 7f9afa7fc640 1 -- 172.21.15.98:0/127573862 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae402c600 con 0x7f9ae8003f70 2023-11-28T17:41:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.359+0000 7f9afa7fc640 1 -- 172.21.15.98:0/127573862 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae402c760 con 0x7f9ae8003f70 2023-11-28T17:41:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.359+0000 7f9afa7fc640 1 -- 172.21.15.98:0/127573862 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae402caf0 con 0x7f9ae8003f70 2023-11-28T17:41:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.359+0000 7f9b0135a640 1 -- 172.21.15.98:0/127573862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8003f70 msgr2=0x7f9ae80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.359+0000 7f9b0135a640 1 --2- 172.21.15.98:0/127573862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ae8003f70 0x7f9ae80047c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4009ac0 tx=0x7f9ae80057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.359+0000 7f9afb7fe640 1 -- 172.21.15.98:0/127573862 reap_dead start 2023-11-28T17:41:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.360+0000 7f9b0135a640 1 -- 172.21.15.98:0/127573862 shutdown_connections 2023-11-28T17:41:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.360+0000 7f9b0135a640 1 -- 172.21.15.98:0/127573862 >> 172.21.15.98:0/127573862 conn(0x7f9afc06a4d0 msgr2=0x7f9afc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.360+0000 7f9b0135a640 1 -- 172.21.15.98:0/127573862 shutdown_connections 2023-11-28T17:41:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.360+0000 7f9b0135a640 1 -- 172.21.15.98:0/127573862 wait complete. 2023-11-28T17:41:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.361+0000 7f9b0135a640 1 Processor -- start 2023-11-28T17:41:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.361+0000 7f9b0135a640 1 -- start start 2023-11-28T17:41:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.362+0000 7f9b0135a640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ae8003f70 0x7f9afc155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.362+0000 7f9b0135a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9afc1ee5d0 0x7f9afc071c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.362+0000 7f9b0135a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9afc143680 0x7f9afc143b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.362+0000 7f9b0135a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc154bd0 con 0x7f9afc143680 2023-11-28T17:41:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.362+0000 7f9b0135a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc156260 con 0x7f9afc1ee5d0 2023-11-28T17:41:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.362+0000 7f9b0135a640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc144040 con 0x7f9ae8003f70 2023-11-28T17:41:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.362+0000 7f9afb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9afc143680 0x7f9afc143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.362+0000 7f9afb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9afc143680 0x7f9afc143b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59050/0 (socket says 172.21.15.98:59050) 2023-11-28T17:41:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.362+0000 7f9afb7fe640 1 -- 172.21.15.98:0/3412274341 learned_addr learned my addr 172.21.15.98:0/3412274341 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.363+0000 7f9afb7fe640 1 -- 172.21.15.98:0/3412274341 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ae8003f70 msgr2=0x7f9afc155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.363+0000 7f9afb7fe640 1 --2- 172.21.15.98:0/3412274341 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ae8003f70 0x7f9afc155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.363+0000 7f9afb7fe640 1 -- 172.21.15.98:0/3412274341 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9afc1ee5d0 msgr2=0x7f9afc071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.363+0000 7f9afb7fe640 1 --2- 172.21.15.98:0/3412274341 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9afc1ee5d0 0x7f9afc071c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.363+0000 7f9afb7fe640 1 -- 172.21.15.98:0/3412274341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc156510 con 0x7f9afc143680 2023-11-28T17:41:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.363+0000 7f9afb7fe640 1 --2- 172.21.15.98:0/3412274341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9afc143680 0x7f9afc143b00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0cb1d0 tx=0x7f9af400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.363+0000 7f9af3fff640 1 -- 172.21.15.98:0/3412274341 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4005d40 con 0x7f9afc143680 2023-11-28T17:41:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.363+0000 7f9af3fff640 1 -- 172.21.15.98:0/3412274341 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af400ced0 con 0x7f9afc143680 2023-11-28T17:41:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.363+0000 7f9af3fff640 1 -- 172.21.15.98:0/3412274341 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af400a850 con 0x7f9afc143680 2023-11-28T17:41:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.363+0000 7f9b0135a640 1 -- 172.21.15.98:0/3412274341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9afc14a070 con 0x7f9afc143680 2023-11-28T17:41:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.363+0000 7f9b0135a640 1 -- 172.21.15.98:0/3412274341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9afc14a630 con 0x7f9afc143680 2023-11-28T17:41:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.364+0000 7f9b0135a640 1 -- 172.21.15.98:0/3412274341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9afc067bf0 con 0x7f9afc143680 2023-11-28T17:41:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.365+0000 7f9af3fff640 1 -- 172.21.15.98:0/3412274341 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) v1 ==== 89493+0+0 (secure 0 0 0) 0x7f9af400a9b0 con 0x7f9afc143680 2023-11-28T17:41:10.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.366+0000 7f9af3fff640 1 --2- 172.21.15.98:0/3412274341 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9adc06b210 0x7f9adc06d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:10.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.366+0000 7f9af3fff640 1 -- 172.21.15.98:0/3412274341 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f9af4088510 con 0x7f9afc143680 2023-11-28T17:41:10.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.367+0000 7f9afaffd640 1 --2- 172.21.15.98:0/3412274341 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9adc06b210 0x7f9adc06d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:10.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.367+0000 7f9afaffd640 1 --2- 172.21.15.98:0/3412274341 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9adc06b210 0x7f9adc06d6d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9aec007a50 tx=0x7f9aec009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:10.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.371+0000 7f9af3fff640 1 -- 172.21.15.98:0/3412274341 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9af4053950 con 0x7f9afc143680 2023-11-28T17:41:10.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.599+0000 7f9b0135a640 1 -- 172.21.15.98:0/3412274341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9afc0c8810 con 0x7f9afc143680 2023-11-28T17:41:10.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.600+0000 7f9af3fff640 1 -- 172.21.15.98:0/3412274341 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f9af4017540 con 0x7f9afc143680 2023-11-28T17:41:10.600 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:10.601 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2429440814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2429440814},{"type":"v1","addr":"172.21.15.98:6836","nonce":2429440814}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-11-28T17:41:10.601 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 2 2023-11-28T17:41:10.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.602+0000 7f9b0135a640 1 -- 172.21.15.98:0/3412274341 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9adc06b210 msgr2=0x7f9adc06d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:10.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.602+0000 7f9b0135a640 1 --2- 172.21.15.98:0/3412274341 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9adc06b210 0x7f9adc06d6d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9aec007a50 tx=0x7f9aec009040 comp rx=0 tx=0).stop 2023-11-28T17:41:10.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.602+0000 7f9b0135a640 1 -- 172.21.15.98:0/3412274341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9afc143680 msgr2=0x7f9afc143b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:10.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.602+0000 7f9b0135a640 1 --2- 172.21.15.98:0/3412274341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9afc143680 0x7f9afc143b00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0cb1d0 tx=0x7f9af400cb90 comp rx=0 tx=0).stop 2023-11-28T17:41:10.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.603+0000 7f9b0135a640 1 -- 172.21.15.98:0/3412274341 shutdown_connections 2023-11-28T17:41:10.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.603+0000 7f9b0135a640 1 --2- 172.21.15.98:0/3412274341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9afc143680 0x7f9afc143b00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:10.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.603+0000 7f9b0135a640 1 --2- 172.21.15.98:0/3412274341 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9adc06b210 0x7f9adc06d6d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:10.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.603+0000 7f9b0135a640 1 --2- 172.21.15.98:0/3412274341 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9afc1ee5d0 0x7f9afc071c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:10.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.603+0000 7f9b0135a640 1 --2- 172.21.15.98:0/3412274341 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ae8003f70 0x7f9afc155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:10.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.603+0000 7f9b0135a640 1 -- 172.21.15.98:0/3412274341 >> 172.21.15.98:0/3412274341 conn(0x7f9afc06a4d0 msgr2=0x7f9afc0c7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:10.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.603+0000 7f9b0135a640 1 -- 172.21.15.98:0/3412274341 shutdown_connections 2023-11-28T17:41:10.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:10.603+0000 7f9b0135a640 1 -- 172.21.15.98:0/3412274341 wait complete. 2023-11-28T17:41:12.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:41:12.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.830+0000 7fcb9c007640 1 Processor -- start 2023-11-28T17:41:12.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.830+0000 7fcb9c007640 1 -- start start 2023-11-28T17:41:12.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.831+0000 7fcb9c007640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcb940cf6d0 0x7fcb940c7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:12.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.831+0000 7fcb9c007640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb940cfab0 0x7fcb94159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.831+0000 7fcb9c007640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcb940cb780 0x7fcb9415a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.831+0000 7fcb9c007640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb9415aa10 con 0x7fcb940cf6d0 2023-11-28T17:41:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.831+0000 7fcb9c007640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb9415acf0 con 0x7fcb940cb780 2023-11-28T17:41:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.831+0000 7fcb9c007640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9415d430 con 0x7fcb940cfab0 2023-11-28T17:41:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.831+0000 7fcb99da3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcb940cf6d0 0x7fcb940c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.831+0000 7fcb99da3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcb940cf6d0 0x7fcb940c7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59088/0 (socket says 172.21.15.98:59088) 2023-11-28T17:41:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.832+0000 7fcb99da3640 1 -- 172.21.15.98:0/3191744735 learned_addr learned my addr 172.21.15.98:0/3191744735 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.832+0000 7fcb9a5a4640 1 --2- 172.21.15.98:0/3191744735 >> v2:172.21.15.113:3300/0 conn(0x7fcb940cb780 0x7fcb9415a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.832+0000 7fcb99da3640 1 -- 172.21.15.98:0/3191744735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb940cfab0 msgr2=0x7fcb94159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.832+0000 7fcb995a2640 1 --2- 172.21.15.98:0/3191744735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb940cfab0 0x7fcb94159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.832+0000 7fcb99da3640 1 --2- 172.21.15.98:0/3191744735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb940cfab0 0x7fcb94159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.832+0000 7fcb99da3640 1 -- 172.21.15.98:0/3191744735 >> v2:172.21.15.113:3300/0 conn(0x7fcb940cb780 msgr2=0x7fcb9415a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.832+0000 7fcb99da3640 1 --2- 172.21.15.98:0/3191744735 >> v2:172.21.15.113:3300/0 conn(0x7fcb940cb780 0x7fcb9415a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.832+0000 7fcb99da3640 1 -- 172.21.15.98:0/3191744735 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9415d710 con 0x7fcb940cf6d0 2023-11-28T17:41:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.832+0000 7fcb9a5a4640 1 --2- 172.21.15.98:0/3191744735 >> v2:172.21.15.113:3300/0 conn(0x7fcb940cb780 0x7fcb9415a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.832+0000 7fcb995a2640 1 --2- 172.21.15.98:0/3191744735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb940cfab0 0x7fcb94159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.833+0000 7fcb99da3640 1 --2- 172.21.15.98:0/3191744735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb940cf6d0 0x7fcb940c7600 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcb940c6380 tx=0x7fcb9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39cad78d00f03456 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.833+0000 7fcb98da1640 1 -- 172.21.15.98:0/3191744735 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90031870 con 0x7fcb940cf6d0 2023-11-28T17:41:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.833+0000 7fcb98da1640 1 -- 172.21.15.98:0/3191744735 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb900319d0 con 0x7fcb940cf6d0 2023-11-28T17:41:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.833+0000 7fcb98da1640 1 -- 172.21.15.98:0/3191744735 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90031d60 con 0x7fcb940cf6d0 2023-11-28T17:41:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.834+0000 7fcb9c007640 1 -- 172.21.15.98:0/3191744735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb940cf6d0 msgr2=0x7fcb940c7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.834+0000 7fcb9c007640 1 --2- 172.21.15.98:0/3191744735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb940cf6d0 0x7fcb940c7600 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcb940c6380 tx=0x7fcb9002b910 comp rx=0 tx=0).stop 2023-11-28T17:41:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.834+0000 7fcb9c007640 1 -- 172.21.15.98:0/3191744735 shutdown_connections 2023-11-28T17:41:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.834+0000 7fcb9c007640 1 --2- 172.21.15.98:0/3191744735 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb940cfab0 0x7fcb94159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.834+0000 7fcb9c007640 1 --2- 172.21.15.98:0/3191744735 >> v2:172.21.15.113:3300/0 conn(0x7fcb940cb780 0x7fcb9415a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.834+0000 7fcb9c007640 1 --2- 172.21.15.98:0/3191744735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb940cf6d0 0x7fcb940c7600 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.834+0000 7fcb9c007640 1 -- 172.21.15.98:0/3191744735 >> 172.21.15.98:0/3191744735 conn(0x7fcb9406af10 msgr2=0x7fcb9406b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.834+0000 7fcb9c007640 1 -- 172.21.15.98:0/3191744735 shutdown_connections 2023-11-28T17:41:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.834+0000 7fcb9c007640 1 -- 172.21.15.98:0/3191744735 wait complete. 2023-11-28T17:41:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.836+0000 7fcb9c007640 1 Processor -- start 2023-11-28T17:41:12.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.836+0000 7fcb9c007640 1 -- start start 2023-11-28T17:41:12.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.837+0000 7fcb9c007640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb940cb780 0x7fcb941ef140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:12.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.837+0000 7fcb9c007640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb940cf6d0 0x7fcb94159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:12.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.837+0000 7fcb9c007640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb940cfab0 0x7fcb940c3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:12.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.837+0000 7fcb9c007640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb941f2750 con 0x7fcb940cb780 2023-11-28T17:41:12.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.837+0000 7fcb9c007640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb941f2a30 con 0x7fcb940cfab0 2023-11-28T17:41:12.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.837+0000 7fcb9c007640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb941f2d10 con 0x7fcb940cf6d0 2023-11-28T17:41:12.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.837+0000 7fcb99da3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb940cb780 0x7fcb941ef140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.837+0000 7fcb99da3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb940cb780 0x7fcb941ef140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59100/0 (socket says 172.21.15.98:59100) 2023-11-28T17:41:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.837+0000 7fcb99da3640 1 -- 172.21.15.98:0/4293415898 learned_addr learned my addr 172.21.15.98:0/4293415898 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.838+0000 7fcb99da3640 1 -- 172.21.15.98:0/4293415898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb940cf6d0 msgr2=0x7fcb94159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.838+0000 7fcb99da3640 1 --2- 172.21.15.98:0/4293415898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb940cf6d0 0x7fcb94159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.838+0000 7fcb99da3640 1 -- 172.21.15.98:0/4293415898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb940cfab0 msgr2=0x7fcb940c3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.838+0000 7fcb99da3640 1 --2- 172.21.15.98:0/4293415898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb940cfab0 0x7fcb940c3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.838+0000 7fcb99da3640 1 -- 172.21.15.98:0/4293415898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb900094d0 con 0x7fcb940cb780 2023-11-28T17:41:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.838+0000 7fcb99da3640 1 --2- 172.21.15.98:0/4293415898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb940cb780 0x7fcb941ef140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcb90000c00 tx=0x7fcb900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.838+0000 7fcb8a7fc640 1 -- 172.21.15.98:0/4293415898 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9003ad10 con 0x7fcb940cb780 2023-11-28T17:41:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.838+0000 7fcb8a7fc640 1 -- 172.21.15.98:0/4293415898 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9003ae90 con 0x7fcb940cb780 2023-11-28T17:41:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.838+0000 7fcb8a7fc640 1 -- 172.21.15.98:0/4293415898 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90044630 con 0x7fcb940cb780 2023-11-28T17:41:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.838+0000 7fcb9c007640 1 -- 172.21.15.98:0/4293415898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9415e370 con 0x7fcb940cb780 2023-11-28T17:41:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.838+0000 7fcb9c007640 1 -- 172.21.15.98:0/4293415898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb941f32d0 con 0x7fcb940cb780 2023-11-28T17:41:12.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.840+0000 7fcb9c007640 1 -- 172.21.15.98:0/4293415898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb94074f60 con 0x7fcb940cb780 2023-11-28T17:41:12.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.840+0000 7fcb8a7fc640 1 -- 172.21.15.98:0/4293415898 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) v1 ==== 89493+0+0 (secure 0 0 0) 0x7fcb9004d450 con 0x7fcb940cb780 2023-11-28T17:41:12.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.842+0000 7fcb8a7fc640 1 --2- 172.21.15.98:0/4293415898 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcb7806b210 0x7fcb7806d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:12.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.842+0000 7fcb8a7fc640 1 -- 172.21.15.98:0/4293415898 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcb9002d040 con 0x7fcb940cb780 2023-11-28T17:41:12.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.842+0000 7fcb995a2640 1 --2- 172.21.15.98:0/4293415898 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcb7806b210 0x7fcb7806d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:12.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.843+0000 7fcb995a2640 1 --2- 172.21.15.98:0/4293415898 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcb7806b210 0x7fcb7806d6d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcb84006fd0 tx=0x7fcb84008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:12.849+0000 7fcb8a7fc640 1 -- 172.21.15.98:0/4293415898 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb90077180 con 0x7fcb940cb780 2023-11-28T17:41:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.101+0000 7fcb9c007640 1 -- 172.21.15.98:0/4293415898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb941578f0 con 0x7fcb940cb780 2023-11-28T17:41:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.101+0000 7fcb8a7fc640 1 -- 172.21.15.98:0/4293415898 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5283 (secure 0 0 0) 0x7fcb9007e4c0 con 0x7fcb940cb780 2023-11-28T17:41:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:13.104 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2429440814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2429440814},{"type":"v1","addr":"172.21.15.98:6836","nonce":2429440814}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1057450253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1057450253},{"type":"v1","addr":"172.21.15.113:6835","nonce":1057450253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1925178583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1925178583},{"type":"v1","addr":"172.21.15.98:6837","nonce":1925178583}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/1732613907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":1732613907},{"type":"v1","addr":"172.21.15.113:6837","nonce":1732613907}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3585869153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3585869153},{"type":"v1","addr":"172.21.15.98:6839","nonce":3585869153}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6834/2183585971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2183585971},{"type":"v1","addr":"172.21.15.113:6834","nonce":2183585971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-11-28T17:41:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 3 2023-11-28T17:41:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.104+0000 7fcb9c007640 1 -- 172.21.15.98:0/4293415898 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcb7806b210 msgr2=0x7fcb7806d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.104+0000 7fcb9c007640 1 --2- 172.21.15.98:0/4293415898 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcb7806b210 0x7fcb7806d6d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcb84006fd0 tx=0x7fcb84008040 comp rx=0 tx=0).stop 2023-11-28T17:41:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.104+0000 7fcb9c007640 1 -- 172.21.15.98:0/4293415898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb940cb780 msgr2=0x7fcb941ef140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.104+0000 7fcb9c007640 1 --2- 172.21.15.98:0/4293415898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb940cb780 0x7fcb941ef140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcb90000c00 tx=0x7fcb900046c0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.104+0000 7fcb9c007640 1 -- 172.21.15.98:0/4293415898 shutdown_connections 2023-11-28T17:41:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.104+0000 7fcb9c007640 1 --2- 172.21.15.98:0/4293415898 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcb7806b210 0x7fcb7806d6d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.104+0000 7fcb9c007640 1 --2- 172.21.15.98:0/4293415898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb940cfab0 0x7fcb940c3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.104+0000 7fcb9c007640 1 --2- 172.21.15.98:0/4293415898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb940cf6d0 0x7fcb94159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.104+0000 7fcb9c007640 1 --2- 172.21.15.98:0/4293415898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb940cb780 0x7fcb941ef140 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.104+0000 7fcb9c007640 1 -- 172.21.15.98:0/4293415898 >> 172.21.15.98:0/4293415898 conn(0x7fcb9406af10 msgr2=0x7fcb940c7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.104+0000 7fcb9c007640 1 -- 172.21.15.98:0/4293415898 shutdown_connections 2023-11-28T17:41:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.104+0000 7fcb9c007640 1 -- 172.21.15.98:0/4293415898 wait complete. 2023-11-28T17:41:13.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:41:13.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.312+0000 7f030e8cf640 1 Processor -- start 2023-11-28T17:41:13.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.313+0000 7f030e8cf640 1 -- start start 2023-11-28T17:41:13.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.314+0000 7f030e8cf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0308154120 0x7f03080c6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.314+0000 7f030e8cf640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0308159d40 0x7f030806ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.314+0000 7f030e8cf640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f030806efd0 0x7f03080c7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.314+0000 7f030e8cf640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f030806e670 con 0x7f0308154120 2023-11-28T17:41:13.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.314+0000 7f030e8cf640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03080c7ba0 con 0x7f0308159d40 2023-11-28T17:41:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.314+0000 7f030e8cf640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0308154500 con 0x7f030806efd0 2023-11-28T17:41:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.315+0000 7f0307fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0308154120 0x7f03080c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.315+0000 7f0307fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0308154120 0x7f03080c6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59110/0 (socket says 172.21.15.98:59110) 2023-11-28T17:41:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.315+0000 7f0307fff640 1 -- 172.21.15.98:0/859054362 learned_addr learned my addr 172.21.15.98:0/859054362 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.315+0000 7f030ce6c640 1 --2- 172.21.15.98:0/859054362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f030806efd0 0x7f03080c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:13.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.315+0000 7f0307fff640 1 -- 172.21.15.98:0/859054362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f030806efd0 msgr2=0x7f03080c7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.315+0000 7f02fffff640 1 --2- 172.21.15.98:0/859054362 >> v2:172.21.15.113:3300/0 conn(0x7f0308159d40 0x7f030806ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.315+0000 7f0307fff640 1 --2- 172.21.15.98:0/859054362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f030806efd0 0x7f03080c7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.315+0000 7f0307fff640 1 -- 172.21.15.98:0/859054362 >> v2:172.21.15.113:3300/0 conn(0x7f0308159d40 msgr2=0x7f030806ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.315+0000 7f0307fff640 1 --2- 172.21.15.98:0/859054362 >> v2:172.21.15.113:3300/0 conn(0x7f0308159d40 0x7f030806ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.315+0000 7f0307fff640 1 -- 172.21.15.98:0/859054362 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030806e320 con 0x7f0308154120 2023-11-28T17:41:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.316+0000 7f0307fff640 1 --2- 172.21.15.98:0/859054362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0308154120 0x7f03080c6560 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0308070ab0 tx=0x7f02f802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cdc5c5f1af7f15d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.316+0000 7f03077fe640 1 -- 172.21.15.98:0/859054362 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f80317e0 con 0x7f0308154120 2023-11-28T17:41:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.317+0000 7f03077fe640 1 -- 172.21.15.98:0/859054362 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f8031940 con 0x7f0308154120 2023-11-28T17:41:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.317+0000 7f03077fe640 1 -- 172.21.15.98:0/859054362 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f8031cd0 con 0x7f0308154120 2023-11-28T17:41:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.317+0000 7f030e8cf640 1 -- 172.21.15.98:0/859054362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0308154120 msgr2=0x7f03080c6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.317+0000 7f030e8cf640 1 --2- 172.21.15.98:0/859054362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0308154120 0x7f03080c6560 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0308070ab0 tx=0x7f02f802b880 comp rx=0 tx=0).stop 2023-11-28T17:41:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.317+0000 7f030e8cf640 1 -- 172.21.15.98:0/859054362 shutdown_connections 2023-11-28T17:41:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.317+0000 7f030e8cf640 1 --2- 172.21.15.98:0/859054362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f030806efd0 0x7f03080c7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.317+0000 7f030e8cf640 1 --2- 172.21.15.98:0/859054362 >> v2:172.21.15.113:3300/0 conn(0x7f0308159d40 0x7f030806ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.317+0000 7f030e8cf640 1 --2- 172.21.15.98:0/859054362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0308154120 0x7f03080c6560 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.317+0000 7f030e8cf640 1 -- 172.21.15.98:0/859054362 >> 172.21.15.98:0/859054362 conn(0x7f030806a4d0 msgr2=0x7f030806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.317+0000 7f030e8cf640 1 -- 172.21.15.98:0/859054362 shutdown_connections 2023-11-28T17:41:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.317+0000 7f030e8cf640 1 -- 172.21.15.98:0/859054362 wait complete. 2023-11-28T17:41:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.319+0000 7f030e8cf640 1 Processor -- start 2023-11-28T17:41:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.319+0000 7f030e8cf640 1 -- start start 2023-11-28T17:41:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.320+0000 7f030e8cf640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f030806efd0 0x7f03081f3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.320+0000 7f030e8cf640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0308154120 0x7f03081ef790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.320+0000 7f030e8cf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0308159d40 0x7f030815d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.320+0000 7f030e8cf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03081efcd0 con 0x7f0308159d40 2023-11-28T17:41:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.320+0000 7f030e8cf640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03080cc3e0 con 0x7f030806efd0 2023-11-28T17:41:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.320+0000 7f030e8cf640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030814ab00 con 0x7f0308154120 2023-11-28T17:41:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.320+0000 7f030ce6c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0308159d40 0x7f030815d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:13.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.321+0000 7f030ce6c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0308159d40 0x7f030815d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59120/0 (socket says 172.21.15.98:59120) 2023-11-28T17:41:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.321+0000 7f030ce6c640 1 -- 172.21.15.98:0/390006247 learned_addr learned my addr 172.21.15.98:0/390006247 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.321+0000 7f02fffff640 1 --2- 172.21.15.98:0/390006247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0308154120 0x7f03081ef790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.321+0000 7f0307fff640 1 --2- 172.21.15.98:0/390006247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f030806efd0 0x7f03081f3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.321+0000 7f030ce6c640 1 -- 172.21.15.98:0/390006247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0308154120 msgr2=0x7f03081ef790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.321+0000 7f030ce6c640 1 --2- 172.21.15.98:0/390006247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0308154120 0x7f03081ef790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.321+0000 7f030ce6c640 1 -- 172.21.15.98:0/390006247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f030806efd0 msgr2=0x7f03081f3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.321+0000 7f030ce6c640 1 --2- 172.21.15.98:0/390006247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f030806efd0 0x7f03081f3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.321+0000 7f030ce6c640 1 -- 172.21.15.98:0/390006247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f80094d0 con 0x7f0308159d40 2023-11-28T17:41:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.321+0000 7f0307fff640 1 --2- 172.21.15.98:0/390006247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f030806efd0 0x7f03081f3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.321+0000 7f030ce6c640 1 --2- 172.21.15.98:0/390006247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0308159d40 0x7f030815d2e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f030000cb80 tx=0x7f030000a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.321+0000 7f03057fa640 1 -- 172.21.15.98:0/390006247 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030000ce70 con 0x7f0308159d40 2023-11-28T17:41:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.322+0000 7f03057fa640 1 -- 172.21.15.98:0/390006247 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030000abd0 con 0x7f0308159d40 2023-11-28T17:41:13.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.322+0000 7f03057fa640 1 -- 172.21.15.98:0/390006247 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300011410 con 0x7f0308159d40 2023-11-28T17:41:13.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.322+0000 7f030e8cf640 1 -- 172.21.15.98:0/390006247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03080cc630 con 0x7f0308159d40 2023-11-28T17:41:13.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.322+0000 7f030e8cf640 1 -- 172.21.15.98:0/390006247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03081f3360 con 0x7f0308159d40 2023-11-28T17:41:13.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.323+0000 7f030e8cf640 1 -- 172.21.15.98:0/390006247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03081f3580 con 0x7f0308159d40 2023-11-28T17:41:13.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.324+0000 7f03057fa640 1 -- 172.21.15.98:0/390006247 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) v1 ==== 89493+0+0 (secure 0 0 0) 0x7f0300011570 con 0x7f0308159d40 2023-11-28T17:41:13.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.325+0000 7f03057fa640 1 --2- 172.21.15.98:0/390006247 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f02e806b210 0x7f02e806d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:13.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.325+0000 7f03057fa640 1 -- 172.21.15.98:0/390006247 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0300089580 con 0x7f0308159d40 2023-11-28T17:41:13.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.325+0000 7f0307fff640 1 --2- 172.21.15.98:0/390006247 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f02e806b210 0x7f02e806d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:13.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.326+0000 7f0307fff640 1 --2- 172.21.15.98:0/390006247 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f02e806b210 0x7f02e806d6d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f02f8036000 tx=0x7f02f803d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:13.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.329+0000 7f03057fa640 1 -- 172.21.15.98:0/390006247 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0300054820 con 0x7f0308159d40 2023-11-28T17:41:13.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.526+0000 7f030e8cf640 1 -- 172.21.15.98:0/390006247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0308074f60 con 0x7f0308159d40 2023-11-28T17:41:13.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.528+0000 7f03057fa640 1 -- 172.21.15.98:0/390006247 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12862 (secure 0 0 0) 0x7f0300057e40 con 0x7f0308159d40 2023-11-28T17:41:13.529 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:13.529 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":18,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:41:13.193710+0000","last_up_change":"2023-11-28T17:41:09.556993+0000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:41:10.303280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6810","nonce":2188704385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6813","nonce":2188704385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6819","nonce":2188704385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6816","nonce":2188704385}]},"public_addr":"172.21.15.98:6810/2188704385","cluster_addr":"172.21.15.98:6813/2188704385","heartbeat_back_addr":"172.21.15.98:6819/2188704385","heartbeat_front_addr":"172.21.15.98:6816/2188704385","state":["exists","up"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6801","nonce":3918226860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6803","nonce":3918226860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6807","nonce":3918226860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6805","nonce":3918226860}]},"public_addr":"172.21.15.98:6801/3918226860","cluster_addr":"172.21.15.98:6803/3918226860","heartbeat_back_addr":"172.21.15.98:6807/3918226860","heartbeat_front_addr":"172.21.15.98:6805/3918226860","state":["exists","up"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6812","nonce":2691532400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6815","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6818","nonce":2691532400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6823","nonce":2691532400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6821","nonce":2691532400}]},"public_addr":"172.21.15.98:6812/2691532400","cluster_addr":"172.21.15.98:6818/2691532400","heartbeat_back_addr":"172.21.15.98:6823/2691532400","heartbeat_front_addr":"172.21.15.98:6821/2691532400","state":["exists","up"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6825","nonce":1905487008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6827","nonce":1905487008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6831","nonce":1905487008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6829","nonce":1905487008}]},"public_addr":"172.21.15.98:6825/1905487008","cluster_addr":"172.21.15.98:6827/1905487008","heartbeat_back_addr":"172.21.15.98:6831/1905487008","heartbeat_front_addr":"172.21.15.98:6829/1905487008","state":["exists","up"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6801","nonce":2974677215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6803","nonce":2974677215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6807","nonce":2974677215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6805","nonce":2974677215}]},"public_addr":"172.21.15.113:6801/2974677215","cluster_addr":"172.21.15.113:6803/2974677215","heartbeat_back_addr":"172.21.15.113:6807/2974677215","heartbeat_front_addr":"172.21.15.113:6805/2974677215","state":["exists","up"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6817","nonce":2685000355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6821","nonce":2685000355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6829","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6831","nonce":2685000355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6823","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6826","nonce":2685000355}]},"public_addr":"172.21.15.113:6817/2685000355","cluster_addr":"172.21.15.113:6821/2685000355","heartbeat_back_addr":"172.21.15.113:6831/2685000355","heartbeat_front_addr":"172.21.15.113:6826/2685000355","state":["exists","up"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6809","nonce":2764300831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6811","nonce":2764300831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6815","nonce":2764300831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6813","nonce":2764300831}]},"public_addr":"172.21.15.113:6809/2764300831","cluster_addr":"172.21.15.113:6811/2764300831","heartbeat_back_addr":"172.21.15.113:6815/2764300831","heartbeat_front_addr":"172.21.15.113:6813/2764300831","state":["exists","up"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6819","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6820","nonce":1656248400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6824","nonce":1656248400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6830","nonce":1656248400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6825","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6827","nonce":1656248400}]},"public_addr":"172.21.15.113:6820/1656248400","cluster_addr":"172.21.15.113:6824/1656248400","heartbeat_back_addr":"172.21.15.113:6830/1656248400","heartbeat_front_addr":"172.21.15.113:6827/1656248400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.379549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.312196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.117494+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.388128+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.360608+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.411101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.355995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.390362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:41:13.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.530+0000 7f030e8cf640 1 -- 172.21.15.98:0/390006247 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f02e806b210 msgr2=0x7f02e806d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:13.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.530+0000 7f030e8cf640 1 --2- 172.21.15.98:0/390006247 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f02e806b210 0x7f02e806d6d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f02f8036000 tx=0x7f02f803d040 comp rx=0 tx=0).stop 2023-11-28T17:41:13.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.530+0000 7f030e8cf640 1 -- 172.21.15.98:0/390006247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0308159d40 msgr2=0x7f030815d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:13.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.530+0000 7f030e8cf640 1 --2- 172.21.15.98:0/390006247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0308159d40 0x7f030815d2e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f030000cb80 tx=0x7f030000a4c0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.531+0000 7f030e8cf640 1 -- 172.21.15.98:0/390006247 shutdown_connections 2023-11-28T17:41:13.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.531+0000 7f030e8cf640 1 --2- 172.21.15.98:0/390006247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0308159d40 0x7f030815d2e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.531+0000 7f030e8cf640 1 --2- 172.21.15.98:0/390006247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0308154120 0x7f03081ef790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.531+0000 7f030e8cf640 1 --2- 172.21.15.98:0/390006247 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f02e806b210 0x7f02e806d6d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.531+0000 7f030e8cf640 1 --2- 172.21.15.98:0/390006247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f030806efd0 0x7f03081f3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.531+0000 7f030e8cf640 1 -- 172.21.15.98:0/390006247 >> 172.21.15.98:0/390006247 conn(0x7f030806a4d0 msgr2=0x7f030806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:13.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.531+0000 7f030e8cf640 1 -- 172.21.15.98:0/390006247 shutdown_connections 2023-11-28T17:41:13.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.531+0000 7f030e8cf640 1 -- 172.21.15.98:0/390006247 wait complete. 2023-11-28T17:41:13.549 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:41:10.303280+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:41:13.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:41:13.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.743+0000 7fcfc5f24640 1 Processor -- start 2023-11-28T17:41:13.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.743+0000 7fcfc5f24640 1 -- start start 2023-11-28T17:41:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.744+0000 7fcfc5f24640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfc0153fb0 0x7fcfc00c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.745+0000 7fcfc5f24640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcfc00c76f0 0x7fcfc00c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.745+0000 7fcfc5f24640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc015db80 con 0x7fcfc00b5270 2023-11-28T17:41:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.745+0000 7fcfc5f24640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc00cb730 con 0x7fcfc00c76f0 2023-11-28T17:41:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.745+0000 7fcfc5f24640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00c80c0 con 0x7fcfc0153fb0 2023-11-28T17:41:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.745+0000 7fcfbffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcfc00c76f0 0x7fcfc00c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.745+0000 7fcfbffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcfc00c76f0 0x7fcfc00c7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59128/0 (socket says 172.21.15.98:59128) 2023-11-28T17:41:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.745+0000 7fcfbffff640 1 -- 172.21.15.98:0/1529330898 learned_addr learned my addr 172.21.15.98:0/1529330898 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.745+0000 7fcfb7fff640 1 --1- 172.21.15.98:0/1529330898 >> v1:172.21.15.98:6789/0 conn(0x7fcfc00b5270 0x7fcfc00ca300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48136/0 (socket says 172.21.15.98:48136) 2023-11-28T17:41:13.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.745+0000 7fcfbf7fe640 1 --2- 172.21.15.98:0/1529330898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfc0153fb0 0x7fcfc00c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:13.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.745+0000 7fcfbffff640 1 -- 172.21.15.98:0/1529330898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfc0153fb0 msgr2=0x7fcfc00c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:13.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.745+0000 7fcfbffff640 1 --2- 172.21.15.98:0/1529330898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfc0153fb0 0x7fcfc00c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.745+0000 7fcfbffff640 1 -- 172.21.15.98:0/1529330898 >> v1:172.21.15.98:6789/0 conn(0x7fcfc00b5270 legacy=0x7fcfc00ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:13.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.745+0000 7fcfbffff640 1 -- 172.21.15.98:0/1529330898 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc015e0c0 con 0x7fcfc00c76f0 2023-11-28T17:41:13.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.745+0000 7fcfbeffd640 1 -- 172.21.15.98:0/1529330898 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793709665 0 0) 0x7fcfc015db80 con 0x7fcfc00b5270 2023-11-28T17:41:13.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.747+0000 7fcfbffff640 1 --2- 172.21.15.98:0/1529330898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc00c76f0 0x7fcfc00c7b80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009e50 tx=0x7fcfb802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a3985d56444ba4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:13.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.747+0000 7fcfbeffd640 1 -- 172.21.15.98:0/1529330898 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb802ca70 con 0x7fcfc00c76f0 2023-11-28T17:41:13.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.747+0000 7fcfbeffd640 1 -- 172.21.15.98:0/1529330898 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb802cbd0 con 0x7fcfc00c76f0 2023-11-28T17:41:13.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.747+0000 7fcfbeffd640 1 -- 172.21.15.98:0/1529330898 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb80343f0 con 0x7fcfc00c76f0 2023-11-28T17:41:13.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.748+0000 7fcfc5f24640 1 -- 172.21.15.98:0/1529330898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc00c76f0 msgr2=0x7fcfc00c7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:13.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.748+0000 7fcfc5f24640 1 --2- 172.21.15.98:0/1529330898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc00c76f0 0x7fcfc00c7b80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009e50 tx=0x7fcfb802bed0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.748+0000 7fcfc5f24640 1 -- 172.21.15.98:0/1529330898 shutdown_connections 2023-11-28T17:41:13.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.748+0000 7fcfc5f24640 1 --2- 172.21.15.98:0/1529330898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc00c76f0 0x7fcfc00c7b80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.748+0000 7fcfc5f24640 1 --2- 172.21.15.98:0/1529330898 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfc0153fb0 0x7fcfc00c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.748+0000 7fcfc5f24640 1 -- 172.21.15.98:0/1529330898 >> 172.21.15.98:0/1529330898 conn(0x7fcfc00cf4a0 msgr2=0x7fcfc00cf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:13.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.748+0000 7fcfc5f24640 1 -- 172.21.15.98:0/1529330898 shutdown_connections 2023-11-28T17:41:13.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.748+0000 7fcfc5f24640 1 -- 172.21.15.98:0/1529330898 wait complete. 2023-11-28T17:41:13.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.750+0000 7fcfc5f24640 1 Processor -- start 2023-11-28T17:41:13.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.750+0000 7fcfc5f24640 1 -- start start 2023-11-28T17:41:13.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.751+0000 7fcfc5f24640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc00c76f0 0x7fcfc00c2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:13.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.751+0000 7fcfc5f24640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfc0153fb0 0x7fcfc014dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:13.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.751+0000 7fcfc5f24640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfc00c4ba0 0x7fcfc00c5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:13.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.751+0000 7fcfc5f24640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00c5570 con 0x7fcfc00c76f0 2023-11-28T17:41:13.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.751+0000 7fcfc5f24640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00c37b0 con 0x7fcfc00c4ba0 2023-11-28T17:41:13.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.751+0000 7fcfc5f24640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc01f00c0 con 0x7fcfc0153fb0 2023-11-28T17:41:13.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.751+0000 7fcfbf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc00c76f0 0x7fcfc00c2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:13.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.751+0000 7fcfbf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc00c76f0 0x7fcfc00c2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59134/0 (socket says 172.21.15.98:59134) 2023-11-28T17:41:13.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.751+0000 7fcfbf7fe640 1 -- 172.21.15.98:0/3467007022 learned_addr learned my addr 172.21.15.98:0/3467007022 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:13.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.752+0000 7fcfbf7fe640 1 -- 172.21.15.98:0/3467007022 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfc0153fb0 msgr2=0x7fcfc014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:13.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.752+0000 7fcfbf7fe640 1 --2- 172.21.15.98:0/3467007022 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfc0153fb0 0x7fcfc014dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.752+0000 7fcfbf7fe640 1 -- 172.21.15.98:0/3467007022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfc00c4ba0 msgr2=0x7fcfc00c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:13.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.752+0000 7fcfbf7fe640 1 --2- 172.21.15.98:0/3467007022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfc00c4ba0 0x7fcfc00c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.752+0000 7fcfbf7fe640 1 -- 172.21.15.98:0/3467007022 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb8009b30 con 0x7fcfc00c76f0 2023-11-28T17:41:13.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.752+0000 7fcfbf7fe640 1 --2- 172.21.15.98:0/3467007022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc00c76f0 0x7fcfc00c2ad0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000d800 tx=0x7fcfb000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:13.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.752+0000 7fcfbcff9640 1 -- 172.21.15.98:0/3467007022 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb000b6c0 con 0x7fcfc00c76f0 2023-11-28T17:41:13.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.752+0000 7fcfbcff9640 1 -- 172.21.15.98:0/3467007022 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb000b820 con 0x7fcfc00c76f0 2023-11-28T17:41:13.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.752+0000 7fcfbcff9640 1 -- 172.21.15.98:0/3467007022 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb000bbb0 con 0x7fcfc00c76f0 2023-11-28T17:41:13.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.752+0000 7fcfc5f24640 1 -- 172.21.15.98:0/3467007022 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc00c3a50 con 0x7fcfc00c76f0 2023-11-28T17:41:13.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.753+0000 7fcfc5f24640 1 -- 172.21.15.98:0/3467007022 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc00c4680 con 0x7fcfc00c76f0 2023-11-28T17:41:13.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.754+0000 7fcfc5f24640 1 -- 172.21.15.98:0/3467007022 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc00c4840 con 0x7fcfc00c76f0 2023-11-28T17:41:13.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.755+0000 7fcfbcff9640 1 -- 172.21.15.98:0/3467007022 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) v1 ==== 89493+0+0 (secure 0 0 0) 0x7fcfb000bd10 con 0x7fcfc00c76f0 2023-11-28T17:41:13.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.755+0000 7fcfbcff9640 1 --2- 172.21.15.98:0/3467007022 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcfa006b210 0x7fcfa006d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:13.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.756+0000 7fcfbcff9640 1 -- 172.21.15.98:0/3467007022 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcfb0089dc0 con 0x7fcfc00c76f0 2023-11-28T17:41:13.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.756+0000 7fcfb7fff640 1 --2- 172.21.15.98:0/3467007022 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcfa006b210 0x7fcfa006d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:13.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.756+0000 7fcfb7fff640 1 --2- 172.21.15.98:0/3467007022 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcfa006b210 0x7fcfa006d6d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8007fd0 tx=0x7fcfa8007960 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:13.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.760+0000 7fcfbcff9640 1 -- 172.21.15.98:0/3467007022 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcfb0054fb0 con 0x7fcfc00c76f0 2023-11-28T17:41:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.954+0000 7fcfc5f24640 1 -- 172.21.15.98:0/3467007022 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcfc0074f60 con 0x7fcfc00c76f0 2023-11-28T17:41:13.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.955+0000 7fcfbcff9640 1 -- 172.21.15.98:0/3467007022 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcfb00585d0 con 0x7fcfc00c76f0 2023-11-28T17:41:13.957 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-11-28T17:41:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.957+0000 7fcfc5f24640 1 -- 172.21.15.98:0/3467007022 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcfa006b210 msgr2=0x7fcfa006d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.957+0000 7fcfc5f24640 1 --2- 172.21.15.98:0/3467007022 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcfa006b210 0x7fcfa006d6d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8007fd0 tx=0x7fcfa8007960 comp rx=0 tx=0).stop 2023-11-28T17:41:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.957+0000 7fcfc5f24640 1 -- 172.21.15.98:0/3467007022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc00c76f0 msgr2=0x7fcfc00c2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.957+0000 7fcfc5f24640 1 --2- 172.21.15.98:0/3467007022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc00c76f0 0x7fcfc00c2ad0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000d800 tx=0x7fcfb000dcc0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.958+0000 7fcfc5f24640 1 -- 172.21.15.98:0/3467007022 shutdown_connections 2023-11-28T17:41:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.958+0000 7fcfc5f24640 1 --2- 172.21.15.98:0/3467007022 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcfa006b210 0x7fcfa006d6d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.958+0000 7fcfc5f24640 1 --2- 172.21.15.98:0/3467007022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfc00c4ba0 0x7fcfc00c5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.958+0000 7fcfc5f24640 1 --2- 172.21.15.98:0/3467007022 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcfc0153fb0 0x7fcfc014dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.958+0000 7fcfc5f24640 1 --2- 172.21.15.98:0/3467007022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfc00c76f0 0x7fcfc00c2ad0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.958+0000 7fcfc5f24640 1 -- 172.21.15.98:0/3467007022 >> 172.21.15.98:0/3467007022 conn(0x7fcfc00cf4a0 msgr2=0x7fcfc0152bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.958+0000 7fcfc5f24640 1 -- 172.21.15.98:0/3467007022 shutdown_connections 2023-11-28T17:41:13.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:13.958+0000 7fcfc5f24640 1 -- 172.21.15.98:0/3467007022 wait complete. 2023-11-28T17:41:13.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T17:41:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.189+0000 7f196e815640 1 Processor -- start 2023-11-28T17:41:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.189+0000 7f196e815640 1 -- start start 2023-11-28T17:41:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.190+0000 7f196e815640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1968151d80 0x7f1968157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.191+0000 7f196e815640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f196815bc10 0x7f19680c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.191+0000 7f196e815640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19680c6140 0x7f19680c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.191+0000 7f196e815640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f196815b150 con 0x7f19680c6140 2023-11-28T17:41:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.191+0000 7f196e815640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f196806e1f0 con 0x7f1968151d80 2023-11-28T17:41:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.191+0000 7f196e815640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196806e4d0 con 0x7f196815bc10 2023-11-28T17:41:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.191+0000 7f196cdb2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19680c6140 0x7f19680c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.191+0000 7f196cdb2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19680c6140 0x7f19680c4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59136/0 (socket says 172.21.15.98:59136) 2023-11-28T17:41:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.191+0000 7f196cdb2640 1 -- 172.21.15.98:0/2494353307 learned_addr learned my addr 172.21.15.98:0/2494353307 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.191+0000 7f195ffff640 1 --2- 172.21.15.98:0/2494353307 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f196815bc10 0x7f19680c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.191+0000 7f1967fff640 1 --2- 172.21.15.98:0/2494353307 >> v2:172.21.15.113:3300/0 conn(0x7f1968151d80 0x7f1968157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.192+0000 7f196cdb2640 1 -- 172.21.15.98:0/2494353307 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f196815bc10 msgr2=0x7f19680c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.192+0000 7f196cdb2640 1 --2- 172.21.15.98:0/2494353307 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f196815bc10 0x7f19680c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.192+0000 7f196cdb2640 1 -- 172.21.15.98:0/2494353307 >> v2:172.21.15.113:3300/0 conn(0x7f1968151d80 msgr2=0x7f1968157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.192+0000 7f196cdb2640 1 --2- 172.21.15.98:0/2494353307 >> v2:172.21.15.113:3300/0 conn(0x7f1968151d80 0x7f1968157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.192+0000 7f196cdb2640 1 -- 172.21.15.98:0/2494353307 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196806e7b0 con 0x7f19680c6140 2023-11-28T17:41:14.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.192+0000 7f1967fff640 1 --2- 172.21.15.98:0/2494353307 >> v2:172.21.15.113:3300/0 conn(0x7f1968151d80 0x7f1968157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:41:14.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.193+0000 7f196cdb2640 1 --2- 172.21.15.98:0/2494353307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19680c6140 0x7f19680c4910 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f19680c7c60 tx=0x7f196002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc18d43253dc6c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:14.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.193+0000 7f19677fe640 1 -- 172.21.15.98:0/2494353307 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196002cb00 con 0x7f19680c6140 2023-11-28T17:41:14.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.193+0000 7f19677fe640 1 -- 172.21.15.98:0/2494353307 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196002cc60 con 0x7f19680c6140 2023-11-28T17:41:14.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.193+0000 7f19677fe640 1 -- 172.21.15.98:0/2494353307 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19600343f0 con 0x7f19680c6140 2023-11-28T17:41:14.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.194+0000 7f196e815640 1 -- 172.21.15.98:0/2494353307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19680c6140 msgr2=0x7f19680c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:14.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.194+0000 7f196e815640 1 --2- 172.21.15.98:0/2494353307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19680c6140 0x7f19680c4910 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f19680c7c60 tx=0x7f196002bf60 comp rx=0 tx=0).stop 2023-11-28T17:41:14.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.194+0000 7f196e815640 1 -- 172.21.15.98:0/2494353307 shutdown_connections 2023-11-28T17:41:14.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.194+0000 7f196e815640 1 --2- 172.21.15.98:0/2494353307 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f196815bc10 0x7f19680c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.194+0000 7f196e815640 1 --2- 172.21.15.98:0/2494353307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19680c6140 0x7f19680c4910 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.194+0000 7f196e815640 1 --2- 172.21.15.98:0/2494353307 >> v2:172.21.15.113:3300/0 conn(0x7f1968151d80 0x7f1968157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.194+0000 7f196e815640 1 -- 172.21.15.98:0/2494353307 >> 172.21.15.98:0/2494353307 conn(0x7f196806a4d0 msgr2=0x7f196806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:14.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.194+0000 7f196e815640 1 -- 172.21.15.98:0/2494353307 shutdown_connections 2023-11-28T17:41:14.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.194+0000 7f196e815640 1 -- 172.21.15.98:0/2494353307 wait complete. 2023-11-28T17:41:14.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.196+0000 7f196e815640 1 Processor -- start 2023-11-28T17:41:14.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.196+0000 7f196e815640 1 -- start start 2023-11-28T17:41:14.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.197+0000 7f196e815640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19680c6140 0x7f1968155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:14.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.197+0000 7f196e815640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1968151d80 0x7f19680c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:14.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.197+0000 7f196e815640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f196815bc10 0x7f19681ed780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:14.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.197+0000 7f196e815640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1968156240 con 0x7f19680c6140 2023-11-28T17:41:14.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.197+0000 7f196e815640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1968154c70 con 0x7f1968151d80 2023-11-28T17:41:14.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.197+0000 7f196e815640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19681546b0 con 0x7f196815bc10 2023-11-28T17:41:14.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.197+0000 7f1967fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19680c6140 0x7f1968155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:14.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.197+0000 7f1967fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19680c6140 0x7f1968155d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59142/0 (socket says 172.21.15.98:59142) 2023-11-28T17:41:14.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.197+0000 7f1967fff640 1 -- 172.21.15.98:0/1487714374 learned_addr learned my addr 172.21.15.98:0/1487714374 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:14.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.197+0000 7f196cdb2640 1 --2- 172.21.15.98:0/1487714374 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f196815bc10 0x7f19681ed780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:14.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.198+0000 7f195ffff640 1 --2- 172.21.15.98:0/1487714374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1968151d80 0x7f19680c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:14.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.198+0000 7f1967fff640 1 -- 172.21.15.98:0/1487714374 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f196815bc10 msgr2=0x7f19681ed780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:14.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.198+0000 7f1967fff640 1 --2- 172.21.15.98:0/1487714374 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f196815bc10 0x7f19681ed780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.198+0000 7f1967fff640 1 -- 172.21.15.98:0/1487714374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1968151d80 msgr2=0x7f19680c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:14.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.198+0000 7f1967fff640 1 --2- 172.21.15.98:0/1487714374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1968151d80 0x7f19680c7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.198+0000 7f1967fff640 1 -- 172.21.15.98:0/1487714374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1960009b30 con 0x7f19680c6140 2023-11-28T17:41:14.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.198+0000 7f196cdb2640 1 --2- 172.21.15.98:0/1487714374 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f196815bc10 0x7f19681ed780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:14.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.198+0000 7f1967fff640 1 --2- 172.21.15.98:0/1487714374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19680c6140 0x7f1968155d00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f195800e810 tx=0x7f195800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:14.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.198+0000 7f19657fa640 1 -- 172.21.15.98:0/1487714374 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958004030 con 0x7f19680c6140 2023-11-28T17:41:14.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.198+0000 7f19657fa640 1 -- 172.21.15.98:0/1487714374 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1958004190 con 0x7f19680c6140 2023-11-28T17:41:14.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.198+0000 7f19657fa640 1 -- 172.21.15.98:0/1487714374 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958004520 con 0x7f19680c6140 2023-11-28T17:41:14.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.198+0000 7f196e815640 1 -- 172.21.15.98:0/1487714374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1968154990 con 0x7f19680c6140 2023-11-28T17:41:14.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.198+0000 7f196e815640 1 -- 172.21.15.98:0/1487714374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f196814ffc0 con 0x7f19680c6140 2023-11-28T17:41:14.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.199+0000 7f196e815640 1 -- 172.21.15.98:0/1487714374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1968067bf0 con 0x7f19680c6140 2023-11-28T17:41:14.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.200+0000 7f19657fa640 1 -- 172.21.15.98:0/1487714374 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) v1 ==== 89493+0+0 (secure 0 0 0) 0x7f195800c6d0 con 0x7f19680c6140 2023-11-28T17:41:14.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.201+0000 7f19657fa640 1 --2- 172.21.15.98:0/1487714374 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f194806b240 0x7f194806d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:14.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.201+0000 7f19657fa640 1 -- 172.21.15.98:0/1487714374 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1958089380 con 0x7f19680c6140 2023-11-28T17:41:14.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.201+0000 7f195ffff640 1 --2- 172.21.15.98:0/1487714374 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f194806b240 0x7f194806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:14.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.202+0000 7f195ffff640 1 --2- 172.21.15.98:0/1487714374 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f194806b240 0x7f194806d700 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f19680c23e0 tx=0x7f1950008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:14.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.205+0000 7f19657fa640 1 -- 172.21.15.98:0/1487714374 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1958054580 con 0x7f19680c6140 2023-11-28T17:41:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.397+0000 7f196e815640 1 -- 172.21.15.98:0/1487714374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f196806b2d0 con 0x7f19680c6140 2023-11-28T17:41:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.398+0000 7f19657fa640 1 -- 172.21.15.98:0/1487714374 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1958057ba0 con 0x7f19680c6140 2023-11-28T17:41:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:14.400 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-11-28T17:41:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:41:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2023-11-28T17:41:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2023-11-28T17:41:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:41:14.400 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-11-28T17:41:14.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.400+0000 7f196e815640 1 -- 172.21.15.98:0/1487714374 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f194806b240 msgr2=0x7f194806d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:14.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.400+0000 7f196e815640 1 --2- 172.21.15.98:0/1487714374 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f194806b240 0x7f194806d700 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f19680c23e0 tx=0x7f1950008040 comp rx=0 tx=0).stop 2023-11-28T17:41:14.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.400+0000 7f196e815640 1 -- 172.21.15.98:0/1487714374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19680c6140 msgr2=0x7f1968155d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:14.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.400+0000 7f196e815640 1 --2- 172.21.15.98:0/1487714374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19680c6140 0x7f1968155d00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f195800e810 tx=0x7f195800ecd0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.400+0000 7f196e815640 1 -- 172.21.15.98:0/1487714374 shutdown_connections 2023-11-28T17:41:14.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.400+0000 7f196e815640 1 --2- 172.21.15.98:0/1487714374 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f196815bc10 0x7f19681ed780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.400+0000 7f196e815640 1 --2- 172.21.15.98:0/1487714374 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f194806b240 0x7f194806d700 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.400+0000 7f196e815640 1 --2- 172.21.15.98:0/1487714374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1968151d80 0x7f19680c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.400+0000 7f196e815640 1 --2- 172.21.15.98:0/1487714374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19680c6140 0x7f1968155d00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.400+0000 7f196e815640 1 -- 172.21.15.98:0/1487714374 >> 172.21.15.98:0/1487714374 conn(0x7f196806a4d0 msgr2=0x7f19680716d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:14.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.400+0000 7f196e815640 1 -- 172.21.15.98:0/1487714374 shutdown_connections 2023-11-28T17:41:14.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.401+0000 7f196e815640 1 -- 172.21.15.98:0/1487714374 wait complete. 2023-11-28T17:41:14.420 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T17:41:14.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T17:41:14.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.630+0000 7f9098eff640 1 Processor -- start 2023-11-28T17:41:14.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.630+0000 7f9098eff640 1 -- start start 2023-11-28T17:41:14.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.632+0000 7f9098eff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f90940c80b0 0x7f90941623e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.632+0000 7f9098eff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90940c08e0 con 0x7f9094072970 2023-11-28T17:41:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.632+0000 7f9098eff640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909415ac50 con 0x7f90940c7cd0 2023-11-28T17:41:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.632+0000 7f9098eff640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90940aea10 con 0x7f90940c80b0 2023-11-28T17:41:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.632+0000 7f908bfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9094072970 0x7f9094156e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48138/0 (socket says 172.21.15.98:48138) 2023-11-28T17:41:14.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.632+0000 7f908bfff640 1 -- 172.21.15.98:0/437204011 learned_addr learned my addr 172.21.15.98:0/437204011 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.632+0000 7f9093fff640 1 --2- 172.21.15.98:0/437204011 >> v2:172.21.15.113:3300/0 conn(0x7f90940c80b0 0x7f90941623e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.632+0000 7f9092ffd640 1 -- 172.21.15.98:0/437204011 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782095465 0 0) 0x7f90940c08e0 con 0x7f9094072970 2023-11-28T17:41:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f9092ffd640 1 -- 172.21.15.98:0/437204011 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9080003520 con 0x7f9094072970 2023-11-28T17:41:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f9092ffd640 1 -- 172.21.15.98:0/437204011 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2424297029 0 0) 0x7f9080003520 con 0x7f9094072970 2023-11-28T17:41:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f9092ffd640 1 -- 172.21.15.98:0/437204011 >> v2:172.21.15.113:3300/0 conn(0x7f90940c80b0 msgr2=0x7f90941623e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f9092ffd640 1 --2- 172.21.15.98:0/437204011 >> v2:172.21.15.113:3300/0 conn(0x7f90940c80b0 0x7f90941623e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f9092ffd640 1 -- 172.21.15.98:0/437204011 >> v1:172.21.15.113:6789/0 conn(0x7f90940c7cd0 legacy=0x7f9094153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f9092ffd640 1 -- 172.21.15.98:0/437204011 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90940c0320 con 0x7f9094072970 2023-11-28T17:41:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f9092ffd640 1 -- 172.21.15.98:0/437204011 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f907c001670 con 0x7f9094072970 2023-11-28T17:41:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f9092ffd640 1 -- 172.21.15.98:0/437204011 >> v1:172.21.15.98:6789/0 conn(0x7f9094072970 legacy=0x7f9094156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f9092ffd640 1 --2- 172.21.15.98:0/437204011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90800039c0 0x7f9080003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f9092ffd640 1 --2- 172.21.15.98:0/437204011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90800047c0 0x7f9080004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f9092ffd640 1 --2- 172.21.15.98:0/437204011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9080005180 0x7f908000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f9092ffd640 1 -- 172.21.15.98:0/437204011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908000c050 con 0x7f9080005180 2023-11-28T17:41:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f9092ffd640 1 -- 172.21.15.98:0/437204011 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908000c1c0 con 0x7f90800039c0 2023-11-28T17:41:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f9092ffd640 1 -- 172.21.15.98:0/437204011 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908000c380 con 0x7f90800047c0 2023-11-28T17:41:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f9093fff640 1 --2- 172.21.15.98:0/437204011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9080005180 0x7f908000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f90937fe640 1 --2- 172.21.15.98:0/437204011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90800039c0 0x7f9080003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.633+0000 7f908bfff640 1 --2- 172.21.15.98:0/437204011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90800047c0 0x7f9080004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.634+0000 7f9093fff640 1 -- 172.21.15.98:0/437204011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90800047c0 msgr2=0x7f9080004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.634+0000 7f9093fff640 1 --2- 172.21.15.98:0/437204011 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f90800047c0 0x7f9080004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.634+0000 7f9093fff640 1 -- 172.21.15.98:0/437204011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90800039c0 msgr2=0x7f9080003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.634+0000 7f9093fff640 1 --2- 172.21.15.98:0/437204011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90800039c0 0x7f9080003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.634+0000 7f9093fff640 1 -- 172.21.15.98:0/437204011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9080003520 con 0x7f9080005180 2023-11-28T17:41:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.635+0000 7f9093fff640 1 -- 172.21.15.98:0/437204011 reap_dead start 2023-11-28T17:41:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.635+0000 7f9093fff640 1 --2- 172.21.15.98:0/437204011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9080005180 0x7f908000b980 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f908c009fc0 tx=0x7f90800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1398d50adf5cdbf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.635+0000 7f9092ffd640 1 -- 172.21.15.98:0/437204011 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c02cc60 con 0x7f9080005180 2023-11-28T17:41:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.635+0000 7f9092ffd640 1 -- 172.21.15.98:0/437204011 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f908c00f040 con 0x7f9080005180 2023-11-28T17:41:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.635+0000 7f9092ffd640 1 -- 172.21.15.98:0/437204011 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c003990 con 0x7f9080005180 2023-11-28T17:41:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.636+0000 7f9098eff640 1 -- 172.21.15.98:0/437204011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9080005180 msgr2=0x7f908000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.636+0000 7f9098eff640 1 --2- 172.21.15.98:0/437204011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9080005180 0x7f908000b980 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f908c009fc0 tx=0x7f90800057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.636+0000 7f9098eff640 1 -- 172.21.15.98:0/437204011 shutdown_connections 2023-11-28T17:41:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.636+0000 7f9098eff640 1 --2- 172.21.15.98:0/437204011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9080005180 0x7f908000b980 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.636+0000 7f9098eff640 1 -- 172.21.15.98:0/437204011 >> 172.21.15.98:0/437204011 conn(0x7f909406e600 msgr2=0x7f909406a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.636+0000 7f9098eff640 1 -- 172.21.15.98:0/437204011 shutdown_connections 2023-11-28T17:41:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.636+0000 7f9098eff640 1 -- 172.21.15.98:0/437204011 wait complete. 2023-11-28T17:41:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.638+0000 7f9098eff640 1 Processor -- start 2023-11-28T17:41:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.638+0000 7f9098eff640 1 -- start start 2023-11-28T17:41:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f9098eff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9080005180 0x7f909414fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f9098eff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9094072090 0x7f90940cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f9098eff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f909414cff0 0x7f909414d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f9098eff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909414d9b0 con 0x7f9080005180 2023-11-28T17:41:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f9098eff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90940701f0 con 0x7f909414cff0 2023-11-28T17:41:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f9098eff640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909414db40 con 0x7f9094072090 2023-11-28T17:41:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f90937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9080005180 0x7f909414fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:14.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f90937fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9080005180 0x7f909414fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59160/0 (socket says 172.21.15.98:59160) 2023-11-28T17:41:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f90937fe640 1 -- 172.21.15.98:0/3580388509 learned_addr learned my addr 172.21.15.98:0/3580388509 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f908bfff640 1 --2- 172.21.15.98:0/3580388509 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9094072090 0x7f90940cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f9093fff640 1 --2- 172.21.15.98:0/3580388509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f909414cff0 0x7f909414d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f90937fe640 1 -- 172.21.15.98:0/3580388509 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9094072090 msgr2=0x7f90940cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f90937fe640 1 --2- 172.21.15.98:0/3580388509 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9094072090 0x7f90940cb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f90937fe640 1 -- 172.21.15.98:0/3580388509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f909414cff0 msgr2=0x7f909414d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f90937fe640 1 --2- 172.21.15.98:0/3580388509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f909414cff0 0x7f909414d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f90937fe640 1 -- 172.21.15.98:0/3580388509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9094150890 con 0x7f9080005180 2023-11-28T17:41:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.639+0000 7f9093fff640 1 --2- 172.21.15.98:0/3580388509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f909414cff0 0x7f909414d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.640+0000 7f90937fe640 1 --2- 172.21.15.98:0/3580388509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9080005180 0x7f909414fb60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f908400f5f0 tx=0x7f908400fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.640+0000 7f9090ff9640 1 -- 172.21.15.98:0/3580388509 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90840053a0 con 0x7f9080005180 2023-11-28T17:41:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.640+0000 7f9090ff9640 1 -- 172.21.15.98:0/3580388509 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90840035f0 con 0x7f9080005180 2023-11-28T17:41:14.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.640+0000 7f9090ff9640 1 -- 172.21.15.98:0/3580388509 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90840039a0 con 0x7f9080005180 2023-11-28T17:41:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.640+0000 7f9098eff640 1 -- 172.21.15.98:0/3580388509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909414dde0 con 0x7f9080005180 2023-11-28T17:41:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.640+0000 7f9098eff640 1 -- 172.21.15.98:0/3580388509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90941efea0 con 0x7f9080005180 2023-11-28T17:41:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.641+0000 7f9098eff640 1 -- 172.21.15.98:0/3580388509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90940ae860 con 0x7f9080005180 2023-11-28T17:41:14.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.642+0000 7f9090ff9640 1 -- 172.21.15.98:0/3580388509 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) v1 ==== 89493+0+0 (secure 0 0 0) 0x7f9084003b00 con 0x7f9080005180 2023-11-28T17:41:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.643+0000 7f9090ff9640 1 --2- 172.21.15.98:0/3580388509 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f907406b210 0x7f907406d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.643+0000 7f9090ff9640 1 -- 172.21.15.98:0/3580388509 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9084089600 con 0x7f9080005180 2023-11-28T17:41:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.644+0000 7f908bfff640 1 --2- 172.21.15.98:0/3580388509 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f907406b210 0x7f907406d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:14.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.644+0000 7f908bfff640 1 --2- 172.21.15.98:0/3580388509 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f907406b210 0x7f907406d6d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f90940716e0 tx=0x7f907c000f90 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:14.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.648+0000 7f9090ff9640 1 -- 172.21.15.98:0/3580388509 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9084054960 con 0x7f9080005180 2023-11-28T17:41:14.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:14.841+0000 7f9098eff640 1 -- 172.21.15.98:0/3580388509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9094074f60 con 0x7f9080005180 2023-11-28T17:41:15.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.212+0000 7f9090ff9640 1 -- 172.21.15.98:0/3580388509 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9084003db0 con 0x7f9080005180 2023-11-28T17:41:15.215 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2023-11-28T17:41:15.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.216+0000 7f9098eff640 1 -- 172.21.15.98:0/3580388509 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f907406b210 msgr2=0x7f907406d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:15.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.216+0000 7f9098eff640 1 --2- 172.21.15.98:0/3580388509 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f907406b210 0x7f907406d6d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f90940716e0 tx=0x7f907c000f90 comp rx=0 tx=0).stop 2023-11-28T17:41:15.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.216+0000 7f9098eff640 1 -- 172.21.15.98:0/3580388509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9080005180 msgr2=0x7f909414fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:15.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.216+0000 7f9098eff640 1 --2- 172.21.15.98:0/3580388509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9080005180 0x7f909414fb60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f908400f5f0 tx=0x7f908400fab0 comp rx=0 tx=0).stop 2023-11-28T17:41:15.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.216+0000 7f9098eff640 1 -- 172.21.15.98:0/3580388509 shutdown_connections 2023-11-28T17:41:15.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.216+0000 7f9098eff640 1 --2- 172.21.15.98:0/3580388509 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f907406b210 0x7f907406d6d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:15.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.216+0000 7f9098eff640 1 --2- 172.21.15.98:0/3580388509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f909414cff0 0x7f909414d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:15.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.216+0000 7f9098eff640 1 --2- 172.21.15.98:0/3580388509 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9094072090 0x7f90940cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:15.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.216+0000 7f9098eff640 1 --2- 172.21.15.98:0/3580388509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9080005180 0x7f909414fb60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:15.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.216+0000 7f9098eff640 1 -- 172.21.15.98:0/3580388509 >> 172.21.15.98:0/3580388509 conn(0x7f909406e600 msgr2=0x7f909406a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:15.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.217+0000 7f9098eff640 1 -- 172.21.15.98:0/3580388509 shutdown_connections 2023-11-28T17:41:15.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.218+0000 7f9098eff640 1 -- 172.21.15.98:0/3580388509 wait complete. 2023-11-28T17:41:15.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T17:41:15.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.441+0000 7f15d7bce640 1 Processor -- start 2023-11-28T17:41:15.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.441+0000 7f15d7bce640 1 -- start start 2023-11-28T17:41:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.442+0000 7f15d7bce640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15d0071770 0x7f15d015e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.442+0000 7f15d7bce640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15d00c5a60 0x7f15d00c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.442+0000 7f15d7bce640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d00c6630 con 0x7f15d0071b50 2023-11-28T17:41:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.442+0000 7f15d7bce640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d00c7000 con 0x7f15d0071770 2023-11-28T17:41:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.442+0000 7f15d7bce640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d006e710 con 0x7f15d00c5a60 2023-11-28T17:41:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.443+0000 7f15d596a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15d0071770 0x7f15d015e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.443+0000 7f15d596a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f15d0071770 0x7f15d015e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59166/0 (socket says 172.21.15.98:59166) 2023-11-28T17:41:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.443+0000 7f15d596a640 1 -- 172.21.15.98:0/690380899 learned_addr learned my addr 172.21.15.98:0/690380899 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.443+0000 7f15d5169640 1 --1- 172.21.15.98:0/690380899 >> v1:172.21.15.98:6789/0 conn(0x7f15d0071b50 0x7f15d0157fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48146/0 (socket says 172.21.15.98:48146) 2023-11-28T17:41:15.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.443+0000 7f15d616b640 1 --2- 172.21.15.98:0/690380899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15d00c5a60 0x7f15d00c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:15.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.443+0000 7f15d596a640 1 -- 172.21.15.98:0/690380899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15d00c5a60 msgr2=0x7f15d00c6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:15.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.443+0000 7f15d596a640 1 --2- 172.21.15.98:0/690380899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15d00c5a60 0x7f15d00c6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:15.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.443+0000 7f15d596a640 1 -- 172.21.15.98:0/690380899 >> v1:172.21.15.98:6789/0 conn(0x7f15d0071b50 legacy=0x7f15d0157fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:15.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.443+0000 7f15d596a640 1 -- 172.21.15.98:0/690380899 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d006de70 con 0x7f15d0071770 2023-11-28T17:41:15.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.444+0000 7f15d596a640 1 --2- 172.21.15.98:0/690380899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d0071770 0x7f15d015e110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f15cc009870 tx=0x7f15cc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c022307b88b48bc5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:15.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.445+0000 7f15d4968640 1 -- 172.21.15.98:0/690380899 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15cc031640 con 0x7f15d0071770 2023-11-28T17:41:15.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.445+0000 7f15d4968640 1 -- 172.21.15.98:0/690380899 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15cc0317a0 con 0x7f15d0071770 2023-11-28T17:41:15.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.445+0000 7f15d4968640 1 -- 172.21.15.98:0/690380899 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15cc031b30 con 0x7f15d0071770 2023-11-28T17:41:15.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.445+0000 7f15d7bce640 1 -- 172.21.15.98:0/690380899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d0071770 msgr2=0x7f15d015e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:15.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.445+0000 7f15d7bce640 1 --2- 172.21.15.98:0/690380899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d0071770 0x7f15d015e110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f15cc009870 tx=0x7f15cc02b8f0 comp rx=0 tx=0).stop 2023-11-28T17:41:15.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.445+0000 7f15d7bce640 1 -- 172.21.15.98:0/690380899 shutdown_connections 2023-11-28T17:41:15.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.445+0000 7f15d7bce640 1 --2- 172.21.15.98:0/690380899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15d00c5a60 0x7f15d00c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:15.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.445+0000 7f15d7bce640 1 --2- 172.21.15.98:0/690380899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d0071770 0x7f15d015e110 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:15.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.445+0000 7f15d7bce640 1 -- 172.21.15.98:0/690380899 >> 172.21.15.98:0/690380899 conn(0x7f15d006c0f0 msgr2=0x7f15d006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:15.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.445+0000 7f15d7bce640 1 -- 172.21.15.98:0/690380899 shutdown_connections 2023-11-28T17:41:15.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.445+0000 7f15d7bce640 1 -- 172.21.15.98:0/690380899 wait complete. 2023-11-28T17:41:15.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.447+0000 7f15d7bce640 1 Processor -- start 2023-11-28T17:41:15.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.447+0000 7f15d7bce640 1 -- start start 2023-11-28T17:41:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.448+0000 7f15d7bce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d0071770 0x7f15d00cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.448+0000 7f15d7bce640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15d0071b50 0x7f15d00c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.448+0000 7f15d7bce640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15d00c5a60 0x7f15d00c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.448+0000 7f15d7bce640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d00c93d0 con 0x7f15d0071770 2023-11-28T17:41:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.448+0000 7f15d7bce640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d0151aa0 con 0x7f15d0071b50 2023-11-28T17:41:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.448+0000 7f15d7bce640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d00caff0 con 0x7f15d00c5a60 2023-11-28T17:41:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.448+0000 7f15d596a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d0071770 0x7f15d00cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.448+0000 7f15d596a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d0071770 0x7f15d00cb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59180/0 (socket says 172.21.15.98:59180) 2023-11-28T17:41:15.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.448+0000 7f15d596a640 1 -- 172.21.15.98:0/2346254228 learned_addr learned my addr 172.21.15.98:0/2346254228 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.449+0000 7f15d5169640 1 --2- 172.21.15.98:0/2346254228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15d0071b50 0x7f15d00c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.449+0000 7f15d616b640 1 --2- 172.21.15.98:0/2346254228 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15d00c5a60 0x7f15d00c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.449+0000 7f15d596a640 1 -- 172.21.15.98:0/2346254228 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15d00c5a60 msgr2=0x7f15d00c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.449+0000 7f15d596a640 1 --2- 172.21.15.98:0/2346254228 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15d00c5a60 0x7f15d00c9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.449+0000 7f15d596a640 1 -- 172.21.15.98:0/2346254228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15d0071b50 msgr2=0x7f15d00c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.449+0000 7f15d596a640 1 --2- 172.21.15.98:0/2346254228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15d0071b50 0x7f15d00c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.449+0000 7f15d596a640 1 -- 172.21.15.98:0/2346254228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15cc0094d0 con 0x7f15d0071770 2023-11-28T17:41:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.449+0000 7f15d5169640 1 --2- 172.21.15.98:0/2346254228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15d0071b50 0x7f15d00c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:41:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.449+0000 7f15d596a640 1 --2- 172.21.15.98:0/2346254228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d0071770 0x7f15d00cb5f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f15cc000c00 tx=0x7f15cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.449+0000 7f15c67fc640 1 -- 172.21.15.98:0/2346254228 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15cc03ac60 con 0x7f15d0071770 2023-11-28T17:41:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.449+0000 7f15c67fc640 1 -- 172.21.15.98:0/2346254228 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15cc03adc0 con 0x7f15d0071770 2023-11-28T17:41:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.449+0000 7f15c67fc640 1 -- 172.21.15.98:0/2346254228 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15cc044630 con 0x7f15d0071770 2023-11-28T17:41:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.450+0000 7f15d7bce640 1 -- 172.21.15.98:0/2346254228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d0156980 con 0x7f15d0071770 2023-11-28T17:41:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.450+0000 7f15d7bce640 1 -- 172.21.15.98:0/2346254228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d00c8ff0 con 0x7f15d0071770 2023-11-28T17:41:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.451+0000 7f15d7bce640 1 -- 172.21.15.98:0/2346254228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d0158bf0 con 0x7f15d0071770 2023-11-28T17:41:15.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.452+0000 7f15c67fc640 1 -- 172.21.15.98:0/2346254228 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f15cc02f050 con 0x7f15d0071770 2023-11-28T17:41:15.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.452+0000 7f15c67fc640 1 --2- 172.21.15.98:0/2346254228 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f15b406b2a0 0x7f15b406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:15.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.453+0000 7f15c67fc640 1 -- 172.21.15.98:0/2346254228 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f15cc0ab8a0 con 0x7f15d0071770 2023-11-28T17:41:15.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.453+0000 7f15d5169640 1 --2- 172.21.15.98:0/2346254228 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f15b406b2a0 0x7f15b406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:15.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.453+0000 7f15d5169640 1 --2- 172.21.15.98:0/2346254228 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f15b406b2a0 0x7f15b406d760 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f15d0162790 tx=0x7f15c0007ca0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:15.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.458+0000 7f15c67fc640 1 -- 172.21.15.98:0/2346254228 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15cc0768e0 con 0x7f15d0071770 2023-11-28T17:41:15.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:15.652+0000 7f15d7bce640 1 -- 172.21.15.98:0/2346254228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f15d0074f60 con 0x7f15d0071770 2023-11-28T17:41:16.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.220+0000 7f15c67fc640 1 -- 172.21.15.98:0/2346254228 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) v1 ==== 170+0+0 (secure 0 0 0) 0x7f15cc031e60 con 0x7f15d0071770 2023-11-28T17:41:16.222 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2023-11-28T17:41:16.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.224+0000 7f15d7bce640 1 -- 172.21.15.98:0/2346254228 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f15b406b2a0 msgr2=0x7f15b406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:16.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.224+0000 7f15d7bce640 1 --2- 172.21.15.98:0/2346254228 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f15b406b2a0 0x7f15b406d760 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f15d0162790 tx=0x7f15c0007ca0 comp rx=0 tx=0).stop 2023-11-28T17:41:16.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.224+0000 7f15d7bce640 1 -- 172.21.15.98:0/2346254228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d0071770 msgr2=0x7f15d00cb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:16.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.224+0000 7f15d7bce640 1 --2- 172.21.15.98:0/2346254228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d0071770 0x7f15d00cb5f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f15cc000c00 tx=0x7f15cc0046c0 comp rx=0 tx=0).stop 2023-11-28T17:41:16.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.224+0000 7f15d7bce640 1 -- 172.21.15.98:0/2346254228 shutdown_connections 2023-11-28T17:41:16.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.224+0000 7f15d7bce640 1 --2- 172.21.15.98:0/2346254228 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f15d00c5a60 0x7f15d00c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:16.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.224+0000 7f15d7bce640 1 --2- 172.21.15.98:0/2346254228 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f15b406b2a0 0x7f15b406d760 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:16.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.224+0000 7f15d7bce640 1 --2- 172.21.15.98:0/2346254228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15d0071b50 0x7f15d00c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:16.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.224+0000 7f15d7bce640 1 --2- 172.21.15.98:0/2346254228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15d0071770 0x7f15d00cb5f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:16.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.224+0000 7f15d7bce640 1 -- 172.21.15.98:0/2346254228 >> 172.21.15.98:0/2346254228 conn(0x7f15d006c0f0 msgr2=0x7f15d00c3930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:16.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.224+0000 7f15d7bce640 1 -- 172.21.15.98:0/2346254228 shutdown_connections 2023-11-28T17:41:16.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.225+0000 7f15d7bce640 1 -- 172.21.15.98:0/2346254228 wait complete. 2023-11-28T17:41:16.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T17:41:16.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.441+0000 7f15aa978640 1 Processor -- start 2023-11-28T17:41:16.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.442+0000 7f15aa978640 1 -- start start 2023-11-28T17:41:16.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.443+0000 7f15aa978640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f15a406e490 0x7f15a40c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:16.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.443+0000 7f15aa978640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a40c8150 con 0x7f15a4151be0 2023-11-28T17:41:16.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.443+0000 7f15aa978640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a40c8850 con 0x7f15a40c5a60 2023-11-28T17:41:16.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.443+0000 7f15aa978640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a40c5f50 con 0x7f15a406e490 2023-11-28T17:41:16.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.443+0000 7f15a3fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f15a4151be0 0x7f15a40c4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48160/0 (socket says 172.21.15.98:48160) 2023-11-28T17:41:16.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.443+0000 7f15a3fff640 1 -- 172.21.15.98:0/1067585961 learned_addr learned my addr 172.21.15.98:0/1067585961 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.444+0000 7f15a8f15640 1 --2- 172.21.15.98:0/1067585961 >> v2:172.21.15.113:3300/0 conn(0x7f15a406e490 0x7f15a40c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.444+0000 7f15a37fe640 1 -- 172.21.15.98:0/1067585961 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703106366 0 0) 0x7f15a40c8150 con 0x7f15a4151be0 2023-11-28T17:41:16.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.444+0000 7f15a37fe640 1 -- 172.21.15.98:0/1067585961 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1590003520 con 0x7f15a4151be0 2023-11-28T17:41:16.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.444+0000 7f15a37fe640 1 -- 172.21.15.98:0/1067585961 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 210580651 0 0) 0x7f1590003520 con 0x7f15a4151be0 2023-11-28T17:41:16.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.444+0000 7f15a37fe640 1 -- 172.21.15.98:0/1067585961 >> v2:172.21.15.113:3300/0 conn(0x7f15a406e490 msgr2=0x7f15a40c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:16.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.444+0000 7f15a37fe640 1 --2- 172.21.15.98:0/1067585961 >> v2:172.21.15.113:3300/0 conn(0x7f15a406e490 0x7f15a40c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:16.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.444+0000 7f15a37fe640 1 -- 172.21.15.98:0/1067585961 >> v1:172.21.15.113:6789/0 conn(0x7f15a40c5a60 legacy=0x7f15a406dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:16.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.444+0000 7f15a37fe640 1 -- 172.21.15.98:0/1067585961 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40c7b90 con 0x7f15a4151be0 2023-11-28T17:41:16.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.444+0000 7f15a37fe640 1 -- 172.21.15.98:0/1067585961 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f1594001670 con 0x7f15a4151be0 2023-11-28T17:41:16.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.444+0000 7f15a37fe640 1 -- 172.21.15.98:0/1067585961 >> v1:172.21.15.98:6789/0 conn(0x7f15a4151be0 legacy=0x7f15a40c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:16.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.444+0000 7f15a37fe640 1 --2- 172.21.15.98:0/1067585961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1590003970 0x7f1590003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:16.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.445+0000 7f15a37fe640 1 --2- 172.21.15.98:0/1067585961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15900047c0 0x7f1590004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:16.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.445+0000 7f15a37fe640 1 --2- 172.21.15.98:0/1067585961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1590005180 0x7f159000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:16.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.445+0000 7f15a37fe640 1 -- 172.21.15.98:0/1067585961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40c5f50 con 0x7f1590003970 2023-11-28T17:41:16.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.445+0000 7f15a37fe640 1 -- 172.21.15.98:0/1067585961 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159000c020 con 0x7f15900047c0 2023-11-28T17:41:16.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.445+0000 7f15a37fe640 1 -- 172.21.15.98:0/1067585961 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159000c1e0 con 0x7f1590005180 2023-11-28T17:41:16.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.445+0000 7f15a3fff640 1 --2- 172.21.15.98:0/1067585961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1590003970 0x7f1590003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:16.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.445+0000 7f159bfff640 1 --2- 172.21.15.98:0/1067585961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15900047c0 0x7f1590004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:16.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.445+0000 7f15a3fff640 1 -- 172.21.15.98:0/1067585961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1590005180 msgr2=0x7f159000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:16.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.445+0000 7f15a8f15640 1 --2- 172.21.15.98:0/1067585961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1590005180 0x7f159000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:16.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.445+0000 7f15a3fff640 1 --2- 172.21.15.98:0/1067585961 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1590005180 0x7f159000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:16.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.445+0000 7f15a3fff640 1 -- 172.21.15.98:0/1067585961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15900047c0 msgr2=0x7f1590004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:16.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.445+0000 7f15a3fff640 1 --2- 172.21.15.98:0/1067585961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15900047c0 0x7f1590004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:16.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.445+0000 7f15a3fff640 1 -- 172.21.15.98:0/1067585961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1590003520 con 0x7f1590003970 2023-11-28T17:41:16.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.445+0000 7f159bfff640 1 --2- 172.21.15.98:0/1067585961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15900047c0 0x7f1590004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:16.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.445+0000 7f15a8f15640 1 -- 172.21.15.98:0/1067585961 reap_dead start 2023-11-28T17:41:16.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.446+0000 7f15a3fff640 1 --2- 172.21.15.98:0/1067585961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1590003970 0x7f1590003d90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f159400c190 tx=0x7f15900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9efe38060c29f403 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:16.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.446+0000 7f15a37fe640 1 -- 172.21.15.98:0/1067585961 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594005ad0 con 0x7f1590003970 2023-11-28T17:41:16.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.447+0000 7f15a37fe640 1 -- 172.21.15.98:0/1067585961 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1594005c50 con 0x7f1590003970 2023-11-28T17:41:16.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.447+0000 7f15a37fe640 1 -- 172.21.15.98:0/1067585961 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159402c7a0 con 0x7f1590003970 2023-11-28T17:41:16.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.447+0000 7f15aa978640 1 -- 172.21.15.98:0/1067585961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1590003970 msgr2=0x7f1590003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:16.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.447+0000 7f15aa978640 1 --2- 172.21.15.98:0/1067585961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1590003970 0x7f1590003d90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f159400c190 tx=0x7f15900057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:16.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.447+0000 7f15aa978640 1 -- 172.21.15.98:0/1067585961 shutdown_connections 2023-11-28T17:41:16.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.447+0000 7f15aa978640 1 --2- 172.21.15.98:0/1067585961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1590003970 0x7f1590003d90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:16.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.447+0000 7f15aa978640 1 -- 172.21.15.98:0/1067585961 >> 172.21.15.98:0/1067585961 conn(0x7f15a406c0f0 msgr2=0x7f15a406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:16.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.447+0000 7f15aa978640 1 -- 172.21.15.98:0/1067585961 shutdown_connections 2023-11-28T17:41:16.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.447+0000 7f15aa978640 1 -- 172.21.15.98:0/1067585961 wait complete. 2023-11-28T17:41:16.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.449+0000 7f15aa978640 1 Processor -- start 2023-11-28T17:41:16.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.449+0000 7f15aa978640 1 -- start start 2023-11-28T17:41:16.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.450+0000 7f15aa978640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1590003970 0x7f15a41f01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:16.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.450+0000 7f15aa978640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a4149e40 0x7f15a414a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:16.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.450+0000 7f15aa978640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a415abd0 0x7f15a415b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:16.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.450+0000 7f15aa978640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a414c6e0 con 0x7f15a415abd0 2023-11-28T17:41:16.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.450+0000 7f15aa978640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a414a800 con 0x7f15a4149e40 2023-11-28T17:41:16.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.450+0000 7f15aa978640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a41f06f0 con 0x7f1590003970 2023-11-28T17:41:16.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.450+0000 7f15a8f15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a415abd0 0x7f15a415b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:16.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.450+0000 7f15a8f15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a415abd0 0x7f15a415b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59204/0 (socket says 172.21.15.98:59204) 2023-11-28T17:41:16.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.450+0000 7f15a8f15640 1 -- 172.21.15.98:0/2311661690 learned_addr learned my addr 172.21.15.98:0/2311661690 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:16.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.451+0000 7f15a3fff640 1 --2- 172.21.15.98:0/2311661690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1590003970 0x7f15a41f01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:16.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.451+0000 7f159bfff640 1 --2- 172.21.15.98:0/2311661690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a4149e40 0x7f15a414a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:16.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.451+0000 7f15a8f15640 1 -- 172.21.15.98:0/2311661690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1590003970 msgr2=0x7f15a41f01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:16.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.451+0000 7f15a8f15640 1 --2- 172.21.15.98:0/2311661690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1590003970 0x7f15a41f01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:16.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.451+0000 7f15a8f15640 1 -- 172.21.15.98:0/2311661690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a4149e40 msgr2=0x7f15a414a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:16.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.451+0000 7f15a8f15640 1 --2- 172.21.15.98:0/2311661690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a4149e40 0x7f15a414a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:16.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.451+0000 7f15a8f15640 1 -- 172.21.15.98:0/2311661690 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a4159ea0 con 0x7f15a415abd0 2023-11-28T17:41:16.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.451+0000 7f15a8f15640 1 --2- 172.21.15.98:0/2311661690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a415abd0 0x7f15a415b050 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f15a41555e0 tx=0x7f159c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:16.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.451+0000 7f15a17fa640 1 -- 172.21.15.98:0/2311661690 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c01a070 con 0x7f15a415abd0 2023-11-28T17:41:16.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.451+0000 7f15a17fa640 1 -- 172.21.15.98:0/2311661690 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159c00fd90 con 0x7f15a415abd0 2023-11-28T17:41:16.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.451+0000 7f15a17fa640 1 -- 172.21.15.98:0/2311661690 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c003bb0 con 0x7f15a415abd0 2023-11-28T17:41:16.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.451+0000 7f15aa978640 1 -- 172.21.15.98:0/2311661690 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a415a740 con 0x7f15a415abd0 2023-11-28T17:41:16.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.452+0000 7f15aa978640 1 -- 172.21.15.98:0/2311661690 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a414d6a0 con 0x7f15a415abd0 2023-11-28T17:41:16.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.453+0000 7f15aa978640 1 -- 172.21.15.98:0/2311661690 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a4074f60 con 0x7f15a415abd0 2023-11-28T17:41:16.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.454+0000 7f15a17fa640 1 -- 172.21.15.98:0/2311661690 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f159c016420 con 0x7f15a415abd0 2023-11-28T17:41:16.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.454+0000 7f15a17fa640 1 --2- 172.21.15.98:0/2311661690 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f158406b2a0 0x7f158406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:16.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.455+0000 7f15a17fa640 1 -- 172.21.15.98:0/2311661690 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f159c08a2d0 con 0x7f15a415abd0 2023-11-28T17:41:16.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.455+0000 7f15a3fff640 1 --2- 172.21.15.98:0/2311661690 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f158406b2a0 0x7f158406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:16.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.455+0000 7f15a3fff640 1 --2- 172.21.15.98:0/2311661690 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f158406b2a0 0x7f158406d760 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1594034000 tx=0x7f1594009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:16.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.460+0000 7f15a17fa640 1 -- 172.21.15.98:0/2311661690 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f159c0551a0 con 0x7f15a415abd0 2023-11-28T17:41:16.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:16.689+0000 7f15aa978640 1 -- 172.21.15.98:0/2311661690 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f15a414d940 con 0x7f15a415abd0 2023-11-28T17:41:17.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.245+0000 7f15a17fa640 1 -- 172.21.15.98:0/2311661690 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f159c0587c0 con 0x7f15a415abd0 2023-11-28T17:41:17.247 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 2 and data pool 3 2023-11-28T17:41:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.248+0000 7f15aa978640 1 -- 172.21.15.98:0/2311661690 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f158406b2a0 msgr2=0x7f158406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.248+0000 7f15aa978640 1 --2- 172.21.15.98:0/2311661690 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f158406b2a0 0x7f158406d760 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1594034000 tx=0x7f1594009030 comp rx=0 tx=0).stop 2023-11-28T17:41:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.248+0000 7f15aa978640 1 -- 172.21.15.98:0/2311661690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a415abd0 msgr2=0x7f15a415b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.248+0000 7f15aa978640 1 --2- 172.21.15.98:0/2311661690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a415abd0 0x7f15a415b050 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f15a41555e0 tx=0x7f159c00f520 comp rx=0 tx=0).stop 2023-11-28T17:41:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.248+0000 7f15aa978640 1 -- 172.21.15.98:0/2311661690 shutdown_connections 2023-11-28T17:41:17.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.248+0000 7f15aa978640 1 --2- 172.21.15.98:0/2311661690 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15a415abd0 0x7f15a415b050 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.248+0000 7f15aa978640 1 --2- 172.21.15.98:0/2311661690 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f158406b2a0 0x7f158406d760 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.248+0000 7f15aa978640 1 --2- 172.21.15.98:0/2311661690 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15a4149e40 0x7f15a414a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.248+0000 7f15aa978640 1 --2- 172.21.15.98:0/2311661690 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1590003970 0x7f15a41f01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.248+0000 7f15aa978640 1 -- 172.21.15.98:0/2311661690 >> 172.21.15.98:0/2311661690 conn(0x7f15a406c0f0 msgr2=0x7f15a406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:17.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.249+0000 7f15aa978640 1 -- 172.21.15.98:0/2311661690 shutdown_connections 2023-11-28T17:41:17.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.249+0000 7f15aa978640 1 -- 172.21.15.98:0/2311661690 wait complete. 2023-11-28T17:41:17.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:41:17.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.452+0000 7fc4444a6640 1 Processor -- start 2023-11-28T17:41:17.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.452+0000 7fc4444a6640 1 -- start start 2023-11-28T17:41:17.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.454+0000 7fc4444a6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc43c151d80 0x7fc43c157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:17.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.454+0000 7fc4444a6640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc43c0c4910 0x7fc43c0c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:17.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.454+0000 7fc4444a6640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43c15b150 con 0x7fc43c15bc10 2023-11-28T17:41:17.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.454+0000 7fc4444a6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc43c0c5260 con 0x7fc43c151d80 2023-11-28T17:41:17.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.454+0000 7fc4444a6640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc43c06e1f0 con 0x7fc43c0c4910 2023-11-28T17:41:17.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.454+0000 7fc442242640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc43c151d80 0x7fc43c157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:17.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.454+0000 7fc442242640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc43c151d80 0x7fc43c157b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59214/0 (socket says 172.21.15.98:59214) 2023-11-28T17:41:17.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.454+0000 7fc442242640 1 -- 172.21.15.98:0/2466919664 learned_addr learned my addr 172.21.15.98:0/2466919664 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:17.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.455+0000 7fc442242640 1 -- 172.21.15.98:0/2466919664 >> v2:172.21.15.113:3300/0 conn(0x7fc43c0c4910 msgr2=0x7fc43c0c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:17.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.455+0000 7fc442242640 1 --2- 172.21.15.98:0/2466919664 >> v2:172.21.15.113:3300/0 conn(0x7fc43c0c4910 0x7fc43c0c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.455+0000 7fc442242640 1 -- 172.21.15.98:0/2466919664 >> v1:172.21.15.113:6789/0 conn(0x7fc43c15bc10 legacy=0x7fc43c0c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:17.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.455+0000 7fc442242640 1 -- 172.21.15.98:0/2466919664 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c06e4d0 con 0x7fc43c151d80 2023-11-28T17:41:17.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.456+0000 7fc442242640 1 --2- 172.21.15.98:0/2466919664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc43c151d80 0x7fc43c157b60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc43c0c55e0 tx=0x7fc43802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd7b0e26bde6dc7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:17.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.456+0000 7fc441240640 1 -- 172.21.15.98:0/2466919664 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc438030860 con 0x7fc43c151d80 2023-11-28T17:41:17.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.456+0000 7fc441240640 1 -- 172.21.15.98:0/2466919664 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4380309c0 con 0x7fc43c151d80 2023-11-28T17:41:17.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.456+0000 7fc441240640 1 -- 172.21.15.98:0/2466919664 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc438030d50 con 0x7fc43c151d80 2023-11-28T17:41:17.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.457+0000 7fc4444a6640 1 -- 172.21.15.98:0/2466919664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc43c151d80 msgr2=0x7fc43c157b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:17.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.457+0000 7fc4444a6640 1 --2- 172.21.15.98:0/2466919664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc43c151d80 0x7fc43c157b60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc43c0c55e0 tx=0x7fc43802b9a0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.457+0000 7fc4444a6640 1 -- 172.21.15.98:0/2466919664 shutdown_connections 2023-11-28T17:41:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.457+0000 7fc4444a6640 1 --2- 172.21.15.98:0/2466919664 >> v2:172.21.15.113:3300/0 conn(0x7fc43c0c4910 0x7fc43c0c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.457+0000 7fc4444a6640 1 --2- 172.21.15.98:0/2466919664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc43c151d80 0x7fc43c157b60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.457+0000 7fc4444a6640 1 -- 172.21.15.98:0/2466919664 >> 172.21.15.98:0/2466919664 conn(0x7fc43c06a4d0 msgr2=0x7fc43c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.457+0000 7fc4444a6640 1 -- 172.21.15.98:0/2466919664 shutdown_connections 2023-11-28T17:41:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.457+0000 7fc4444a6640 1 -- 172.21.15.98:0/2466919664 wait complete. 2023-11-28T17:41:17.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.458+0000 7fc4444a6640 1 Processor -- start 2023-11-28T17:41:17.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.459+0000 7fc4444a6640 1 -- start start 2023-11-28T17:41:17.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.459+0000 7fc4444a6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc43c0c4910 0x7fc43c1e2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:17.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.459+0000 7fc4444a6640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc43c151d80 0x7fc43c1f0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:17.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.460+0000 7fc4444a6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc43c15bc10 0x7fc43c149a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:17.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.460+0000 7fc4444a6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c14ad50 con 0x7fc43c15bc10 2023-11-28T17:41:17.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.460+0000 7fc4444a6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c1efe20 con 0x7fc43c0c4910 2023-11-28T17:41:17.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.460+0000 7fc4444a6640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c1f3880 con 0x7fc43c151d80 2023-11-28T17:41:17.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.460+0000 7fc442a43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc43c15bc10 0x7fc43c149a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:17.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.460+0000 7fc442a43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc43c15bc10 0x7fc43c149a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59224/0 (socket says 172.21.15.98:59224) 2023-11-28T17:41:17.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.460+0000 7fc442a43640 1 -- 172.21.15.98:0/551357536 learned_addr learned my addr 172.21.15.98:0/551357536 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:17.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.460+0000 7fc442242640 1 --2- 172.21.15.98:0/551357536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc43c0c4910 0x7fc43c1e2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.460+0000 7fc441a41640 1 --2- 172.21.15.98:0/551357536 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc43c151d80 0x7fc43c1f0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.460+0000 7fc442a43640 1 -- 172.21.15.98:0/551357536 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc43c151d80 msgr2=0x7fc43c1f0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.460+0000 7fc442a43640 1 --2- 172.21.15.98:0/551357536 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc43c151d80 0x7fc43c1f0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.460+0000 7fc442a43640 1 -- 172.21.15.98:0/551357536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc43c0c4910 msgr2=0x7fc43c1e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.460+0000 7fc442a43640 1 --2- 172.21.15.98:0/551357536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc43c0c4910 0x7fc43c1e2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.460+0000 7fc442a43640 1 -- 172.21.15.98:0/551357536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4380094d0 con 0x7fc43c15bc10 2023-11-28T17:41:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.461+0000 7fc442a43640 1 --2- 172.21.15.98:0/551357536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc43c15bc10 0x7fc43c149a20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc43400b970 tx=0x7fc43400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:17.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.461+0000 7fc432ffd640 1 -- 172.21.15.98:0/551357536 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43400cb40 con 0x7fc43c15bc10 2023-11-28T17:41:17.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.461+0000 7fc432ffd640 1 -- 172.21.15.98:0/551357536 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43400ccc0 con 0x7fc43c15bc10 2023-11-28T17:41:17.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.461+0000 7fc4444a6640 1 -- 172.21.15.98:0/551357536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43c1f3b60 con 0x7fc43c15bc10 2023-11-28T17:41:17.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.461+0000 7fc432ffd640 1 -- 172.21.15.98:0/551357536 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc434014680 con 0x7fc43c15bc10 2023-11-28T17:41:17.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.461+0000 7fc4444a6640 1 -- 172.21.15.98:0/551357536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43c1f4120 con 0x7fc43c15bc10 2023-11-28T17:41:17.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.462+0000 7fc4444a6640 1 -- 172.21.15.98:0/551357536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc43c067bf0 con 0x7fc43c15bc10 2023-11-28T17:41:17.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.464+0000 7fc432ffd640 1 -- 172.21.15.98:0/551357536 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fc43401d3b0 con 0x7fc43c15bc10 2023-11-28T17:41:17.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.464+0000 7fc432ffd640 1 --2- 172.21.15.98:0/551357536 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc42006b2a0 0x7fc42006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:17.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.465+0000 7fc432ffd640 1 -- 172.21.15.98:0/551357536 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc434089c30 con 0x7fc43c15bc10 2023-11-28T17:41:17.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.465+0000 7fc442242640 1 --2- 172.21.15.98:0/551357536 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc42006b2a0 0x7fc42006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:17.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.465+0000 7fc442242640 1 --2- 172.21.15.98:0/551357536 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc42006b2a0 0x7fc42006d760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc438036000 tx=0x7fc438008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:17.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.469+0000 7fc432ffd640 1 -- 172.21.15.98:0/551357536 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc434090080 con 0x7fc43c15bc10 2023-11-28T17:41:17.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.663+0000 7fc4444a6640 1 -- 172.21.15.98:0/551357536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc43c074f60 con 0x7fc43c15bc10 2023-11-28T17:41:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.665+0000 7fc432ffd640 1 -- 172.21.15.98:0/551357536 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16530 (secure 0 0 0) 0x7fc434019070 con 0x7fc43c15bc10 2023-11-28T17:41:17.667 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:17.668 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":21,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:41:16.691297+0000","last_up_change":"2023-11-28T17:41:09.556993+0000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:41:10.303280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:41:14.890871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:41:15.704103+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6810","nonce":2188704385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6813","nonce":2188704385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6819","nonce":2188704385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6816","nonce":2188704385}]},"public_addr":"172.21.15.98:6810/2188704385","cluster_addr":"172.21.15.98:6813/2188704385","heartbeat_back_addr":"172.21.15.98:6819/2188704385","heartbeat_front_addr":"172.21.15.98:6816/2188704385","state":["exists","up"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6801","nonce":3918226860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6803","nonce":3918226860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6807","nonce":3918226860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6805","nonce":3918226860}]},"public_addr":"172.21.15.98:6801/3918226860","cluster_addr":"172.21.15.98:6803/3918226860","heartbeat_back_addr":"172.21.15.98:6807/3918226860","heartbeat_front_addr":"172.21.15.98:6805/3918226860","state":["exists","up"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6812","nonce":2691532400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6815","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6818","nonce":2691532400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6823","nonce":2691532400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6821","nonce":2691532400}]},"public_addr":"172.21.15.98:6812/2691532400","cluster_addr":"172.21.15.98:6818/2691532400","heartbeat_back_addr":"172.21.15.98:6823/2691532400","heartbeat_front_addr":"172.21.15.98:6821/2691532400","state":["exists","up"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6825","nonce":1905487008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6827","nonce":1905487008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6831","nonce":1905487008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6829","nonce":1905487008}]},"public_addr":"172.21.15.98:6825/1905487008","cluster_addr":"172.21.15.98:6827/1905487008","heartbeat_back_addr":"172.21.15.98:6831/1905487008","heartbeat_front_addr":"172.21.15.98:6829/1905487008","state":["exists","up"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6801","nonce":2974677215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6803","nonce":2974677215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6807","nonce":2974677215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6805","nonce":2974677215}]},"public_addr":"172.21.15.113:6801/2974677215","cluster_addr":"172.21.15.113:6803/2974677215","heartbeat_back_addr":"172.21.15.113:6807/2974677215","heartbeat_front_addr":"172.21.15.113:6805/2974677215","state":["exists","up"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6817","nonce":2685000355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6821","nonce":2685000355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6829","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6831","nonce":2685000355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6823","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6826","nonce":2685000355}]},"public_addr":"172.21.15.113:6817/2685000355","cluster_addr":"172.21.15.113:6821/2685000355","heartbeat_back_addr":"172.21.15.113:6831/2685000355","heartbeat_front_addr":"172.21.15.113:6826/2685000355","state":["exists","up"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6809","nonce":2764300831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6811","nonce":2764300831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6815","nonce":2764300831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6813","nonce":2764300831}]},"public_addr":"172.21.15.113:6809/2764300831","cluster_addr":"172.21.15.113:6811/2764300831","heartbeat_back_addr":"172.21.15.113:6815/2764300831","heartbeat_front_addr":"172.21.15.113:6813/2764300831","state":["exists","up"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6819","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6820","nonce":1656248400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6824","nonce":1656248400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6830","nonce":1656248400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6825","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6827","nonce":1656248400}]},"public_addr":"172.21.15.113:6820/1656248400","cluster_addr":"172.21.15.113:6824/1656248400","heartbeat_back_addr":"172.21.15.113:6830/1656248400","heartbeat_front_addr":"172.21.15.113:6827/1656248400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.379549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.312196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.117494+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.388128+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.360608+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.411101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.355995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.390362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:41:17.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.668+0000 7fc4444a6640 1 -- 172.21.15.98:0/551357536 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc42006b2a0 msgr2=0x7fc42006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:17.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.668+0000 7fc4444a6640 1 --2- 172.21.15.98:0/551357536 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc42006b2a0 0x7fc42006d760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc438036000 tx=0x7fc438008030 comp rx=0 tx=0).stop 2023-11-28T17:41:17.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.668+0000 7fc4444a6640 1 -- 172.21.15.98:0/551357536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc43c15bc10 msgr2=0x7fc43c149a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:17.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.668+0000 7fc4444a6640 1 --2- 172.21.15.98:0/551357536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc43c15bc10 0x7fc43c149a20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc43400b970 tx=0x7fc43400be30 comp rx=0 tx=0).stop 2023-11-28T17:41:17.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.668+0000 7fc4444a6640 1 -- 172.21.15.98:0/551357536 shutdown_connections 2023-11-28T17:41:17.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.668+0000 7fc4444a6640 1 --2- 172.21.15.98:0/551357536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc43c15bc10 0x7fc43c149a20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.668+0000 7fc4444a6640 1 --2- 172.21.15.98:0/551357536 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc43c151d80 0x7fc43c1f0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.668+0000 7fc4444a6640 1 --2- 172.21.15.98:0/551357536 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc42006b2a0 0x7fc42006d760 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.668+0000 7fc4444a6640 1 --2- 172.21.15.98:0/551357536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc43c0c4910 0x7fc43c1e2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.668+0000 7fc4444a6640 1 -- 172.21.15.98:0/551357536 >> 172.21.15.98:0/551357536 conn(0x7fc43c06a4d0 msgr2=0x7fc43c150280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:17.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.668+0000 7fc4444a6640 1 -- 172.21.15.98:0/551357536 shutdown_connections 2023-11-28T17:41:17.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.668+0000 7fc4444a6640 1 -- 172.21.15.98:0/551357536 wait complete. 2023-11-28T17:41:17.687 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:41:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.894+0000 7f0078a94640 1 Processor -- start 2023-11-28T17:41:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.894+0000 7f0078a94640 1 -- start start 2023-11-28T17:41:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.895+0000 7f0078a94640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0074150c00 0x7f00740c2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.896+0000 7f0078a94640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0074157850 con 0x7f00740c6600 2023-11-28T17:41:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.896+0000 7f0078a94640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0074156980 con 0x7f00740c5c00 2023-11-28T17:41:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.896+0000 7f0078a94640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0074156c40 con 0x7f0074150c00 2023-11-28T17:41:17.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.896+0000 7f0072d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f00740c6600 0x7f00740c6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48162/0 (socket says 172.21.15.98:48162) 2023-11-28T17:41:17.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.896+0000 7f0072d9d640 1 -- 172.21.15.98:0/3732664018 learned_addr learned my addr 172.21.15.98:0/3732664018 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:17.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.896+0000 7f007259c640 1 --2- 172.21.15.98:0/3732664018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0074150c00 0x7f00740c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:17.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.896+0000 7f007159a640 1 -- 172.21.15.98:0/3732664018 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29787054 0 0) 0x7f0074157850 con 0x7f00740c6600 2023-11-28T17:41:17.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.896+0000 7f007159a640 1 -- 172.21.15.98:0/3732664018 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f005c003520 con 0x7f00740c6600 2023-11-28T17:41:17.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f007159a640 1 -- 172.21.15.98:0/3732664018 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 612491914 0 0) 0x7f005c003520 con 0x7f00740c6600 2023-11-28T17:41:17.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f007159a640 1 -- 172.21.15.98:0/3732664018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0074150c00 msgr2=0x7f00740c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:17.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f007159a640 1 --2- 172.21.15.98:0/3732664018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0074150c00 0x7f00740c2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f007159a640 1 -- 172.21.15.98:0/3732664018 >> v1:172.21.15.113:6789/0 conn(0x7f00740c5c00 legacy=0x7f00740c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:17.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f007159a640 1 -- 172.21.15.98:0/3732664018 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0074156f20 con 0x7f00740c6600 2023-11-28T17:41:17.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f007159a640 1 -- 172.21.15.98:0/3732664018 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f006c0019f0 con 0x7f00740c6600 2023-11-28T17:41:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f007159a640 1 -- 172.21.15.98:0/3732664018 >> v1:172.21.15.98:6789/0 conn(0x7f00740c6600 legacy=0x7f00740c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f007159a640 1 --2- 172.21.15.98:0/3732664018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0074150c00 0x7f005c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f007159a640 1 --2- 172.21.15.98:0/3732664018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f005c003f70 0x7f005c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f007159a640 1 --2- 172.21.15.98:0/3732664018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f005c004d00 0x7f005c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f007159a640 1 -- 172.21.15.98:0/3732664018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0074156c40 con 0x7f005c003f70 2023-11-28T17:41:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f007159a640 1 -- 172.21.15.98:0/3732664018 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c00bac0 con 0x7f005c004d00 2023-11-28T17:41:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f007159a640 1 -- 172.21.15.98:0/3732664018 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005c00bc80 con 0x7f0074150c00 2023-11-28T17:41:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f0071d9b640 1 --2- 172.21.15.98:0/3732664018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f005c003f70 0x7f005c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f007259c640 1 --2- 172.21.15.98:0/3732664018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0074150c00 0x7f005c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.897+0000 7f0072d9d640 1 --2- 172.21.15.98:0/3732664018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f005c004d00 0x7f005c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.898+0000 7f0071d9b640 1 -- 172.21.15.98:0/3732664018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0074150c00 msgr2=0x7f005c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.898+0000 7f0071d9b640 1 --2- 172.21.15.98:0/3732664018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0074150c00 0x7f005c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.898+0000 7f0071d9b640 1 -- 172.21.15.98:0/3732664018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f005c004d00 msgr2=0x7f005c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.898+0000 7f0071d9b640 1 --2- 172.21.15.98:0/3732664018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f005c004d00 0x7f005c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.898+0000 7f0071d9b640 1 -- 172.21.15.98:0/3732664018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c003520 con 0x7f005c003f70 2023-11-28T17:41:17.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.898+0000 7f007259c640 1 --2- 172.21.15.98:0/3732664018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0074150c00 0x7f005c0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:17.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.898+0000 7f0072d9d640 1 --2- 172.21.15.98:0/3732664018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f005c004d00 0x7f005c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.899+0000 7f0071d9b640 1 --2- 172.21.15.98:0/3732664018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f005c003f70 0x7f005c0047c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0068001750 tx=0x7f005c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a91c41ef57eb16bd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.899+0000 7f007159a640 1 -- 172.21.15.98:0/3732664018 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00680052d0 con 0x7f005c003f70 2023-11-28T17:41:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.899+0000 7f007159a640 1 -- 172.21.15.98:0/3732664018 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006802e050 con 0x7f005c003f70 2023-11-28T17:41:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.899+0000 7f007159a640 1 -- 172.21.15.98:0/3732664018 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0068003830 con 0x7f005c003f70 2023-11-28T17:41:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.899+0000 7f0078a94640 1 -- 172.21.15.98:0/3732664018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f005c003f70 msgr2=0x7f005c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.899+0000 7f0078a94640 1 --2- 172.21.15.98:0/3732664018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f005c003f70 0x7f005c0047c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0068001750 tx=0x7f005c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.899+0000 7f0072d9d640 1 -- 172.21.15.98:0/3732664018 reap_dead start 2023-11-28T17:41:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.900+0000 7f0078a94640 1 -- 172.21.15.98:0/3732664018 shutdown_connections 2023-11-28T17:41:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.900+0000 7f0078a94640 1 -- 172.21.15.98:0/3732664018 >> 172.21.15.98:0/3732664018 conn(0x7f007406a4d0 msgr2=0x7f007406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.900+0000 7f0078a94640 1 -- 172.21.15.98:0/3732664018 shutdown_connections 2023-11-28T17:41:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.900+0000 7f0078a94640 1 -- 172.21.15.98:0/3732664018 wait complete. 2023-11-28T17:41:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.901+0000 7f0078a94640 1 Processor -- start 2023-11-28T17:41:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.901+0000 7f0078a94640 1 -- start start 2023-11-28T17:41:17.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.902+0000 7f0078a94640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f005c003f70 0x7f00740713e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.902+0000 7f0078a94640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f007414c490 0x7f00741ed200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.902+0000 7f0078a94640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00741ed740 0x7f0074146640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.902+0000 7f0078a94640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0074148a60 con 0x7f00741ed740 2023-11-28T17:41:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.902+0000 7f0078a94640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0074156510 con 0x7f005c003f70 2023-11-28T17:41:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.902+0000 7f0078a94640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00741483e0 con 0x7f007414c490 2023-11-28T17:41:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.903+0000 7f0072d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00741ed740 0x7f0074146640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.903+0000 7f0072d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00741ed740 0x7f0074146640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59252/0 (socket says 172.21.15.98:59252) 2023-11-28T17:41:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.903+0000 7f0072d9d640 1 -- 172.21.15.98:0/1406685030 learned_addr learned my addr 172.21.15.98:0/1406685030 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.903+0000 7f0071d9b640 1 --2- 172.21.15.98:0/1406685030 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f007414c490 0x7f00741ed200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.903+0000 7f007259c640 1 --2- 172.21.15.98:0/1406685030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f005c003f70 0x7f00740713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.903+0000 7f0072d9d640 1 -- 172.21.15.98:0/1406685030 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f007414c490 msgr2=0x7f00741ed200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.903+0000 7f0072d9d640 1 --2- 172.21.15.98:0/1406685030 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f007414c490 0x7f00741ed200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.903+0000 7f0072d9d640 1 -- 172.21.15.98:0/1406685030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f005c003f70 msgr2=0x7f00740713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.903+0000 7f0072d9d640 1 --2- 172.21.15.98:0/1406685030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f005c003f70 0x7f00740713e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.903+0000 7f0072d9d640 1 -- 172.21.15.98:0/1406685030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00741edbf0 con 0x7f00741ed740 2023-11-28T17:41:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.903+0000 7f007259c640 1 --2- 172.21.15.98:0/1406685030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f005c003f70 0x7f00740713e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.903+0000 7f0072d9d640 1 --2- 172.21.15.98:0/1406685030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00741ed740 0x7f0074146640 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f00740c8ce0 tx=0x7f006c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.904+0000 7f0062ffd640 1 -- 172.21.15.98:0/1406685030 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c004d60 con 0x7f00741ed740 2023-11-28T17:41:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.904+0000 7f0062ffd640 1 -- 172.21.15.98:0/1406685030 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006c00bb80 con 0x7f00741ed740 2023-11-28T17:41:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.904+0000 7f0078a94640 1 -- 172.21.15.98:0/1406685030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0074147e20 con 0x7f00741ed740 2023-11-28T17:41:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.904+0000 7f0062ffd640 1 -- 172.21.15.98:0/1406685030 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006c007110 con 0x7f00741ed740 2023-11-28T17:41:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.904+0000 7f0078a94640 1 -- 172.21.15.98:0/1406685030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0074155ec0 con 0x7f00741ed740 2023-11-28T17:41:17.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.905+0000 7f0078a94640 1 -- 172.21.15.98:0/1406685030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0074067bf0 con 0x7f00741ed740 2023-11-28T17:41:17.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.906+0000 7f0062ffd640 1 -- 172.21.15.98:0/1406685030 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f006c007270 con 0x7f00741ed740 2023-11-28T17:41:17.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.907+0000 7f0062ffd640 1 --2- 172.21.15.98:0/1406685030 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f005406b2d0 0x7f005406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:17.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.907+0000 7f0062ffd640 1 -- 172.21.15.98:0/1406685030 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f006c089fa0 con 0x7f00741ed740 2023-11-28T17:41:17.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.907+0000 7f007259c640 1 --2- 172.21.15.98:0/1406685030 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f005406b2d0 0x7f005406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:17.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.908+0000 7f007259c640 1 --2- 172.21.15.98:0/1406685030 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f005406b2d0 0x7f005406d790 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0064009aa0 tx=0x7f0064008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:17.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:17.911+0000 7f0062ffd640 1 -- 172.21.15.98:0/1406685030 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f006c054e00 con 0x7f00741ed740 2023-11-28T17:41:18.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.101+0000 7f0078a94640 1 -- 172.21.15.98:0/1406685030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0074155d00 con 0x7f00741ed740 2023-11-28T17:41:18.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.103+0000 7f0062ffd640 1 -- 172.21.15.98:0/1406685030 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16530 (secure 0 0 0) 0x7f006c0019b0 con 0x7f00741ed740 2023-11-28T17:41:18.106 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:18.106 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":21,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:41:16.691297+0000","last_up_change":"2023-11-28T17:41:09.556993+0000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:41:10.303280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:41:14.890871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:41:15.704103+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6810","nonce":2188704385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6813","nonce":2188704385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6819","nonce":2188704385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6816","nonce":2188704385}]},"public_addr":"172.21.15.98:6810/2188704385","cluster_addr":"172.21.15.98:6813/2188704385","heartbeat_back_addr":"172.21.15.98:6819/2188704385","heartbeat_front_addr":"172.21.15.98:6816/2188704385","state":["exists","up"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6801","nonce":3918226860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6803","nonce":3918226860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6807","nonce":3918226860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6805","nonce":3918226860}]},"public_addr":"172.21.15.98:6801/3918226860","cluster_addr":"172.21.15.98:6803/3918226860","heartbeat_back_addr":"172.21.15.98:6807/3918226860","heartbeat_front_addr":"172.21.15.98:6805/3918226860","state":["exists","up"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6812","nonce":2691532400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6815","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6818","nonce":2691532400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6823","nonce":2691532400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6821","nonce":2691532400}]},"public_addr":"172.21.15.98:6812/2691532400","cluster_addr":"172.21.15.98:6818/2691532400","heartbeat_back_addr":"172.21.15.98:6823/2691532400","heartbeat_front_addr":"172.21.15.98:6821/2691532400","state":["exists","up"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6825","nonce":1905487008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6827","nonce":1905487008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6831","nonce":1905487008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6829","nonce":1905487008}]},"public_addr":"172.21.15.98:6825/1905487008","cluster_addr":"172.21.15.98:6827/1905487008","heartbeat_back_addr":"172.21.15.98:6831/1905487008","heartbeat_front_addr":"172.21.15.98:6829/1905487008","state":["exists","up"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6801","nonce":2974677215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6803","nonce":2974677215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6807","nonce":2974677215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6805","nonce":2974677215}]},"public_addr":"172.21.15.113:6801/2974677215","cluster_addr":"172.21.15.113:6803/2974677215","heartbeat_back_addr":"172.21.15.113:6807/2974677215","heartbeat_front_addr":"172.21.15.113:6805/2974677215","state":["exists","up"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6817","nonce":2685000355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6821","nonce":2685000355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6829","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6831","nonce":2685000355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6823","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6826","nonce":2685000355}]},"public_addr":"172.21.15.113:6817/2685000355","cluster_addr":"172.21.15.113:6821/2685000355","heartbeat_back_addr":"172.21.15.113:6831/2685000355","heartbeat_front_addr":"172.21.15.113:6826/2685000355","state":["exists","up"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6809","nonce":2764300831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6811","nonce":2764300831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6815","nonce":2764300831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6813","nonce":2764300831}]},"public_addr":"172.21.15.113:6809/2764300831","cluster_addr":"172.21.15.113:6811/2764300831","heartbeat_back_addr":"172.21.15.113:6815/2764300831","heartbeat_front_addr":"172.21.15.113:6813/2764300831","state":["exists","up"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6819","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6820","nonce":1656248400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6824","nonce":1656248400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6830","nonce":1656248400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6825","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6827","nonce":1656248400}]},"public_addr":"172.21.15.113:6820/1656248400","cluster_addr":"172.21.15.113:6824/1656248400","heartbeat_back_addr":"172.21.15.113:6830/1656248400","heartbeat_front_addr":"172.21.15.113:6827/1656248400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.379549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.312196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.117494+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.388128+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.360608+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.411101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.355995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.390362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:41:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.106+0000 7f0078a94640 1 -- 172.21.15.98:0/1406685030 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f005406b2d0 msgr2=0x7f005406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.106+0000 7f0078a94640 1 --2- 172.21.15.98:0/1406685030 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f005406b2d0 0x7f005406d790 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0064009aa0 tx=0x7f0064008040 comp rx=0 tx=0).stop 2023-11-28T17:41:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.106+0000 7f0078a94640 1 -- 172.21.15.98:0/1406685030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00741ed740 msgr2=0x7f0074146640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.106+0000 7f0078a94640 1 --2- 172.21.15.98:0/1406685030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00741ed740 0x7f0074146640 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f00740c8ce0 tx=0x7f006c00b4a0 comp rx=0 tx=0).stop 2023-11-28T17:41:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.106+0000 7f0078a94640 1 -- 172.21.15.98:0/1406685030 shutdown_connections 2023-11-28T17:41:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.106+0000 7f0078a94640 1 --2- 172.21.15.98:0/1406685030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00741ed740 0x7f0074146640 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.106+0000 7f0078a94640 1 --2- 172.21.15.98:0/1406685030 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f007414c490 0x7f00741ed200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.106+0000 7f0078a94640 1 --2- 172.21.15.98:0/1406685030 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f005406b2d0 0x7f005406d790 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.106+0000 7f0078a94640 1 --2- 172.21.15.98:0/1406685030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f005c003f70 0x7f00740713e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.106+0000 7f0078a94640 1 -- 172.21.15.98:0/1406685030 >> 172.21.15.98:0/1406685030 conn(0x7f007406a4d0 msgr2=0x7f00740c7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:18.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.106+0000 7f0078a94640 1 -- 172.21.15.98:0/1406685030 shutdown_connections 2023-11-28T17:41:18.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.106+0000 7f0078a94640 1 -- 172.21.15.98:0/1406685030 wait complete. 2023-11-28T17:41:18.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T17:41:18.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.344+0000 7f82d19d5640 1 Processor -- start 2023-11-28T17:41:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.345+0000 7f82d19d5640 1 -- start start 2023-11-28T17:41:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.346+0000 7f82d19d5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f82cc1540d0 0x7f82cc15dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.346+0000 7f82d19d5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82cc06c060 con 0x7f82cc0cb070 2023-11-28T17:41:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.346+0000 7f82d19d5640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82cc06a660 con 0x7f82cc0ca280 2023-11-28T17:41:18.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.346+0000 7f82d19d5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82cc0cbdd0 con 0x7f82cc1540d0 2023-11-28T17:41:18.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.347+0000 7f82caffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f82cc1540d0 0x7f82cc15dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:18.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.347+0000 7f82caffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f82cc1540d0 0x7f82cc15dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59256/0 (socket says 172.21.15.98:59256) 2023-11-28T17:41:18.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.347+0000 7f82caffd640 1 -- 172.21.15.98:0/1910063306 learned_addr learned my addr 172.21.15.98:0/1910063306 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:18.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.347+0000 7f82caffd640 1 -- 172.21.15.98:0/1910063306 >> v1:172.21.15.113:6789/0 conn(0x7f82cc0ca280 legacy=0x7f82cc1645f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.347+0000 7f82caffd640 1 -- 172.21.15.98:0/1910063306 >> v1:172.21.15.98:6789/0 conn(0x7f82cc0cb070 legacy=0x7f82cc06b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.347+0000 7f82caffd640 1 -- 172.21.15.98:0/1910063306 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82cc15e1b0 con 0x7f82cc1540d0 2023-11-28T17:41:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.347+0000 7f82c9ffb640 1 -- 172.21.15.98:0/1910063306 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049061648 0 0) 0x7f82b4001110 con 0x7f82cc0cb070 2023-11-28T17:41:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.348+0000 7f82caffd640 1 --2- 172.21.15.98:0/1910063306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82cc1540d0 0x7f82cc15dc70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f82bc009e50 tx=0x7f82bc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df89f962389ee77 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.349+0000 7f82c9ffb640 1 -- 172.21.15.98:0/1910063306 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc02cb20 con 0x7f82cc1540d0 2023-11-28T17:41:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.349+0000 7f82c9ffb640 1 -- 172.21.15.98:0/1910063306 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82bc02cc80 con 0x7f82cc1540d0 2023-11-28T17:41:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.349+0000 7f82c9ffb640 1 -- 172.21.15.98:0/1910063306 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc0343f0 con 0x7f82cc1540d0 2023-11-28T17:41:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.349+0000 7f82d19d5640 1 -- 172.21.15.98:0/1910063306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82cc1540d0 msgr2=0x7f82cc15dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.349+0000 7f82d19d5640 1 --2- 172.21.15.98:0/1910063306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82cc1540d0 0x7f82cc15dc70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f82bc009e50 tx=0x7f82bc02bf80 comp rx=0 tx=0).stop 2023-11-28T17:41:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.349+0000 7f82d19d5640 1 -- 172.21.15.98:0/1910063306 shutdown_connections 2023-11-28T17:41:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.349+0000 7f82d19d5640 1 --2- 172.21.15.98:0/1910063306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82cc1540d0 0x7f82cc15dc70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.349+0000 7f82d19d5640 1 -- 172.21.15.98:0/1910063306 >> 172.21.15.98:0/1910063306 conn(0x7f82cc0cf490 msgr2=0x7f82cc0cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.349+0000 7f82d19d5640 1 -- 172.21.15.98:0/1910063306 shutdown_connections 2023-11-28T17:41:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.349+0000 7f82d19d5640 1 -- 172.21.15.98:0/1910063306 wait complete. 2023-11-28T17:41:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.351+0000 7f82d19d5640 1 Processor -- start 2023-11-28T17:41:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.351+0000 7f82d19d5640 1 -- start start 2023-11-28T17:41:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.352+0000 7f82d19d5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82cc0ca280 0x7f82cc14f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.352+0000 7f82d19d5640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82cc0cb070 0x7f82cc15cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.352+0000 7f82d19d5640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f82cc1540d0 0x7f82cc1552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.352+0000 7f82d19d5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc1576e0 con 0x7f82cc0ca280 2023-11-28T17:41:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.352+0000 7f82d19d5640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc1570f0 con 0x7f82cc0cb070 2023-11-28T17:41:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.352+0000 7f82d19d5640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc156810 con 0x7f82cc1540d0 2023-11-28T17:41:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.352+0000 7f82caffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82cc0ca280 0x7f82cc14f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.352+0000 7f82caffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82cc0ca280 0x7f82cc14f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59270/0 (socket says 172.21.15.98:59270) 2023-11-28T17:41:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.352+0000 7f82caffd640 1 -- 172.21.15.98:0/878236017 learned_addr learned my addr 172.21.15.98:0/878236017 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.353+0000 7f82caffd640 1 -- 172.21.15.98:0/878236017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f82cc1540d0 msgr2=0x7f82cc1552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.353+0000 7f82caffd640 1 --2- 172.21.15.98:0/878236017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f82cc1540d0 0x7f82cc1552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.353+0000 7f82caffd640 1 -- 172.21.15.98:0/878236017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82cc0cb070 msgr2=0x7f82cc15cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.353+0000 7f82caffd640 1 --2- 172.21.15.98:0/878236017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82cc0cb070 0x7f82cc15cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.353+0000 7f82caffd640 1 -- 172.21.15.98:0/878236017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc009b30 con 0x7f82cc0ca280 2023-11-28T17:41:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.353+0000 7f82caffd640 1 --2- 172.21.15.98:0/878236017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82cc0ca280 0x7f82cc14f470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f82bc009e50 tx=0x7f82bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.353+0000 7f82c3fff640 1 -- 172.21.15.98:0/878236017 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc009510 con 0x7f82cc0ca280 2023-11-28T17:41:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.353+0000 7f82c3fff640 1 -- 172.21.15.98:0/878236017 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82bc03e450 con 0x7f82cc0ca280 2023-11-28T17:41:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.353+0000 7f82c3fff640 1 -- 172.21.15.98:0/878236017 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc03e7e0 con 0x7f82cc0ca280 2023-11-28T17:41:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.353+0000 7f82d19d5640 1 -- 172.21.15.98:0/878236017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82cc15dca0 con 0x7f82cc0ca280 2023-11-28T17:41:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.354+0000 7f82d19d5640 1 -- 172.21.15.98:0/878236017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82cc1535d0 con 0x7f82cc0ca280 2023-11-28T17:41:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.355+0000 7f82d19d5640 1 -- 172.21.15.98:0/878236017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82cc074f60 con 0x7f82cc0ca280 2023-11-28T17:41:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.355+0000 7f82c3fff640 1 -- 172.21.15.98:0/878236017 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f82bc03e940 con 0x7f82cc0ca280 2023-11-28T17:41:18.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.357+0000 7f82c3fff640 1 --2- 172.21.15.98:0/878236017 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f82ac06b2a0 0x7f82ac06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:18.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.357+0000 7f82c3fff640 1 -- 172.21.15.98:0/878236017 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82bc0acb60 con 0x7f82cc0ca280 2023-11-28T17:41:18.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.357+0000 7f82ca7fc640 1 --2- 172.21.15.98:0/878236017 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f82ac06b2a0 0x7f82ac06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:18.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.358+0000 7f82ca7fc640 1 --2- 172.21.15.98:0/878236017 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f82ac06b2a0 0x7f82ac06d760 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f82cc1e3d40 tx=0x7f82b40077c0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:18.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.361+0000 7f82c3fff640 1 -- 172.21.15.98:0/878236017 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82bc077a60 con 0x7f82cc0ca280 2023-11-28T17:41:18.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:18.597+0000 7f82d19d5640 1 -- 172.21.15.98:0/878236017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f82cc0ca010 con 0x7f82cc0ca280 2023-11-28T17:41:19.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.259+0000 7f82c3fff640 1 -- 172.21.15.98:0/878236017 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7f82bc047e90 con 0x7f82cc0ca280 2023-11-28T17:41:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.263+0000 7f82d19d5640 1 -- 172.21.15.98:0/878236017 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f82ac06b2a0 msgr2=0x7f82ac06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.263+0000 7f82d19d5640 1 --2- 172.21.15.98:0/878236017 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f82ac06b2a0 0x7f82ac06d760 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f82cc1e3d40 tx=0x7f82b40077c0 comp rx=0 tx=0).stop 2023-11-28T17:41:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.263+0000 7f82d19d5640 1 -- 172.21.15.98:0/878236017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82cc0ca280 msgr2=0x7f82cc14f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.263+0000 7f82d19d5640 1 --2- 172.21.15.98:0/878236017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82cc0ca280 0x7f82cc14f470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f82bc009e50 tx=0x7f82bc004a40 comp rx=0 tx=0).stop 2023-11-28T17:41:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.263+0000 7f82d19d5640 1 -- 172.21.15.98:0/878236017 shutdown_connections 2023-11-28T17:41:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.263+0000 7f82d19d5640 1 --2- 172.21.15.98:0/878236017 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f82cc1540d0 0x7f82cc1552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.263+0000 7f82d19d5640 1 --2- 172.21.15.98:0/878236017 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f82ac06b2a0 0x7f82ac06d760 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.263+0000 7f82d19d5640 1 --2- 172.21.15.98:0/878236017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82cc0cb070 0x7f82cc15cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.263+0000 7f82d19d5640 1 --2- 172.21.15.98:0/878236017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f82cc0ca280 0x7f82cc14f470 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.263+0000 7f82d19d5640 1 -- 172.21.15.98:0/878236017 >> 172.21.15.98:0/878236017 conn(0x7f82cc0cf490 msgr2=0x7f82cc1518f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:19.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.263+0000 7f82d19d5640 1 -- 172.21.15.98:0/878236017 shutdown_connections 2023-11-28T17:41:19.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.263+0000 7f82d19d5640 1 -- 172.21.15.98:0/878236017 wait complete. 2023-11-28T17:41:19.287 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300} 2023-11-28T17:41:19.287 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-11-28T17:41:19.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.507+0000 7f468e89b640 1 Processor -- start 2023-11-28T17:41:19.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.507+0000 7f468e89b640 1 -- start start 2023-11-28T17:41:19.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.509+0000 7f468e89b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f46880c4d80 0x7f4688157a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:19.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.509+0000 7f468e89b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46880cbff0 con 0x7f4688071b60 2023-11-28T17:41:19.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.509+0000 7f468e89b640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4688070860 con 0x7f46880c49a0 2023-11-28T17:41:19.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.509+0000 7f468e89b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f468806e390 con 0x7f46880c4d80 2023-11-28T17:41:19.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.509+0000 7f468ce38640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f46880c4d80 0x7f4688157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:19.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.509+0000 7f46877fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4688071b60 0x7f4688160370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48182/0 (socket says 172.21.15.98:48182) 2023-11-28T17:41:19.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.509+0000 7f46877fe640 1 -- 172.21.15.98:0/630302699 learned_addr learned my addr 172.21.15.98:0/630302699 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:19.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.509+0000 7f468ce38640 1 -- 172.21.15.98:0/630302699 >> v1:172.21.15.113:6789/0 conn(0x7f46880c49a0 legacy=0x7f46881518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:19.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.509+0000 7f468ce38640 1 -- 172.21.15.98:0/630302699 >> v1:172.21.15.98:6789/0 conn(0x7f4688071b60 legacy=0x7f4688160370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:19.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.509+0000 7f468ce38640 1 -- 172.21.15.98:0/630302699 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4688070b40 con 0x7f46880c4d80 2023-11-28T17:41:19.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.511+0000 7f468ce38640 1 --2- 172.21.15.98:0/630302699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46880c4d80 0x7f4688157a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4680009d90 tx=0x7f468002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be8958615c07141 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:19.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.511+0000 7f4686ffd640 1 -- 172.21.15.98:0/630302699 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468002cc60 con 0x7f46880c4d80 2023-11-28T17:41:19.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.511+0000 7f4686ffd640 1 -- 172.21.15.98:0/630302699 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468002cdc0 con 0x7f46880c4d80 2023-11-28T17:41:19.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.511+0000 7f4686ffd640 1 -- 172.21.15.98:0/630302699 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680034660 con 0x7f46880c4d80 2023-11-28T17:41:19.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.511+0000 7f468e89b640 1 -- 172.21.15.98:0/630302699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46880c4d80 msgr2=0x7f4688157a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:19.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.511+0000 7f468e89b640 1 --2- 172.21.15.98:0/630302699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46880c4d80 0x7f4688157a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4680009d90 tx=0x7f468002beb0 comp rx=0 tx=0).stop 2023-11-28T17:41:19.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.511+0000 7f468e89b640 1 -- 172.21.15.98:0/630302699 shutdown_connections 2023-11-28T17:41:19.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.511+0000 7f468e89b640 1 --2- 172.21.15.98:0/630302699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46880c4d80 0x7f4688157a00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:19.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.511+0000 7f468e89b640 1 -- 172.21.15.98:0/630302699 >> 172.21.15.98:0/630302699 conn(0x7f468806c170 msgr2=0x7f468806c550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:19.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.512+0000 7f468e89b640 1 -- 172.21.15.98:0/630302699 shutdown_connections 2023-11-28T17:41:19.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.512+0000 7f468e89b640 1 -- 172.21.15.98:0/630302699 wait complete. 2023-11-28T17:41:19.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.513+0000 7f468e89b640 1 Processor -- start 2023-11-28T17:41:19.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.513+0000 7f468e89b640 1 -- start start 2023-11-28T17:41:19.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.514+0000 7f468e89b640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4688071b60 0x7f468814db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:19.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.514+0000 7f468e89b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880c49a0 0x7f46881f15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:19.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.514+0000 7f468e89b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46880c4d80 0x7f4688147480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:19.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.514+0000 7f468e89b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46881ee380 con 0x7f46880c4d80 2023-11-28T17:41:19.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.514+0000 7f468e89b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46881f1af0 con 0x7f46880c49a0 2023-11-28T17:41:19.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.514+0000 7f468e89b640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46881ed740 con 0x7f4688071b60 2023-11-28T17:41:19.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.515+0000 7f468ce38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46880c4d80 0x7f4688147480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:19.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.515+0000 7f468ce38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46880c4d80 0x7f4688147480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59284/0 (socket says 172.21.15.98:59284) 2023-11-28T17:41:19.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.515+0000 7f468ce38640 1 -- 172.21.15.98:0/2963139893 learned_addr learned my addr 172.21.15.98:0/2963139893 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:19.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.515+0000 7f46877fe640 1 --2- 172.21.15.98:0/2963139893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880c49a0 0x7f46881f15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:19.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.515+0000 7f468ce38640 1 -- 172.21.15.98:0/2963139893 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4688071b60 msgr2=0x7f468814db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:19.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.515+0000 7f4687fff640 1 --2- 172.21.15.98:0/2963139893 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4688071b60 0x7f468814db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:19.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.515+0000 7f468ce38640 1 --2- 172.21.15.98:0/2963139893 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4688071b60 0x7f468814db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:19.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.515+0000 7f468ce38640 1 -- 172.21.15.98:0/2963139893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880c49a0 msgr2=0x7f46881f15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:19.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.515+0000 7f468ce38640 1 --2- 172.21.15.98:0/2963139893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880c49a0 0x7f46881f15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:19.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.515+0000 7f468ce38640 1 -- 172.21.15.98:0/2963139893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4680009a70 con 0x7f46880c4d80 2023-11-28T17:41:19.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.515+0000 7f46877fe640 1 --2- 172.21.15.98:0/2963139893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880c49a0 0x7f46881f15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:19.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.515+0000 7f468ce38640 1 --2- 172.21.15.98:0/2963139893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46880c4d80 0x7f4688147480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4680037040 tx=0x7f4680004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:19.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.516+0000 7f4684ff9640 1 -- 172.21.15.98:0/2963139893 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680040450 con 0x7f46880c4d80 2023-11-28T17:41:19.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.516+0000 7f4684ff9640 1 -- 172.21.15.98:0/2963139893 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46800405d0 con 0x7f46880c4d80 2023-11-28T17:41:19.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.516+0000 7f4684ff9640 1 -- 172.21.15.98:0/2963139893 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680040980 con 0x7f46880c4d80 2023-11-28T17:41:19.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.516+0000 7f468e89b640 1 -- 172.21.15.98:0/2963139893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46881ed9e0 con 0x7f46880c4d80 2023-11-28T17:41:19.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.516+0000 7f468e89b640 1 -- 172.21.15.98:0/2963139893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46881edfa0 con 0x7f46880c4d80 2023-11-28T17:41:19.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.517+0000 7f468e89b640 1 -- 172.21.15.98:0/2963139893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46881ee4c0 con 0x7f46880c4d80 2023-11-28T17:41:19.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.518+0000 7f4684ff9640 1 -- 172.21.15.98:0/2963139893 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4680040ae0 con 0x7f46880c4d80 2023-11-28T17:41:19.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.519+0000 7f4684ff9640 1 --2- 172.21.15.98:0/2963139893 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f467406b2d0 0x7f467406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:19.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.519+0000 7f4684ff9640 1 -- 172.21.15.98:0/2963139893 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46800ac270 con 0x7f46880c4d80 2023-11-28T17:41:19.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.519+0000 7f4687fff640 1 --2- 172.21.15.98:0/2963139893 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f467406b2d0 0x7f467406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:19.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.520+0000 7f4687fff640 1 --2- 172.21.15.98:0/2963139893 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f467406b2d0 0x7f467406d790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4678003d90 tx=0x7f4678009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:19.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.523+0000 7f4684ff9640 1 -- 172.21.15.98:0/2963139893 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46881ee4c0 con 0x7f46880c4d80 2023-11-28T17:41:19.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:19.753+0000 7f468e89b640 1 -- 172.21.15.98:0/2963139893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f46881ee4c0 con 0x7f46880c4d80 2023-11-28T17:41:20.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.270+0000 7f4684ff9640 1 -- 172.21.15.98:0/2963139893 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f46881ee4c0 con 0x7f46880c4d80 2023-11-28T17:41:20.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.274+0000 7f468e89b640 1 -- 172.21.15.98:0/2963139893 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f467406b2d0 msgr2=0x7f467406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:20.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.274+0000 7f468e89b640 1 --2- 172.21.15.98:0/2963139893 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f467406b2d0 0x7f467406d790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4678003d90 tx=0x7f4678009040 comp rx=0 tx=0).stop 2023-11-28T17:41:20.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.274+0000 7f468e89b640 1 -- 172.21.15.98:0/2963139893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46880c4d80 msgr2=0x7f4688147480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:20.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.274+0000 7f468e89b640 1 --2- 172.21.15.98:0/2963139893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46880c4d80 0x7f4688147480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4680037040 tx=0x7f4680004a40 comp rx=0 tx=0).stop 2023-11-28T17:41:20.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.274+0000 7f468e89b640 1 -- 172.21.15.98:0/2963139893 shutdown_connections 2023-11-28T17:41:20.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.274+0000 7f468e89b640 1 --2- 172.21.15.98:0/2963139893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46880c4d80 0x7f4688147480 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:20.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.274+0000 7f468e89b640 1 --2- 172.21.15.98:0/2963139893 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f467406b2d0 0x7f467406d790 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:20.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.274+0000 7f468e89b640 1 --2- 172.21.15.98:0/2963139893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46880c49a0 0x7f46881f15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:20.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.274+0000 7f468e89b640 1 --2- 172.21.15.98:0/2963139893 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4688071b60 0x7f468814db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:20.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.274+0000 7f468e89b640 1 -- 172.21.15.98:0/2963139893 >> 172.21.15.98:0/2963139893 conn(0x7f468806c170 msgr2=0x7f46880ca4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:20.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.275+0000 7f468e89b640 1 -- 172.21.15.98:0/2963139893 shutdown_connections 2023-11-28T17:41:20.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.275+0000 7f468e89b640 1 -- 172.21.15.98:0/2963139893 wait complete. 2023-11-28T17:41:20.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T17:41:20.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.512+0000 7fd912b3e640 1 Processor -- start 2023-11-28T17:41:20.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.512+0000 7fd912b3e640 1 -- start start 2023-11-28T17:41:20.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.514+0000 7fd912b3e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd90c1541d0 0x7fd90c070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:20.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.514+0000 7fd912b3e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd90c0c3550 0x7fd90c070840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:20.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.514+0000 7fd912b3e640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c15a0b0 con 0x7fd90c0c7d60 2023-11-28T17:41:20.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.514+0000 7fd912b3e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90c15ad30 con 0x7fd90c0c3550 2023-11-28T17:41:20.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.514+0000 7fd912b3e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c159ea0 con 0x7fd90c1541d0 2023-11-28T17:41:20.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.514+0000 7fd9110db640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd90c0c3550 0x7fd90c070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:20.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.514+0000 7fd9110db640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd90c0c3550 0x7fd90c070840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49152/0 (socket says 172.21.15.98:49152) 2023-11-28T17:41:20.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.514+0000 7fd9110db640 1 -- 172.21.15.98:0/3154399212 learned_addr learned my addr 172.21.15.98:0/3154399212 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:20.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.515+0000 7fd9108da640 1 --2- 172.21.15.98:0/3154399212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd90c1541d0 0x7fd90c070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:20.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.515+0000 7fd9110db640 1 -- 172.21.15.98:0/3154399212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd90c1541d0 msgr2=0x7fd90c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:20.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.515+0000 7fd9110db640 1 --2- 172.21.15.98:0/3154399212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd90c1541d0 0x7fd90c070300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:20.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.515+0000 7fd9110db640 1 -- 172.21.15.98:0/3154399212 >> v1:172.21.15.113:6789/0 conn(0x7fd90c0c7d60 legacy=0x7fd90c0c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:20.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.515+0000 7fd9110db640 1 -- 172.21.15.98:0/3154399212 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c15a390 con 0x7fd90c0c3550 2023-11-28T17:41:20.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.516+0000 7fd9110db640 1 --2- 172.21.15.98:0/3154399212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c0c3550 0x7fd90c070840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd904009e50 tx=0x7fd90402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77891c80b73a0326 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:20.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.516+0000 7fd90b7fe640 1 -- 172.21.15.98:0/3154399212 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90402ca70 con 0x7fd90c0c3550 2023-11-28T17:41:20.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.517+0000 7fd90b7fe640 1 -- 172.21.15.98:0/3154399212 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90402cbd0 con 0x7fd90c0c3550 2023-11-28T17:41:20.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.517+0000 7fd90b7fe640 1 -- 172.21.15.98:0/3154399212 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9040343f0 con 0x7fd90c0c3550 2023-11-28T17:41:20.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.517+0000 7fd912b3e640 1 -- 172.21.15.98:0/3154399212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c0c3550 msgr2=0x7fd90c070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:20.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.517+0000 7fd912b3e640 1 --2- 172.21.15.98:0/3154399212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c0c3550 0x7fd90c070840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd904009e50 tx=0x7fd90402bed0 comp rx=0 tx=0).stop 2023-11-28T17:41:20.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.517+0000 7fd912b3e640 1 -- 172.21.15.98:0/3154399212 shutdown_connections 2023-11-28T17:41:20.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.517+0000 7fd912b3e640 1 --2- 172.21.15.98:0/3154399212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c0c3550 0x7fd90c070840 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:20.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.517+0000 7fd912b3e640 1 --2- 172.21.15.98:0/3154399212 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd90c1541d0 0x7fd90c070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:20.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.517+0000 7fd912b3e640 1 -- 172.21.15.98:0/3154399212 >> 172.21.15.98:0/3154399212 conn(0x7fd90c06e620 msgr2=0x7fd90c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:20.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.518+0000 7fd912b3e640 1 -- 172.21.15.98:0/3154399212 shutdown_connections 2023-11-28T17:41:20.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.518+0000 7fd912b3e640 1 -- 172.21.15.98:0/3154399212 wait complete. 2023-11-28T17:41:20.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.519+0000 7fd912b3e640 1 Processor -- start 2023-11-28T17:41:20.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.519+0000 7fd912b3e640 1 -- start start 2023-11-28T17:41:20.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.520+0000 7fd912b3e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c0c3550 0x7fd90c1533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:20.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.520+0000 7fd912b3e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd90c0c7d60 0x7fd90c0cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:20.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.520+0000 7fd912b3e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd90c1541d0 0x7fd90c151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:20.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.520+0000 7fd912b3e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c155f20 con 0x7fd90c0c3550 2023-11-28T17:41:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.520+0000 7fd912b3e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c151810 con 0x7fd90c1541d0 2023-11-28T17:41:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.520+0000 7fd912b3e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c153920 con 0x7fd90c0c7d60 2023-11-28T17:41:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.521+0000 7fd9108da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c0c3550 0x7fd90c1533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.521+0000 7fd9108da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c0c3550 0x7fd90c1533e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49160/0 (socket says 172.21.15.98:49160) 2023-11-28T17:41:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.521+0000 7fd9108da640 1 -- 172.21.15.98:0/770872977 learned_addr learned my addr 172.21.15.98:0/770872977 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.521+0000 7fd9110db640 1 --2- 172.21.15.98:0/770872977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd90c1541d0 0x7fd90c151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.521+0000 7fd90bfff640 1 --2- 172.21.15.98:0/770872977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd90c0c7d60 0x7fd90c0cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.521+0000 7fd9108da640 1 -- 172.21.15.98:0/770872977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd90c0c7d60 msgr2=0x7fd90c0cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.521+0000 7fd9108da640 1 --2- 172.21.15.98:0/770872977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd90c0c7d60 0x7fd90c0cc9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.521+0000 7fd9108da640 1 -- 172.21.15.98:0/770872977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd90c1541d0 msgr2=0x7fd90c151f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.521+0000 7fd9108da640 1 --2- 172.21.15.98:0/770872977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd90c1541d0 0x7fd90c151f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.521+0000 7fd9108da640 1 -- 172.21.15.98:0/770872977 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd904009b30 con 0x7fd90c0c3550 2023-11-28T17:41:20.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.521+0000 7fd9108da640 1 --2- 172.21.15.98:0/770872977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c0c3550 0x7fd90c1533e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd90000d620 tx=0x7fd90000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:20.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.522+0000 7fd9097fa640 1 -- 172.21.15.98:0/770872977 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90000b620 con 0x7fd90c0c3550 2023-11-28T17:41:20.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.522+0000 7fd9097fa640 1 -- 172.21.15.98:0/770872977 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90000b780 con 0x7fd90c0c3550 2023-11-28T17:41:20.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.522+0000 7fd912b3e640 1 -- 172.21.15.98:0/770872977 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90c1efb70 con 0x7fd90c0c3550 2023-11-28T17:41:20.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.522+0000 7fd9097fa640 1 -- 172.21.15.98:0/770872977 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90000bb30 con 0x7fd90c0c3550 2023-11-28T17:41:20.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.522+0000 7fd912b3e640 1 -- 172.21.15.98:0/770872977 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90c1f0130 con 0x7fd90c0c3550 2023-11-28T17:41:20.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.523+0000 7fd912b3e640 1 -- 172.21.15.98:0/770872977 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90c0b30f0 con 0x7fd90c0c3550 2023-11-28T17:41:20.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.524+0000 7fd9097fa640 1 -- 172.21.15.98:0/770872977 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fd90000bc90 con 0x7fd90c0c3550 2023-11-28T17:41:20.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.525+0000 7fd9097fa640 1 --2- 172.21.15.98:0/770872977 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd8f006b2a0 0x7fd8f006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:20.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.525+0000 7fd9097fa640 1 -- 172.21.15.98:0/770872977 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd90008a0c0 con 0x7fd90c0c3550 2023-11-28T17:41:20.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.525+0000 7fd90bfff640 1 --2- 172.21.15.98:0/770872977 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd8f006b2a0 0x7fd8f006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:20.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.526+0000 7fd90bfff640 1 --2- 172.21.15.98:0/770872977 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd8f006b2a0 0x7fd8f006d760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd90c158580 tx=0x7fd8f8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:20.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.530+0000 7fd9097fa640 1 -- 172.21.15.98:0/770872977 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd900054f10 con 0x7fd90c0c3550 2023-11-28T17:41:20.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:20.759+0000 7fd912b3e640 1 -- 172.21.15.98:0/770872977 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd90c074f60 con 0x7fd90c0c3550 2023-11-28T17:41:21.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.290+0000 7fd9097fa640 1 -- 172.21.15.98:0/770872977 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd900058530 con 0x7fd90c0c3550 2023-11-28T17:41:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.293+0000 7fd912b3e640 1 -- 172.21.15.98:0/770872977 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd8f006b2a0 msgr2=0x7fd8f006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.293+0000 7fd912b3e640 1 --2- 172.21.15.98:0/770872977 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd8f006b2a0 0x7fd8f006d760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd90c158580 tx=0x7fd8f8009040 comp rx=0 tx=0).stop 2023-11-28T17:41:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.293+0000 7fd912b3e640 1 -- 172.21.15.98:0/770872977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c0c3550 msgr2=0x7fd90c1533e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.293+0000 7fd912b3e640 1 --2- 172.21.15.98:0/770872977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c0c3550 0x7fd90c1533e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd90000d620 tx=0x7fd90000dae0 comp rx=0 tx=0).stop 2023-11-28T17:41:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.293+0000 7fd912b3e640 1 -- 172.21.15.98:0/770872977 shutdown_connections 2023-11-28T17:41:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.293+0000 7fd912b3e640 1 --2- 172.21.15.98:0/770872977 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd8f006b2a0 0x7fd8f006d760 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.293+0000 7fd912b3e640 1 --2- 172.21.15.98:0/770872977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd90c1541d0 0x7fd90c151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.293+0000 7fd912b3e640 1 --2- 172.21.15.98:0/770872977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd90c0c7d60 0x7fd90c0cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:21.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.293+0000 7fd912b3e640 1 --2- 172.21.15.98:0/770872977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c0c3550 0x7fd90c1533e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:21.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.293+0000 7fd912b3e640 1 -- 172.21.15.98:0/770872977 >> 172.21.15.98:0/770872977 conn(0x7fd90c06e620 msgr2=0x7fd90c157ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:21.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.293+0000 7fd912b3e640 1 -- 172.21.15.98:0/770872977 shutdown_connections 2023-11-28T17:41:21.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.293+0000 7fd912b3e640 1 -- 172.21.15.98:0/770872977 wait complete. 2023-11-28T17:41:21.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-11-28T17:41:21.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.530+0000 7f32afabb640 1 Processor -- start 2023-11-28T17:41:21.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.530+0000 7f32afabb640 1 -- start start 2023-11-28T17:41:21.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.532+0000 7f32afabb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f32a8071b50 0x7f32a815e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:21.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.532+0000 7f32afabb640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32a80c5a60 0x7f32a80c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:21.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.532+0000 7f32afabb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a80c6630 con 0x7f32a8071770 2023-11-28T17:41:21.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.532+0000 7f32afabb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a80c7000 con 0x7f32a8071b50 2023-11-28T17:41:21.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.532+0000 7f32afabb640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a806e710 con 0x7f32a80c5a60 2023-11-28T17:41:21.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.532+0000 7f32ad056640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f32a8071b50 0x7f32a815e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:21.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.532+0000 7f32ad056640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f32a8071b50 0x7f32a815e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49174/0 (socket says 172.21.15.98:49174) 2023-11-28T17:41:21.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.532+0000 7f32ad056640 1 -- 172.21.15.98:0/1354978519 learned_addr learned my addr 172.21.15.98:0/1354978519 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:21.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.533+0000 7f32ad056640 1 -- 172.21.15.98:0/1354978519 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32a80c5a60 msgr2=0x7f32a80c6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:21.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.533+0000 7f32ad056640 1 --2- 172.21.15.98:0/1354978519 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32a80c5a60 0x7f32a80c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:21.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.533+0000 7f32ad056640 1 -- 172.21.15.98:0/1354978519 >> v1:172.21.15.98:6789/0 conn(0x7f32a8071770 legacy=0x7f32a8157fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:21.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.533+0000 7f32ad056640 1 -- 172.21.15.98:0/1354978519 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a806de70 con 0x7f32a8071b50 2023-11-28T17:41:21.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.534+0000 7f32ad056640 1 --2- 172.21.15.98:0/1354978519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8071b50 0x7f32a815e110 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3298009870 tx=0x7f329802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8316ca708e6adfe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:21.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.534+0000 7f32ac855640 1 -- 172.21.15.98:0/1354978519 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298031640 con 0x7f32a8071b50 2023-11-28T17:41:21.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.534+0000 7f32ac855640 1 -- 172.21.15.98:0/1354978519 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32980317a0 con 0x7f32a8071b50 2023-11-28T17:41:21.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.534+0000 7f32ac855640 1 -- 172.21.15.98:0/1354978519 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298031b30 con 0x7f32a8071b50 2023-11-28T17:41:21.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.535+0000 7f32afabb640 1 -- 172.21.15.98:0/1354978519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8071b50 msgr2=0x7f32a815e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:21.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.535+0000 7f32afabb640 1 --2- 172.21.15.98:0/1354978519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8071b50 0x7f32a815e110 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3298009870 tx=0x7f329802b8f0 comp rx=0 tx=0).stop 2023-11-28T17:41:21.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.535+0000 7f32afabb640 1 -- 172.21.15.98:0/1354978519 shutdown_connections 2023-11-28T17:41:21.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.535+0000 7f32afabb640 1 --2- 172.21.15.98:0/1354978519 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32a80c5a60 0x7f32a80c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:21.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.535+0000 7f32afabb640 1 --2- 172.21.15.98:0/1354978519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8071b50 0x7f32a815e110 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:21.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.535+0000 7f32afabb640 1 -- 172.21.15.98:0/1354978519 >> 172.21.15.98:0/1354978519 conn(0x7f32a806c0f0 msgr2=0x7f32a806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:21.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.535+0000 7f32afabb640 1 -- 172.21.15.98:0/1354978519 shutdown_connections 2023-11-28T17:41:21.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.535+0000 7f32afabb640 1 -- 172.21.15.98:0/1354978519 wait complete. 2023-11-28T17:41:21.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.537+0000 7f32afabb640 1 Processor -- start 2023-11-28T17:41:21.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.537+0000 7f32afabb640 1 -- start start 2023-11-28T17:41:21.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.538+0000 7f32afabb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8071770 0x7f32a8152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:21.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.538+0000 7f32afabb640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32a8071b50 0x7f32a80ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:21.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.538+0000 7f32afabb640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32a80c5a60 0x7f32a81f7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:21.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.538+0000 7f32afabb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a81e43d0 con 0x7f32a8071770 2023-11-28T17:41:21.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.538+0000 7f32afabb640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a80c98d0 con 0x7f32a8071b50 2023-11-28T17:41:21.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.538+0000 7f32afabb640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a81f8470 con 0x7f32a80c5a60 2023-11-28T17:41:21.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.538+0000 7f32ad857640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8071770 0x7f32a8152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:21.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.538+0000 7f32ad857640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8071770 0x7f32a8152be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49176/0 (socket says 172.21.15.98:49176) 2023-11-28T17:41:21.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.538+0000 7f32ad857640 1 -- 172.21.15.98:0/847157310 learned_addr learned my addr 172.21.15.98:0/847157310 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:21.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.538+0000 7f32ae058640 1 --2- 172.21.15.98:0/847157310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32a80c5a60 0x7f32a81f7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:21.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.538+0000 7f32ad857640 1 -- 172.21.15.98:0/847157310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32a80c5a60 msgr2=0x7f32a81f7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:21.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.538+0000 7f32ad056640 1 --2- 172.21.15.98:0/847157310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32a8071b50 0x7f32a80ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:21.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.538+0000 7f32ad857640 1 --2- 172.21.15.98:0/847157310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32a80c5a60 0x7f32a81f7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:21.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.538+0000 7f32ad857640 1 -- 172.21.15.98:0/847157310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32a8071b50 msgr2=0x7f32a80ca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:21.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.538+0000 7f32ad857640 1 --2- 172.21.15.98:0/847157310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32a8071b50 0x7f32a80ca320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:21.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.539+0000 7f32ad857640 1 -- 172.21.15.98:0/847157310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32980094d0 con 0x7f32a8071770 2023-11-28T17:41:21.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.539+0000 7f32ad857640 1 --2- 172.21.15.98:0/847157310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8071770 0x7f32a8152be0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f32a400c4d0 tx=0x7f32a400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:21.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.539+0000 7f329e7fc640 1 -- 172.21.15.98:0/847157310 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a40052d0 con 0x7f32a8071770 2023-11-28T17:41:21.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.539+0000 7f329e7fc640 1 -- 172.21.15.98:0/847157310 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a4005430 con 0x7f32a8071770 2023-11-28T17:41:21.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.539+0000 7f329e7fc640 1 -- 172.21.15.98:0/847157310 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a4012850 con 0x7f32a8071770 2023-11-28T17:41:21.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.539+0000 7f32afabb640 1 -- 172.21.15.98:0/847157310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a8151970 con 0x7f32a8071770 2023-11-28T17:41:21.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.539+0000 7f32afabb640 1 -- 172.21.15.98:0/847157310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a80c9b20 con 0x7f32a8071770 2023-11-28T17:41:21.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.540+0000 7f32afabb640 1 -- 172.21.15.98:0/847157310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a80c2310 con 0x7f32a8071770 2023-11-28T17:41:21.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.541+0000 7f329e7fc640 1 -- 172.21.15.98:0/847157310 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f32a40129b0 con 0x7f32a8071770 2023-11-28T17:41:21.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.542+0000 7f329e7fc640 1 --2- 172.21.15.98:0/847157310 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f328c06b2a0 0x7f328c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:21.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.542+0000 7f329e7fc640 1 -- 172.21.15.98:0/847157310 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32a4089780 con 0x7f32a8071770 2023-11-28T17:41:21.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.542+0000 7f32ad056640 1 --2- 172.21.15.98:0/847157310 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f328c06b2a0 0x7f328c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:21.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.543+0000 7f32ad056640 1 --2- 172.21.15.98:0/847157310 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f328c06b2a0 0x7f328c06d760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3298036000 tx=0x7f329803c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:21.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.547+0000 7f329e7fc640 1 -- 172.21.15.98:0/847157310 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f32a4054560 con 0x7f32a8071770 2023-11-28T17:41:21.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:21.775+0000 7f32afabb640 1 -- 172.21.15.98:0/847157310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f32a8074f60 con 0x7f32a8071770 2023-11-28T17:41:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.302+0000 7f329e7fc640 1 -- 172.21.15.98:0/847157310 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) v1 ==== 115+0+0 (secure 0 0 0) 0x7f32a4057b80 con 0x7f32a8071770 2023-11-28T17:41:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.305+0000 7f32afabb640 1 -- 172.21.15.98:0/847157310 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f328c06b2a0 msgr2=0x7f328c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.305+0000 7f32afabb640 1 --2- 172.21.15.98:0/847157310 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f328c06b2a0 0x7f328c06d760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3298036000 tx=0x7f329803c040 comp rx=0 tx=0).stop 2023-11-28T17:41:22.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.305+0000 7f32afabb640 1 -- 172.21.15.98:0/847157310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8071770 msgr2=0x7f32a8152be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:22.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.305+0000 7f32afabb640 1 --2- 172.21.15.98:0/847157310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8071770 0x7f32a8152be0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f32a400c4d0 tx=0x7f32a400c990 comp rx=0 tx=0).stop 2023-11-28T17:41:22.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.305+0000 7f32afabb640 1 -- 172.21.15.98:0/847157310 shutdown_connections 2023-11-28T17:41:22.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.305+0000 7f32afabb640 1 --2- 172.21.15.98:0/847157310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f32a80c5a60 0x7f32a81f7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.305+0000 7f32afabb640 1 --2- 172.21.15.98:0/847157310 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f328c06b2a0 0x7f328c06d760 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.306+0000 7f32afabb640 1 --2- 172.21.15.98:0/847157310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32a8071b50 0x7f32a80ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.306+0000 7f32afabb640 1 --2- 172.21.15.98:0/847157310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32a8071770 0x7f32a8152be0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.306+0000 7f32afabb640 1 -- 172.21.15.98:0/847157310 >> 172.21.15.98:0/847157310 conn(0x7f32a806c0f0 msgr2=0x7f32a815e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:22.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.306+0000 7f32afabb640 1 -- 172.21.15.98:0/847157310 shutdown_connections 2023-11-28T17:41:22.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.306+0000 7f32afabb640 1 -- 172.21.15.98:0/847157310 wait complete. 2023-11-28T17:41:22.327 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T17:41:22.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:41:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.516+0000 7fad27f68640 1 Processor -- start 2023-11-28T17:41:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.516+0000 7fad27f68640 1 -- start start 2023-11-28T17:41:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.517+0000 7fad27f68640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad20154040 0x7fad2015dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.517+0000 7fad27f68640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fad200c9420 0x7fad2015a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.517+0000 7fad27f68640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad20159c50 con 0x7fad200c9000 2023-11-28T17:41:22.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.517+0000 7fad27f68640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad200c2ad0 con 0x7fad20154040 2023-11-28T17:41:22.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.518+0000 7fad27f68640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad20154420 con 0x7fad200c9420 2023-11-28T17:41:22.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.518+0000 7fad25d04640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad20154040 0x7fad2015dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:22.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.518+0000 7fad25503640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fad200c9000 0x7fad200c7ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37454/0 (socket says 172.21.15.98:37454) 2023-11-28T17:41:22.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.518+0000 7fad25503640 1 -- 172.21.15.98:0/3025815861 learned_addr learned my addr 172.21.15.98:0/3025815861 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:22.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.518+0000 7fad25d04640 1 -- 172.21.15.98:0/3025815861 >> v2:172.21.15.113:3300/0 conn(0x7fad200c9420 msgr2=0x7fad2015a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:22.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.518+0000 7fad25d04640 1 --2- 172.21.15.98:0/3025815861 >> v2:172.21.15.113:3300/0 conn(0x7fad200c9420 0x7fad2015a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.518+0000 7fad25d04640 1 -- 172.21.15.98:0/3025815861 >> v1:172.21.15.98:6789/0 conn(0x7fad200c9000 legacy=0x7fad200c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:22.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.518+0000 7fad24d02640 1 -- 172.21.15.98:0/3025815861 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057106950 0 0) 0x7fad20159c50 con 0x7fad200c9000 2023-11-28T17:41:22.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.518+0000 7fad25d04640 1 -- 172.21.15.98:0/3025815861 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad200cab90 con 0x7fad20154040 2023-11-28T17:41:22.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.519+0000 7fad25d04640 1 --2- 172.21.15.98:0/3025815861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad20154040 0x7fad2015dd70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fad200c57f0 tx=0x7fad1c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a4c030b1fbc813 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:22.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.520+0000 7fad24d02640 1 -- 172.21.15.98:0/3025815861 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c030610 con 0x7fad20154040 2023-11-28T17:41:22.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.520+0000 7fad24d02640 1 -- 172.21.15.98:0/3025815861 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1c030770 con 0x7fad20154040 2023-11-28T17:41:22.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.520+0000 7fad24d02640 1 -- 172.21.15.98:0/3025815861 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c030b00 con 0x7fad20154040 2023-11-28T17:41:22.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.520+0000 7fad27f68640 1 -- 172.21.15.98:0/3025815861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad20154040 msgr2=0x7fad2015dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:22.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.520+0000 7fad27f68640 1 --2- 172.21.15.98:0/3025815861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad20154040 0x7fad2015dd70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fad200c57f0 tx=0x7fad1c02b960 comp rx=0 tx=0).stop 2023-11-28T17:41:22.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.520+0000 7fad27f68640 1 -- 172.21.15.98:0/3025815861 shutdown_connections 2023-11-28T17:41:22.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.520+0000 7fad27f68640 1 --2- 172.21.15.98:0/3025815861 >> v2:172.21.15.113:3300/0 conn(0x7fad200c9420 0x7fad2015a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.520+0000 7fad27f68640 1 --2- 172.21.15.98:0/3025815861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad20154040 0x7fad2015dd70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.520+0000 7fad27f68640 1 -- 172.21.15.98:0/3025815861 >> 172.21.15.98:0/3025815861 conn(0x7fad2006a4d0 msgr2=0x7fad2006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:22.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.520+0000 7fad27f68640 1 -- 172.21.15.98:0/3025815861 shutdown_connections 2023-11-28T17:41:22.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.521+0000 7fad27f68640 1 -- 172.21.15.98:0/3025815861 wait complete. 2023-11-28T17:41:22.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.522+0000 7fad27f68640 1 Processor -- start 2023-11-28T17:41:22.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.522+0000 7fad27f68640 1 -- start start 2023-11-28T17:41:22.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.523+0000 7fad27f68640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad200c9000 0x7fad20156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:22.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.523+0000 7fad27f68640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad200c9420 0x7fad201555e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:22.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.523+0000 7fad27f68640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad20154040 0x7fad201ef570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:22.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.523+0000 7fad27f68640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad20157340 con 0x7fad200c9420 2023-11-28T17:41:22.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.523+0000 7fad27f68640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad200c4e00 con 0x7fad20154040 2023-11-28T17:41:22.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.523+0000 7fad27f68640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad200c4ad0 con 0x7fad200c9000 2023-11-28T17:41:22.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.524+0000 7fad25503640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad200c9420 0x7fad201555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:22.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.524+0000 7fad25503640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad200c9420 0x7fad201555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49204/0 (socket says 172.21.15.98:49204) 2023-11-28T17:41:22.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.524+0000 7fad25503640 1 -- 172.21.15.98:0/1141013008 learned_addr learned my addr 172.21.15.98:0/1141013008 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:22.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.524+0000 7fad25d04640 1 --2- 172.21.15.98:0/1141013008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad200c9000 0x7fad20156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:22.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.524+0000 7fad26505640 1 --2- 172.21.15.98:0/1141013008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad20154040 0x7fad201ef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:22.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.524+0000 7fad25503640 1 -- 172.21.15.98:0/1141013008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad200c9000 msgr2=0x7fad20156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:22.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.524+0000 7fad25503640 1 --2- 172.21.15.98:0/1141013008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad200c9000 0x7fad20156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.524+0000 7fad25503640 1 -- 172.21.15.98:0/1141013008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad20154040 msgr2=0x7fad201ef570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:22.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.524+0000 7fad25503640 1 --2- 172.21.15.98:0/1141013008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad20154040 0x7fad201ef570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.524+0000 7fad25503640 1 -- 172.21.15.98:0/1141013008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c009400 con 0x7fad200c9420 2023-11-28T17:41:22.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.524+0000 7fad25503640 1 --2- 172.21.15.98:0/1141013008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad200c9420 0x7fad201555e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fad1000c4d0 tx=0x7fad1000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:22.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.525+0000 7fad167fc640 1 -- 172.21.15.98:0/1141013008 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad100074e0 con 0x7fad200c9420 2023-11-28T17:41:22.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.525+0000 7fad167fc640 1 -- 172.21.15.98:0/1141013008 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad10007660 con 0x7fad200c9420 2023-11-28T17:41:22.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.525+0000 7fad167fc640 1 -- 172.21.15.98:0/1141013008 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad10004a20 con 0x7fad200c9420 2023-11-28T17:41:22.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.525+0000 7fad27f68640 1 -- 172.21.15.98:0/1141013008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad20155b20 con 0x7fad200c9420 2023-11-28T17:41:22.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.525+0000 7fad27f68640 1 -- 172.21.15.98:0/1141013008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad200c3c50 con 0x7fad200c9420 2023-11-28T17:41:22.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.526+0000 7fad27f68640 1 -- 172.21.15.98:0/1141013008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad20067bf0 con 0x7fad200c9420 2023-11-28T17:41:22.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.527+0000 7fad167fc640 1 -- 172.21.15.98:0/1141013008 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fad10004b80 con 0x7fad200c9420 2023-11-28T17:41:22.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.528+0000 7fad167fc640 1 --2- 172.21.15.98:0/1141013008 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fad0406b2d0 0x7fad0406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:22.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.528+0000 7fad167fc640 1 -- 172.21.15.98:0/1141013008 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad10088cb0 con 0x7fad200c9420 2023-11-28T17:41:22.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.529+0000 7fad25d04640 1 --2- 172.21.15.98:0/1141013008 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fad0406b2d0 0x7fad0406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:22.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.529+0000 7fad25d04640 1 --2- 172.21.15.98:0/1141013008 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fad0406b2d0 0x7fad0406d790 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fad1c036000 tx=0x7fad1c03b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:22.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.533+0000 7fad167fc640 1 -- 172.21.15.98:0/1141013008 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad10053bb0 con 0x7fad200c9420 2023-11-28T17:41:22.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.760+0000 7fad27f68640 1 -- 172.21.15.98:0/1141013008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad200c4f70 con 0x7fad200c9420 2023-11-28T17:41:22.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.762+0000 7fad167fc640 1 -- 172.21.15.98:0/1141013008 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6513 (secure 0 0 0) 0x7fad10018020 con 0x7fad200c9420 2023-11-28T17:41:22.764 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:22.764 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2429440814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2429440814},{"type":"v1","addr":"172.21.15.98:6836","nonce":2429440814}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4269,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/1057450253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1057450253},{"type":"v1","addr":"172.21.15.113:6835","nonce":1057450253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1925178583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1925178583},{"type":"v1","addr":"172.21.15.98:6837","nonce":1925178583}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/1732613907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":1732613907},{"type":"v1","addr":"172.21.15.113:6837","nonce":1732613907}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:41:16.691765+0000","modified":"2023-11-28T17:41:22.306810+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1],"up":{"mds_0":4291,"mds_1":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"e","rank":1,"incarnation":9,"state":"up:creating","state_seq":1,"addr":"172.21.15.98:6839/3585869153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3585869153},{"type":"v1","addr":"172.21.15.98:6839","nonce":3585869153}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.113:6834/2183585971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2183585971},{"type":"v1","addr":"172.21.15.113:6834","nonce":2183585971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T17:41:22.764 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 9 2023-11-28T17:41:22.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.764+0000 7fad27f68640 1 -- 172.21.15.98:0/1141013008 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fad0406b2d0 msgr2=0x7fad0406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:22.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.764+0000 7fad27f68640 1 --2- 172.21.15.98:0/1141013008 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fad0406b2d0 0x7fad0406d790 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fad1c036000 tx=0x7fad1c03b040 comp rx=0 tx=0).stop 2023-11-28T17:41:22.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.764+0000 7fad27f68640 1 -- 172.21.15.98:0/1141013008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad200c9420 msgr2=0x7fad201555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:22.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.764+0000 7fad27f68640 1 --2- 172.21.15.98:0/1141013008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad200c9420 0x7fad201555e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fad1000c4d0 tx=0x7fad1000c990 comp rx=0 tx=0).stop 2023-11-28T17:41:22.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.764+0000 7fad27f68640 1 -- 172.21.15.98:0/1141013008 shutdown_connections 2023-11-28T17:41:22.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.764+0000 7fad27f68640 1 --2- 172.21.15.98:0/1141013008 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fad0406b2d0 0x7fad0406d790 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.764+0000 7fad27f68640 1 --2- 172.21.15.98:0/1141013008 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad20154040 0x7fad201ef570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.764+0000 7fad27f68640 1 --2- 172.21.15.98:0/1141013008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad200c9420 0x7fad201555e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.764+0000 7fad27f68640 1 --2- 172.21.15.98:0/1141013008 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fad200c9000 0x7fad20156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.764+0000 7fad27f68640 1 -- 172.21.15.98:0/1141013008 >> 172.21.15.98:0/1141013008 conn(0x7fad2006a4d0 msgr2=0x7fad20152c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:22.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.765+0000 7fad27f68640 1 -- 172.21.15.98:0/1141013008 shutdown_connections 2023-11-28T17:41:22.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.765+0000 7fad27f68640 1 -- 172.21.15.98:0/1141013008 wait complete. 2023-11-28T17:41:22.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:41:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.981+0000 7fbb06071640 1 Processor -- start 2023-11-28T17:41:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.981+0000 7fbb06071640 1 -- start start 2023-11-28T17:41:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.982+0000 7fbb06071640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbb000c0df0 0x7fbb001673c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.982+0000 7fbb06071640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb0006e5d0 0x7fbb000c5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.982+0000 7fbb06071640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0006e140 con 0x7fbb000c0a10 2023-11-28T17:41:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.982+0000 7fbb06071640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb000c5e40 con 0x7fbb000c0df0 2023-11-28T17:41:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.982+0000 7fbb06071640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000bff20 con 0x7fbb0006e5d0 2023-11-28T17:41:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.983+0000 7fbb0486e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbb000c0df0 0x7fbb001673c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:22.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.983+0000 7fbb0486e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbb000c0df0 0x7fbb001673c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49218/0 (socket says 172.21.15.98:49218) 2023-11-28T17:41:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.983+0000 7fbb0486e640 1 -- 172.21.15.98:0/525219915 learned_addr learned my addr 172.21.15.98:0/525219915 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.983+0000 7fbb05870640 1 --2- 172.21.15.98:0/525219915 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb0006e5d0 0x7fbb000c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.983+0000 7fbb0506f640 1 --1- 172.21.15.98:0/525219915 >> v1:172.21.15.98:6789/0 conn(0x7fbb000c0a10 0x7fbb000c68f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37458/0 (socket says 172.21.15.98:37458) 2023-11-28T17:41:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.985+0000 7fbb0486e640 1 -- 172.21.15.98:0/525219915 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb0006e5d0 msgr2=0x7fbb000c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.985+0000 7fbb0486e640 1 --2- 172.21.15.98:0/525219915 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb0006e5d0 0x7fbb000c5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.985+0000 7fbb0486e640 1 -- 172.21.15.98:0/525219915 >> v1:172.21.15.98:6789/0 conn(0x7fbb000c0a10 legacy=0x7fbb000c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.985+0000 7fbb0486e640 1 -- 172.21.15.98:0/525219915 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb000c0200 con 0x7fbb000c0df0 2023-11-28T17:41:22.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.985+0000 7fbb05870640 1 --2- 172.21.15.98:0/525219915 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb0006e5d0 0x7fbb000c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:41:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.986+0000 7fbb0486e640 1 --2- 172.21.15.98:0/525219915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb000c0df0 0x7fbb001673c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbaf00088f0 tx=0x7fbaf0008230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef5d681ffd4c40ea server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.986+0000 7fbaf7fff640 1 -- 172.21.15.98:0/525219915 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf002d900 con 0x7fbb000c0df0 2023-11-28T17:41:22.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.986+0000 7fbaf7fff640 1 -- 172.21.15.98:0/525219915 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf002da60 con 0x7fbb000c0df0 2023-11-28T17:41:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.986+0000 7fbaf7fff640 1 -- 172.21.15.98:0/525219915 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf002ddf0 con 0x7fbb000c0df0 2023-11-28T17:41:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.987+0000 7fbb06071640 1 -- 172.21.15.98:0/525219915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb000c0df0 msgr2=0x7fbb001673c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.987+0000 7fbb06071640 1 --2- 172.21.15.98:0/525219915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb000c0df0 0x7fbb001673c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbaf00088f0 tx=0x7fbaf0008230 comp rx=0 tx=0).stop 2023-11-28T17:41:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.987+0000 7fbb06071640 1 -- 172.21.15.98:0/525219915 shutdown_connections 2023-11-28T17:41:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.987+0000 7fbb06071640 1 --2- 172.21.15.98:0/525219915 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb0006e5d0 0x7fbb000c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.987+0000 7fbb06071640 1 --2- 172.21.15.98:0/525219915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb000c0df0 0x7fbb001673c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.987+0000 7fbb06071640 1 -- 172.21.15.98:0/525219915 >> 172.21.15.98:0/525219915 conn(0x7fbb00066230 msgr2=0x7fbb0006a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.987+0000 7fbb06071640 1 -- 172.21.15.98:0/525219915 shutdown_connections 2023-11-28T17:41:22.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.987+0000 7fbb06071640 1 -- 172.21.15.98:0/525219915 wait complete. 2023-11-28T17:41:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.989+0000 7fbb06071640 1 Processor -- start 2023-11-28T17:41:22.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.989+0000 7fbb06071640 1 -- start start 2023-11-28T17:41:22.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.990+0000 7fbb06071640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb0006e5d0 0x7fbb000c27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:22.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.990+0000 7fbb06071640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb000c0a10 0x7fbb000caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:22.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.990+0000 7fbb06071640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb000c0df0 0x7fbb0015cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:22.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.990+0000 7fbb06071640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000c9690 con 0x7fbb0006e5d0 2023-11-28T17:41:22.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.990+0000 7fbb06071640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb000c9f30 con 0x7fbb000c0df0 2023-11-28T17:41:22.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.990+0000 7fbb06071640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb001fd010 con 0x7fbb000c0a10 2023-11-28T17:41:22.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.990+0000 7fbb0506f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb0006e5d0 0x7fbb000c27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:22.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.990+0000 7fbb0506f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb0006e5d0 0x7fbb000c27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49232/0 (socket says 172.21.15.98:49232) 2023-11-28T17:41:22.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.990+0000 7fbb0506f640 1 -- 172.21.15.98:0/1322070617 learned_addr learned my addr 172.21.15.98:0/1322070617 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.991+0000 7fbb0486e640 1 --2- 172.21.15.98:0/1322070617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb000c0a10 0x7fbb000caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.991+0000 7fbb0506f640 1 -- 172.21.15.98:0/1322070617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb000c0a10 msgr2=0x7fbb000caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.991+0000 7fbb0506f640 1 --2- 172.21.15.98:0/1322070617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb000c0a10 0x7fbb000caa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.991+0000 7fbb05870640 1 --2- 172.21.15.98:0/1322070617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb000c0df0 0x7fbb0015cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.991+0000 7fbb0506f640 1 -- 172.21.15.98:0/1322070617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb000c0df0 msgr2=0x7fbb0015cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.991+0000 7fbb0506f640 1 --2- 172.21.15.98:0/1322070617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb000c0df0 0x7fbb0015cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.991+0000 7fbb0506f640 1 -- 172.21.15.98:0/1322070617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf0008550 con 0x7fbb0006e5d0 2023-11-28T17:41:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.991+0000 7fbb0506f640 1 --2- 172.21.15.98:0/1322070617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb0006e5d0 0x7fbb000c27f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbafc010cb0 tx=0x7fbafc00e430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.991+0000 7fbaf5ffb640 1 -- 172.21.15.98:0/1322070617 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc00b070 con 0x7fbb0006e5d0 2023-11-28T17:41:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.991+0000 7fbaf5ffb640 1 -- 172.21.15.98:0/1322070617 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbafc00b1d0 con 0x7fbb0006e5d0 2023-11-28T17:41:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.991+0000 7fbb06071640 1 -- 172.21.15.98:0/1322070617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb001fd280 con 0x7fbb0006e5d0 2023-11-28T17:41:22.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.991+0000 7fbaf5ffb640 1 -- 172.21.15.98:0/1322070617 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc00b540 con 0x7fbb0006e5d0 2023-11-28T17:41:22.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.992+0000 7fbb06071640 1 -- 172.21.15.98:0/1322070617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb000c9bc0 con 0x7fbb0006e5d0 2023-11-28T17:41:22.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.993+0000 7fbb06071640 1 -- 172.21.15.98:0/1322070617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb00074db0 con 0x7fbb0006e5d0 2023-11-28T17:41:22.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.993+0000 7fbaf5ffb640 1 -- 172.21.15.98:0/1322070617 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fbafc00b6a0 con 0x7fbb0006e5d0 2023-11-28T17:41:22.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.995+0000 7fbaf5ffb640 1 --2- 172.21.15.98:0/1322070617 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbae406b2a0 0x7fbae406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:22.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.995+0000 7fbaf5ffb640 1 -- 172.21.15.98:0/1322070617 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbafc08dc70 con 0x7fbb0006e5d0 2023-11-28T17:41:22.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.995+0000 7fbb0486e640 1 --2- 172.21.15.98:0/1322070617 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbae406b2a0 0x7fbae406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:22.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.996+0000 7fbb0486e640 1 --2- 172.21.15.98:0/1322070617 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbae406b2a0 0x7fbae406d760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0035000 tx=0x7fbaf0038040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:23.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:22.999+0000 7fbaf5ffb640 1 -- 172.21.15.98:0/1322070617 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbafc058af0 con 0x7fbb0006e5d0 2023-11-28T17:41:23.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.181+0000 7fbb06071640 1 -- 172.21.15.98:0/1322070617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb000c7a60 con 0x7fbb0006e5d0 2023-11-28T17:41:23.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.185+0000 7fbaf5ffb640 1 -- 172.21.15.98:0/1322070617 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16517 (secure 0 0 0) 0x7fbafc01b070 con 0x7fbb0006e5d0 2023-11-28T17:41:23.187 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:23.187 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:41:18.195933+0000","last_up_change":"2023-11-28T17:41:09.556993+0000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:41:10.303280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:41:14.890871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:41:15.704103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6810","nonce":2188704385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6813","nonce":2188704385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6819","nonce":2188704385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6816","nonce":2188704385}]},"public_addr":"172.21.15.98:6810/2188704385","cluster_addr":"172.21.15.98:6813/2188704385","heartbeat_back_addr":"172.21.15.98:6819/2188704385","heartbeat_front_addr":"172.21.15.98:6816/2188704385","state":["exists","up"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6801","nonce":3918226860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6803","nonce":3918226860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6807","nonce":3918226860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6805","nonce":3918226860}]},"public_addr":"172.21.15.98:6801/3918226860","cluster_addr":"172.21.15.98:6803/3918226860","heartbeat_back_addr":"172.21.15.98:6807/3918226860","heartbeat_front_addr":"172.21.15.98:6805/3918226860","state":["exists","up"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6812","nonce":2691532400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6815","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6818","nonce":2691532400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6823","nonce":2691532400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6821","nonce":2691532400}]},"public_addr":"172.21.15.98:6812/2691532400","cluster_addr":"172.21.15.98:6818/2691532400","heartbeat_back_addr":"172.21.15.98:6823/2691532400","heartbeat_front_addr":"172.21.15.98:6821/2691532400","state":["exists","up"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6825","nonce":1905487008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6827","nonce":1905487008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6831","nonce":1905487008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6829","nonce":1905487008}]},"public_addr":"172.21.15.98:6825/1905487008","cluster_addr":"172.21.15.98:6827/1905487008","heartbeat_back_addr":"172.21.15.98:6831/1905487008","heartbeat_front_addr":"172.21.15.98:6829/1905487008","state":["exists","up"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6801","nonce":2974677215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6803","nonce":2974677215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6807","nonce":2974677215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6805","nonce":2974677215}]},"public_addr":"172.21.15.113:6801/2974677215","cluster_addr":"172.21.15.113:6803/2974677215","heartbeat_back_addr":"172.21.15.113:6807/2974677215","heartbeat_front_addr":"172.21.15.113:6805/2974677215","state":["exists","up"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6817","nonce":2685000355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6821","nonce":2685000355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6829","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6831","nonce":2685000355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6823","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6826","nonce":2685000355}]},"public_addr":"172.21.15.113:6817/2685000355","cluster_addr":"172.21.15.113:6821/2685000355","heartbeat_back_addr":"172.21.15.113:6831/2685000355","heartbeat_front_addr":"172.21.15.113:6826/2685000355","state":["exists","up"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6809","nonce":2764300831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6811","nonce":2764300831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6815","nonce":2764300831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6813","nonce":2764300831}]},"public_addr":"172.21.15.113:6809/2764300831","cluster_addr":"172.21.15.113:6811/2764300831","heartbeat_back_addr":"172.21.15.113:6815/2764300831","heartbeat_front_addr":"172.21.15.113:6813/2764300831","state":["exists","up"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6819","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6820","nonce":1656248400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6824","nonce":1656248400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6830","nonce":1656248400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6825","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6827","nonce":1656248400}]},"public_addr":"172.21.15.113:6820/1656248400","cluster_addr":"172.21.15.113:6824/1656248400","heartbeat_back_addr":"172.21.15.113:6830/1656248400","heartbeat_front_addr":"172.21.15.113:6827/1656248400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.379549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.312196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.117494+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.388128+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.360608+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.411101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.355995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.390362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:41:23.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.187+0000 7fbb06071640 1 -- 172.21.15.98:0/1322070617 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbae406b2a0 msgr2=0x7fbae406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:23.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.187+0000 7fbb06071640 1 --2- 172.21.15.98:0/1322070617 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbae406b2a0 0x7fbae406d760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0035000 tx=0x7fbaf0038040 comp rx=0 tx=0).stop 2023-11-28T17:41:23.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.187+0000 7fbb06071640 1 -- 172.21.15.98:0/1322070617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb0006e5d0 msgr2=0x7fbb000c27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.187+0000 7fbb06071640 1 --2- 172.21.15.98:0/1322070617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb0006e5d0 0x7fbb000c27f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbafc010cb0 tx=0x7fbafc00e430 comp rx=0 tx=0).stop 2023-11-28T17:41:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.187+0000 7fbb06071640 1 -- 172.21.15.98:0/1322070617 shutdown_connections 2023-11-28T17:41:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.187+0000 7fbb06071640 1 --2- 172.21.15.98:0/1322070617 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbae406b2a0 0x7fbae406d760 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.187+0000 7fbb06071640 1 --2- 172.21.15.98:0/1322070617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb000c0df0 0x7fbb0015cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.187+0000 7fbb06071640 1 --2- 172.21.15.98:0/1322070617 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbb000c0a10 0x7fbb000caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.187+0000 7fbb06071640 1 --2- 172.21.15.98:0/1322070617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb0006e5d0 0x7fbb000c27f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.187+0000 7fbb06071640 1 -- 172.21.15.98:0/1322070617 >> 172.21.15.98:0/1322070617 conn(0x7fbb00066230 msgr2=0x7fbb00072310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.188+0000 7fbb06071640 1 -- 172.21.15.98:0/1322070617 shutdown_connections 2023-11-28T17:41:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.188+0000 7fbb06071640 1 -- 172.21.15.98:0/1322070617 wait complete. 2023-11-28T17:41:23.206 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T17:41:23.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:41:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.414+0000 7ff2dffff640 1 Processor -- start 2023-11-28T17:41:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.414+0000 7ff2dffff640 1 -- start start 2023-11-28T17:41:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.416+0000 7ff2dffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff2e0153bb0 0x7ff2e015a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.416+0000 7ff2dffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2e00ca7d0 0x7ff2e00b3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.416+0000 7ff2dffff640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e00cac40 con 0x7ff2e00ca390 2023-11-28T17:41:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.416+0000 7ff2dffff640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e015d120 con 0x7ff2e00ca7d0 2023-11-28T17:41:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.416+0000 7ff2dffff640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e015d330 con 0x7ff2e0153bb0 2023-11-28T17:41:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.416+0000 7ff2df7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2e00ca7d0 0x7ff2e00b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.416+0000 7ff2df7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2e00ca7d0 0x7ff2e00b3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49248/0 (socket says 172.21.15.98:49248) 2023-11-28T17:41:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.416+0000 7ff2df7fe640 1 -- 172.21.15.98:0/1898179306 learned_addr learned my addr 172.21.15.98:0/1898179306 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.416+0000 7ff2deffd640 1 --2- 172.21.15.98:0/1898179306 >> v2:172.21.15.113:3300/0 conn(0x7ff2e0153bb0 0x7ff2e015a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.416+0000 7ff2df7fe640 1 -- 172.21.15.98:0/1898179306 >> v2:172.21.15.113:3300/0 conn(0x7ff2e0153bb0 msgr2=0x7ff2e015a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.416+0000 7ff2df7fe640 1 --2- 172.21.15.98:0/1898179306 >> v2:172.21.15.113:3300/0 conn(0x7ff2e0153bb0 0x7ff2e015a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.416+0000 7ff2df7fe640 1 -- 172.21.15.98:0/1898179306 >> v1:172.21.15.113:6789/0 conn(0x7ff2e00ca390 legacy=0x7ff2e00cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.416+0000 7ff2df7fe640 1 -- 172.21.15.98:0/1898179306 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e015d610 con 0x7ff2e00ca7d0 2023-11-28T17:41:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.417+0000 7ff2df7fe640 1 --2- 172.21.15.98:0/1898179306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e00ca7d0 0x7ff2e00b3760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff2e00d0430 tx=0x7ff2d802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bebf73d59ba0ceb6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.418+0000 7ff2ddffb640 1 -- 172.21.15.98:0/1898179306 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8030860 con 0x7ff2e00ca7d0 2023-11-28T17:41:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.418+0000 7ff2ddffb640 1 -- 172.21.15.98:0/1898179306 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d80309c0 con 0x7ff2e00ca7d0 2023-11-28T17:41:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.418+0000 7ff2ddffb640 1 -- 172.21.15.98:0/1898179306 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8030d50 con 0x7ff2e00ca7d0 2023-11-28T17:41:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.418+0000 7ff2dffff640 1 -- 172.21.15.98:0/1898179306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e00ca7d0 msgr2=0x7ff2e00b3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.418+0000 7ff2dffff640 1 --2- 172.21.15.98:0/1898179306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e00ca7d0 0x7ff2e00b3760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff2e00d0430 tx=0x7ff2d802b9a0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.418+0000 7ff2dffff640 1 -- 172.21.15.98:0/1898179306 shutdown_connections 2023-11-28T17:41:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.418+0000 7ff2dffff640 1 --2- 172.21.15.98:0/1898179306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e00ca7d0 0x7ff2e00b3760 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.418+0000 7ff2dffff640 1 --2- 172.21.15.98:0/1898179306 >> v2:172.21.15.113:3300/0 conn(0x7ff2e0153bb0 0x7ff2e015a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.418+0000 7ff2dffff640 1 -- 172.21.15.98:0/1898179306 >> 172.21.15.98:0/1898179306 conn(0x7ff2e0071c00 msgr2=0x7ff2e0072010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.419+0000 7ff2dffff640 1 -- 172.21.15.98:0/1898179306 shutdown_connections 2023-11-28T17:41:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.419+0000 7ff2dffff640 1 -- 172.21.15.98:0/1898179306 wait complete. 2023-11-28T17:41:23.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.420+0000 7ff2dffff640 1 Processor -- start 2023-11-28T17:41:23.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.420+0000 7ff2dffff640 1 -- start start 2023-11-28T17:41:23.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.421+0000 7ff2dffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2e00ca390 0x7ff2e0151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.421+0000 7ff2dffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2e00ca7d0 0x7ff2e00cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.421+0000 7ff2dffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e0153bb0 0x7ff2e00cde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.421+0000 7ff2dffff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e0151f30 con 0x7ff2e0153bb0 2023-11-28T17:41:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.421+0000 7ff2dffff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e01f02d0 con 0x7ff2e00ca390 2023-11-28T17:41:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.421+0000 7ff2dffff640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e01f0590 con 0x7ff2e00ca7d0 2023-11-28T17:41:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.422+0000 7ff2df7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e0153bb0 0x7ff2e00cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.422+0000 7ff2df7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e0153bb0 0x7ff2e00cde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49264/0 (socket says 172.21.15.98:49264) 2023-11-28T17:41:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.422+0000 7ff2df7fe640 1 -- 172.21.15.98:0/1680225630 learned_addr learned my addr 172.21.15.98:0/1680225630 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.422+0000 7ff2deffd640 1 --2- 172.21.15.98:0/1680225630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2e00ca390 0x7ff2e0151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.422+0000 7ff2de7fc640 1 --2- 172.21.15.98:0/1680225630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2e00ca7d0 0x7ff2e00cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.422+0000 7ff2df7fe640 1 -- 172.21.15.98:0/1680225630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2e00ca7d0 msgr2=0x7ff2e00cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:23.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.422+0000 7ff2df7fe640 1 --2- 172.21.15.98:0/1680225630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2e00ca7d0 0x7ff2e00cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.422+0000 7ff2df7fe640 1 -- 172.21.15.98:0/1680225630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2e00ca390 msgr2=0x7ff2e0151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:23.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.422+0000 7ff2df7fe640 1 --2- 172.21.15.98:0/1680225630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2e00ca390 0x7ff2e0151550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.422+0000 7ff2df7fe640 1 -- 172.21.15.98:0/1680225630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d80094d0 con 0x7ff2e0153bb0 2023-11-28T17:41:23.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.422+0000 7ff2df7fe640 1 --2- 172.21.15.98:0/1680225630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e0153bb0 0x7ff2e00cde10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff2d80090b0 tx=0x7ff2d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:23.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.423+0000 7ff2d7fff640 1 -- 172.21.15.98:0/1680225630 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8039c90 con 0x7ff2e0153bb0 2023-11-28T17:41:23.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.423+0000 7ff2d7fff640 1 -- 172.21.15.98:0/1680225630 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d8039df0 con 0x7ff2e0153bb0 2023-11-28T17:41:23.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.423+0000 7ff2d7fff640 1 -- 172.21.15.98:0/1680225630 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8043640 con 0x7ff2e0153bb0 2023-11-28T17:41:23.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.423+0000 7ff2dffff640 1 -- 172.21.15.98:0/1680225630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e00cef80 con 0x7ff2e0153bb0 2023-11-28T17:41:23.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.423+0000 7ff2dffff640 1 -- 172.21.15.98:0/1680225630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e00b5290 con 0x7ff2e0153bb0 2023-11-28T17:41:23.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.424+0000 7ff2dffff640 1 -- 172.21.15.98:0/1680225630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e00cd5f0 con 0x7ff2e0153bb0 2023-11-28T17:41:23.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.425+0000 7ff2d7fff640 1 -- 172.21.15.98:0/1680225630 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff2d80437a0 con 0x7ff2e0153bb0 2023-11-28T17:41:23.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.425+0000 7ff2d7fff640 1 --2- 172.21.15.98:0/1680225630 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff2c006b2d0 0x7ff2c006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:23.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.426+0000 7ff2d7fff640 1 -- 172.21.15.98:0/1680225630 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2d803e070 con 0x7ff2e0153bb0 2023-11-28T17:41:23.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.426+0000 7ff2deffd640 1 --2- 172.21.15.98:0/1680225630 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff2c006b2d0 0x7ff2c006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:23.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.426+0000 7ff2deffd640 1 --2- 172.21.15.98:0/1680225630 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff2c006b2d0 0x7ff2c006d790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8006fd0 tx=0x7ff2c8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:23.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.430+0000 7ff2d7fff640 1 -- 172.21.15.98:0/1680225630 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff2e00cd5f0 con 0x7ff2e0153bb0 2023-11-28T17:41:23.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.620+0000 7ff2dffff640 1 -- 172.21.15.98:0/1680225630 --> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff2e00c6400 con 0x7ff2c006b2d0 2023-11-28T17:41:23.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.630+0000 7ff2d7fff640 1 -- 172.21.15.98:0/1680225630 <== mgr.4098 v2:172.21.15.98:6832/2274392793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264402 (secure 0 0 0) 0x7ff2e00c6400 con 0x7ff2c006b2d0 2023-11-28T17:41:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:23.637 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-11-28T17:41:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.636+0000 7ff2dffff640 1 -- 172.21.15.98:0/1680225630 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff2c006b2d0 msgr2=0x7ff2c006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.636+0000 7ff2dffff640 1 --2- 172.21.15.98:0/1680225630 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff2c006b2d0 0x7ff2c006d790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8006fd0 tx=0x7ff2c8008040 comp rx=0 tx=0).stop 2023-11-28T17:41:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.636+0000 7ff2dffff640 1 -- 172.21.15.98:0/1680225630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e0153bb0 msgr2=0x7ff2e00cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.636+0000 7ff2dffff640 1 --2- 172.21.15.98:0/1680225630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e0153bb0 0x7ff2e00cde10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff2d80090b0 tx=0x7ff2d8004730 comp rx=0 tx=0).stop 2023-11-28T17:41:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.636+0000 7ff2dffff640 1 -- 172.21.15.98:0/1680225630 shutdown_connections 2023-11-28T17:41:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.636+0000 7ff2dffff640 1 --2- 172.21.15.98:0/1680225630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2e0153bb0 0x7ff2e00cde10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.636+0000 7ff2dffff640 1 --2- 172.21.15.98:0/1680225630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff2e00ca7d0 0x7ff2e00cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.636+0000 7ff2dffff640 1 --2- 172.21.15.98:0/1680225630 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff2c006b2d0 0x7ff2c006d790 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.636+0000 7ff2dffff640 1 --2- 172.21.15.98:0/1680225630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2e00ca390 0x7ff2e0151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.636+0000 7ff2dffff640 1 -- 172.21.15.98:0/1680225630 >> 172.21.15.98:0/1680225630 conn(0x7ff2e0071c00 msgr2=0x7ff2e00cb840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:23.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.637+0000 7ff2dffff640 1 -- 172.21.15.98:0/1680225630 shutdown_connections 2023-11-28T17:41:23.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.637+0000 7ff2dffff640 1 -- 172.21.15.98:0/1680225630 wait complete. 2023-11-28T17:41:23.655 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-11-28T17:41:22.235317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218012,"kb_used_data":1820,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756708,"statfs":{"total":773094113280,"available":772870868992,"internally_reserved":0,"allocated":1863680,"data_stored":1773617,"data_compressed":65062,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.011876"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339217+0000","last_change":"2023-11-28T17:41:17.238145+0000","last_active":"2023-11-28T17:41:18.339217+0000","last_peered":"2023-11-28T17:41:18.339217+0000","last_clean":"2023-11-28T17:41:18.339217+0000","last_became_active":"2023-11-28T17:41:17.237776+0000","last_became_peered":"2023-11-28T17:41:17.237776+0000","last_unstale":"2023-11-28T17:41:18.339217+0000","last_undegraded":"2023-11-28T17:41:18.339217+0000","last_fullsized":"2023-11-28T17:41:18.339217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:41:06.902665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632799+0000","last_change":"2023-11-28T17:41:17.231913+0000","last_active":"2023-11-28T17:41:19.632799+0000","last_peered":"2023-11-28T17:41:19.632799+0000","last_clean":"2023-11-28T17:41:19.632799+0000","last_became_active":"2023-11-28T17:41:17.231622+0000","last_became_peered":"2023-11-28T17:41:17.231622+0000","last_unstale":"2023-11-28T17:41:19.632799+0000","last_undegraded":"2023-11-28T17:41:19.632799+0000","last_fullsized":"2023-11-28T17:41:19.632799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:06.911223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649842+0000","last_change":"2023-11-28T17:41:17.230232+0000","last_active":"2023-11-28T17:41:18.649842+0000","last_peered":"2023-11-28T17:41:18.649842+0000","last_clean":"2023-11-28T17:41:18.649842+0000","last_became_active":"2023-11-28T17:41:17.229795+0000","last_became_peered":"2023-11-28T17:41:17.229795+0000","last_unstale":"2023-11-28T17:41:18.649842+0000","last_undegraded":"2023-11-28T17:41:18.649842+0000","last_fullsized":"2023-11-28T17:41:18.649842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:09:09.507547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207139+0000","last_change":"2023-11-28T17:41:17.234948+0000","last_active":"2023-11-28T17:41:18.207139+0000","last_peered":"2023-11-28T17:41:18.207139+0000","last_clean":"2023-11-28T17:41:18.207139+0000","last_became_active":"2023-11-28T17:41:17.233674+0000","last_became_peered":"2023-11-28T17:41:17.233674+0000","last_unstale":"2023-11-28T17:41:18.207139+0000","last_undegraded":"2023-11-28T17:41:18.207139+0000","last_fullsized":"2023-11-28T17:41:18.207139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:45:10.831719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338225+0000","last_change":"2023-11-28T17:41:17.233908+0000","last_active":"2023-11-28T17:41:18.338225+0000","last_peered":"2023-11-28T17:41:18.338225+0000","last_clean":"2023-11-28T17:41:18.338225+0000","last_became_active":"2023-11-28T17:41:17.233589+0000","last_became_peered":"2023-11-28T17:41:17.233589+0000","last_unstale":"2023-11-28T17:41:18.338225+0000","last_undegraded":"2023-11-28T17:41:18.338225+0000","last_fullsized":"2023-11-28T17:41:18.338225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:21:37.379103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206516+0000","last_change":"2023-11-28T17:41:17.239186+0000","last_active":"2023-11-28T17:41:18.206516+0000","last_peered":"2023-11-28T17:41:18.206516+0000","last_clean":"2023-11-28T17:41:18.206516+0000","last_became_active":"2023-11-28T17:41:17.233898+0000","last_became_peered":"2023-11-28T17:41:17.233898+0000","last_unstale":"2023-11-28T17:41:18.206516+0000","last_undegraded":"2023-11-28T17:41:18.206516+0000","last_fullsized":"2023-11-28T17:41:18.206516+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:54.918589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338198+0000","last_change":"2023-11-28T17:41:17.227923+0000","last_active":"2023-11-28T17:41:18.338198+0000","last_peered":"2023-11-28T17:41:18.338198+0000","last_clean":"2023-11-28T17:41:18.338198+0000","last_became_active":"2023-11-28T17:41:17.227642+0000","last_became_peered":"2023-11-28T17:41:17.227642+0000","last_unstale":"2023-11-28T17:41:18.338198+0000","last_undegraded":"2023-11-28T17:41:18.338198+0000","last_fullsized":"2023-11-28T17:41:18.338198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:33:29.400533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567710+0000","last_change":"2023-11-28T17:41:17.239818+0000","last_active":"2023-11-28T17:41:18.567710+0000","last_peered":"2023-11-28T17:41:18.567710+0000","last_clean":"2023-11-28T17:41:18.567710+0000","last_became_active":"2023-11-28T17:41:17.239594+0000","last_became_peered":"2023-11-28T17:41:17.239594+0000","last_unstale":"2023-11-28T17:41:18.567710+0000","last_undegraded":"2023-11-28T17:41:18.567710+0000","last_fullsized":"2023-11-28T17:41:18.567710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:23.969471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339318+0000","last_change":"2023-11-28T17:41:17.243214+0000","last_active":"2023-11-28T17:41:18.339318+0000","last_peered":"2023-11-28T17:41:18.339318+0000","last_clean":"2023-11-28T17:41:18.339318+0000","last_became_active":"2023-11-28T17:41:17.242933+0000","last_became_peered":"2023-11-28T17:41:17.242933+0000","last_unstale":"2023-11-28T17:41:18.339318+0000","last_undegraded":"2023-11-28T17:41:18.339318+0000","last_fullsized":"2023-11-28T17:41:18.339318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:47:47.697137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633703+0000","last_change":"2023-11-28T17:41:17.242507+0000","last_active":"2023-11-28T17:41:19.633703+0000","last_peered":"2023-11-28T17:41:19.633703+0000","last_clean":"2023-11-28T17:41:19.633703+0000","last_became_active":"2023-11-28T17:41:17.242196+0000","last_became_peered":"2023-11-28T17:41:17.242196+0000","last_unstale":"2023-11-28T17:41:19.633703+0000","last_undegraded":"2023-11-28T17:41:19.633703+0000","last_fullsized":"2023-11-28T17:41:19.633703+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:06.103318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633386+0000","last_change":"2023-11-28T17:41:17.244694+0000","last_active":"2023-11-28T17:41:19.633386+0000","last_peered":"2023-11-28T17:41:19.633386+0000","last_clean":"2023-11-28T17:41:19.633386+0000","last_became_active":"2023-11-28T17:41:17.244301+0000","last_became_peered":"2023-11-28T17:41:17.244301+0000","last_unstale":"2023-11-28T17:41:19.633386+0000","last_undegraded":"2023-11-28T17:41:19.633386+0000","last_fullsized":"2023-11-28T17:41:19.633386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:40.030685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208504+0000","last_change":"2023-11-28T17:41:17.242800+0000","last_active":"2023-11-28T17:41:18.208504+0000","last_peered":"2023-11-28T17:41:18.208504+0000","last_clean":"2023-11-28T17:41:18.208504+0000","last_became_active":"2023-11-28T17:41:17.242463+0000","last_became_peered":"2023-11-28T17:41:17.242463+0000","last_unstale":"2023-11-28T17:41:18.208504+0000","last_undegraded":"2023-11-28T17:41:18.208504+0000","last_fullsized":"2023-11-28T17:41:18.208504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:57:34.802348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568335+0000","last_change":"2023-11-28T17:41:17.232485+0000","last_active":"2023-11-28T17:41:18.568335+0000","last_peered":"2023-11-28T17:41:18.568335+0000","last_clean":"2023-11-28T17:41:18.568335+0000","last_became_active":"2023-11-28T17:41:17.232132+0000","last_became_peered":"2023-11-28T17:41:17.232132+0000","last_unstale":"2023-11-28T17:41:18.568335+0000","last_undegraded":"2023-11-28T17:41:18.568335+0000","last_fullsized":"2023-11-28T17:41:18.568335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:49.292565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649479+0000","last_change":"2023-11-28T17:41:17.234514+0000","last_active":"2023-11-28T17:41:18.649479+0000","last_peered":"2023-11-28T17:41:18.649479+0000","last_clean":"2023-11-28T17:41:18.649479+0000","last_became_active":"2023-11-28T17:41:17.234154+0000","last_became_peered":"2023-11-28T17:41:17.234154+0000","last_unstale":"2023-11-28T17:41:18.649479+0000","last_undegraded":"2023-11-28T17:41:18.649479+0000","last_fullsized":"2023-11-28T17:41:18.649479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:22.430159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632129+0000","last_change":"2023-11-28T17:41:17.225905+0000","last_active":"2023-11-28T17:41:19.632129+0000","last_peered":"2023-11-28T17:41:19.632129+0000","last_clean":"2023-11-28T17:41:19.632129+0000","last_became_active":"2023-11-28T17:41:17.223533+0000","last_became_peered":"2023-11-28T17:41:17.223533+0000","last_unstale":"2023-11-28T17:41:19.632129+0000","last_undegraded":"2023-11-28T17:41:19.632129+0000","last_fullsized":"2023-11-28T17:41:19.632129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:32.314586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.205612+0000","last_change":"2023-11-28T17:41:17.236242+0000","last_active":"2023-11-28T17:41:18.205612+0000","last_peered":"2023-11-28T17:41:18.205612+0000","last_clean":"2023-11-28T17:41:18.205612+0000","last_became_active":"2023-11-28T17:41:17.235890+0000","last_became_peered":"2023-11-28T17:41:17.235890+0000","last_unstale":"2023-11-28T17:41:18.205612+0000","last_undegraded":"2023-11-28T17:41:18.205612+0000","last_fullsized":"2023-11-28T17:41:18.205612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:20:09.004702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208885+0000","last_change":"2023-11-28T17:41:17.234610+0000","last_active":"2023-11-28T17:41:18.208885+0000","last_peered":"2023-11-28T17:41:18.208885+0000","last_clean":"2023-11-28T17:41:18.208885+0000","last_became_active":"2023-11-28T17:41:17.234255+0000","last_became_peered":"2023-11-28T17:41:17.234255+0000","last_unstale":"2023-11-28T17:41:18.208885+0000","last_undegraded":"2023-11-28T17:41:18.208885+0000","last_fullsized":"2023-11-28T17:41:18.208885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:25:51.137148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.234674+0000","last_change":"2023-11-28T17:41:17.231838+0000","last_active":"2023-11-28T17:41:17.234674+0000","last_peered":"2023-11-28T17:41:17.234674+0000","last_clean":"2023-11-28T17:41:17.234674+0000","last_became_active":"2023-11-28T17:41:17.231533+0000","last_became_peered":"2023-11-28T17:41:17.231533+0000","last_unstale":"2023-11-28T17:41:17.234674+0000","last_undegraded":"2023-11-28T17:41:17.234674+0000","last_fullsized":"2023-11-28T17:41:17.234674+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:21:03.904863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339038+0000","last_change":"2023-11-28T17:41:17.238888+0000","last_active":"2023-11-28T17:41:18.339038+0000","last_peered":"2023-11-28T17:41:18.339038+0000","last_clean":"2023-11-28T17:41:18.339038+0000","last_became_active":"2023-11-28T17:41:17.236140+0000","last_became_peered":"2023-11-28T17:41:17.236140+0000","last_unstale":"2023-11-28T17:41:18.339038+0000","last_undegraded":"2023-11-28T17:41:18.339038+0000","last_fullsized":"2023-11-28T17:41:18.339038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:20:57.936710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633201+0000","last_change":"2023-11-28T17:41:17.242150+0000","last_active":"2023-11-28T17:41:19.633201+0000","last_peered":"2023-11-28T17:41:19.633201+0000","last_clean":"2023-11-28T17:41:19.633201+0000","last_became_active":"2023-11-28T17:41:17.241832+0000","last_became_peered":"2023-11-28T17:41:17.241832+0000","last_unstale":"2023-11-28T17:41:19.633201+0000","last_undegraded":"2023-11-28T17:41:19.633201+0000","last_fullsized":"2023-11-28T17:41:19.633201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:30:37.837711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649928+0000","last_change":"2023-11-28T17:41:17.233651+0000","last_active":"2023-11-28T17:41:18.649928+0000","last_peered":"2023-11-28T17:41:18.649928+0000","last_clean":"2023-11-28T17:41:18.649928+0000","last_became_active":"2023-11-28T17:41:17.233393+0000","last_became_peered":"2023-11-28T17:41:17.233393+0000","last_unstale":"2023-11-28T17:41:18.649928+0000","last_undegraded":"2023-11-28T17:41:18.649928+0000","last_fullsized":"2023-11-28T17:41:18.649928+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:52:08.152046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206688+0000","last_change":"2023-11-28T17:41:17.239696+0000","last_active":"2023-11-28T17:41:18.206688+0000","last_peered":"2023-11-28T17:41:18.206688+0000","last_clean":"2023-11-28T17:41:18.206688+0000","last_became_active":"2023-11-28T17:41:17.239449+0000","last_became_peered":"2023-11-28T17:41:17.239449+0000","last_unstale":"2023-11-28T17:41:18.206688+0000","last_undegraded":"2023-11-28T17:41:18.206688+0000","last_fullsized":"2023-11-28T17:41:18.206688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:08:43.019689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567770+0000","last_change":"2023-11-28T17:41:17.244917+0000","last_active":"2023-11-28T17:41:18.567770+0000","last_peered":"2023-11-28T17:41:18.567770+0000","last_clean":"2023-11-28T17:41:18.567770+0000","last_became_active":"2023-11-28T17:41:17.244676+0000","last_became_peered":"2023-11-28T17:41:17.244676+0000","last_unstale":"2023-11-28T17:41:18.567770+0000","last_undegraded":"2023-11-28T17:41:18.567770+0000","last_fullsized":"2023-11-28T17:41:18.567770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:08:10.818556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206063+0000","last_change":"2023-11-28T17:41:17.236488+0000","last_active":"2023-11-28T17:41:18.206063+0000","last_peered":"2023-11-28T17:41:18.206063+0000","last_clean":"2023-11-28T17:41:18.206063+0000","last_became_active":"2023-11-28T17:41:17.235554+0000","last_became_peered":"2023-11-28T17:41:17.235554+0000","last_unstale":"2023-11-28T17:41:18.206063+0000","last_undegraded":"2023-11-28T17:41:18.206063+0000","last_fullsized":"2023-11-28T17:41:18.206063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:29:35.500772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568927+0000","last_change":"2023-11-28T17:41:17.245042+0000","last_active":"2023-11-28T17:41:18.568927+0000","last_peered":"2023-11-28T17:41:18.568927+0000","last_clean":"2023-11-28T17:41:18.568927+0000","last_became_active":"2023-11-28T17:41:17.244729+0000","last_became_peered":"2023-11-28T17:41:17.244729+0000","last_unstale":"2023-11-28T17:41:18.568927+0000","last_undegraded":"2023-11-28T17:41:18.568927+0000","last_fullsized":"2023-11-28T17:41:18.568927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:57:00.728653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633914+0000","last_change":"2023-11-28T17:41:17.249367+0000","last_active":"2023-11-28T17:41:19.633914+0000","last_peered":"2023-11-28T17:41:19.633914+0000","last_clean":"2023-11-28T17:41:19.633914+0000","last_became_active":"2023-11-28T17:41:17.249093+0000","last_became_peered":"2023-11-28T17:41:17.249093+0000","last_unstale":"2023-11-28T17:41:19.633914+0000","last_undegraded":"2023-11-28T17:41:19.633914+0000","last_fullsized":"2023-11-28T17:41:19.633914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:12:41.994354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632733+0000","last_change":"2023-11-28T17:41:17.230256+0000","last_active":"2023-11-28T17:41:19.632733+0000","last_peered":"2023-11-28T17:41:19.632733+0000","last_clean":"2023-11-28T17:41:19.632733+0000","last_became_active":"2023-11-28T17:41:17.229551+0000","last_became_peered":"2023-11-28T17:41:17.229551+0000","last_unstale":"2023-11-28T17:41:19.632733+0000","last_undegraded":"2023-11-28T17:41:19.632733+0000","last_fullsized":"2023-11-28T17:41:19.632733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:54:09.421121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568751+0000","last_change":"2023-11-28T17:41:17.241672+0000","last_active":"2023-11-28T17:41:18.568751+0000","last_peered":"2023-11-28T17:41:18.568751+0000","last_clean":"2023-11-28T17:41:18.568751+0000","last_became_active":"2023-11-28T17:41:17.241366+0000","last_became_peered":"2023-11-28T17:41:17.241366+0000","last_unstale":"2023-11-28T17:41:18.568751+0000","last_undegraded":"2023-11-28T17:41:18.568751+0000","last_fullsized":"2023-11-28T17:41:18.568751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:29:01.718362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650047+0000","last_change":"2023-11-28T17:41:17.229925+0000","last_active":"2023-11-28T17:41:18.650047+0000","last_peered":"2023-11-28T17:41:18.650047+0000","last_clean":"2023-11-28T17:41:18.650047+0000","last_became_active":"2023-11-28T17:41:17.229617+0000","last_became_peered":"2023-11-28T17:41:17.229617+0000","last_unstale":"2023-11-28T17:41:18.650047+0000","last_undegraded":"2023-11-28T17:41:18.650047+0000","last_fullsized":"2023-11-28T17:41:18.650047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:40:14.646558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206950+0000","last_change":"2023-11-28T17:41:17.241933+0000","last_active":"2023-11-28T17:41:18.206950+0000","last_peered":"2023-11-28T17:41:18.206950+0000","last_clean":"2023-11-28T17:41:18.206950+0000","last_became_active":"2023-11-28T17:41:17.241634+0000","last_became_peered":"2023-11-28T17:41:17.241634+0000","last_unstale":"2023-11-28T17:41:18.206950+0000","last_undegraded":"2023-11-28T17:41:18.206950+0000","last_fullsized":"2023-11-28T17:41:18.206950+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:44:01.525920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.238580+0000","last_change":"2023-11-28T17:41:17.233872+0000","last_active":"2023-11-28T17:41:17.238580+0000","last_peered":"2023-11-28T17:41:17.238580+0000","last_clean":"2023-11-28T17:41:17.238580+0000","last_became_active":"2023-11-28T17:41:17.233586+0000","last_became_peered":"2023-11-28T17:41:17.233586+0000","last_unstale":"2023-11-28T17:41:17.238580+0000","last_undegraded":"2023-11-28T17:41:17.238580+0000","last_fullsized":"2023-11-28T17:41:17.238580+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:57:07.694916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633270+0000","last_change":"2023-11-28T17:41:17.236164+0000","last_active":"2023-11-28T17:41:19.633270+0000","last_peered":"2023-11-28T17:41:19.633270+0000","last_clean":"2023-11-28T17:41:19.633270+0000","last_became_active":"2023-11-28T17:41:17.235821+0000","last_became_peered":"2023-11-28T17:41:17.235821+0000","last_unstale":"2023-11-28T17:41:19.633270+0000","last_undegraded":"2023-11-28T17:41:19.633270+0000","last_fullsized":"2023-11-28T17:41:19.633270+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:54:16.568394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206920+0000","last_change":"2023-11-28T17:41:17.235816+0000","last_active":"2023-11-28T17:41:18.206920+0000","last_peered":"2023-11-28T17:41:18.206920+0000","last_clean":"2023-11-28T17:41:18.206920+0000","last_became_active":"2023-11-28T17:41:17.235506+0000","last_became_peered":"2023-11-28T17:41:17.235506+0000","last_unstale":"2023-11-28T17:41:18.206920+0000","last_undegraded":"2023-11-28T17:41:18.206920+0000","last_fullsized":"2023-11-28T17:41:18.206920+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:18:00.597727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.228065+0000","last_change":"2023-11-28T17:41:16.266076+0000","last_active":"2023-11-28T17:41:17.228065+0000","last_peered":"2023-11-28T17:41:17.228065+0000","last_clean":"2023-11-28T17:41:17.228065+0000","last_became_active":"2023-11-28T17:41:16.265837+0000","last_became_peered":"2023-11-28T17:41:16.265837+0000","last_unstale":"2023-11-28T17:41:17.228065+0000","last_undegraded":"2023-11-28T17:41:17.228065+0000","last_fullsized":"2023-11-28T17:41:17.228065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:01:51.477975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650250+0000","last_change":"2023-11-28T17:41:17.238915+0000","last_active":"2023-11-28T17:41:18.650250+0000","last_peered":"2023-11-28T17:41:18.650250+0000","last_clean":"2023-11-28T17:41:18.650250+0000","last_became_active":"2023-11-28T17:41:17.238581+0000","last_became_peered":"2023-11-28T17:41:17.238581+0000","last_unstale":"2023-11-28T17:41:18.650250+0000","last_undegraded":"2023-11-28T17:41:18.650250+0000","last_fullsized":"2023-11-28T17:41:18.650250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:52:31.640239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632930+0000","last_change":"2023-11-28T17:41:16.261044+0000","last_active":"2023-11-28T17:41:19.632930+0000","last_peered":"2023-11-28T17:41:19.632930+0000","last_clean":"2023-11-28T17:41:19.632930+0000","last_became_active":"2023-11-28T17:41:16.260752+0000","last_became_peered":"2023-11-28T17:41:16.260752+0000","last_unstale":"2023-11-28T17:41:19.632930+0000","last_undegraded":"2023-11-28T17:41:19.632930+0000","last_fullsized":"2023-11-28T17:41:19.632930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:07:01.443140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.227924+0000","last_change":"2023-11-28T17:41:16.261095+0000","last_active":"2023-11-28T17:41:17.227924+0000","last_peered":"2023-11-28T17:41:17.227924+0000","last_clean":"2023-11-28T17:41:17.227924+0000","last_became_active":"2023-11-28T17:41:16.260793+0000","last_became_peered":"2023-11-28T17:41:16.260793+0000","last_unstale":"2023-11-28T17:41:17.227924+0000","last_undegraded":"2023-11-28T17:41:17.227924+0000","last_fullsized":"2023-11-28T17:41:17.227924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:54:16.821314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567897+0000","last_change":"2023-11-28T17:41:17.236517+0000","last_active":"2023-11-28T17:41:18.567897+0000","last_peered":"2023-11-28T17:41:18.567897+0000","last_clean":"2023-11-28T17:41:18.567897+0000","last_became_active":"2023-11-28T17:41:17.235925+0000","last_became_peered":"2023-11-28T17:41:17.235925+0000","last_unstale":"2023-11-28T17:41:18.567897+0000","last_undegraded":"2023-11-28T17:41:18.567897+0000","last_fullsized":"2023-11-28T17:41:18.567897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:45:09.817844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.205947+0000","last_change":"2023-11-28T17:41:16.258176+0000","last_active":"2023-11-28T17:41:18.205947+0000","last_peered":"2023-11-28T17:41:18.205947+0000","last_clean":"2023-11-28T17:41:18.205947+0000","last_became_active":"2023-11-28T17:41:16.257838+0000","last_became_peered":"2023-11-28T17:41:16.257838+0000","last_unstale":"2023-11-28T17:41:18.205947+0000","last_undegraded":"2023-11-28T17:41:18.205947+0000","last_fullsized":"2023-11-28T17:41:18.205947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:31:01.443661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.631980+0000","last_change":"2023-11-28T17:41:17.227823+0000","last_active":"2023-11-28T17:41:19.631980+0000","last_peered":"2023-11-28T17:41:19.631980+0000","last_clean":"2023-11-28T17:41:19.631980+0000","last_became_active":"2023-11-28T17:41:17.227545+0000","last_became_peered":"2023-11-28T17:41:17.227545+0000","last_unstale":"2023-11-28T17:41:19.631980+0000","last_undegraded":"2023-11-28T17:41:19.631980+0000","last_fullsized":"2023-11-28T17:41:19.631980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:20:33.253155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633476+0000","last_change":"2023-11-28T17:41:16.262372+0000","last_active":"2023-11-28T17:41:19.633476+0000","last_peered":"2023-11-28T17:41:19.633476+0000","last_clean":"2023-11-28T17:41:19.633476+0000","last_became_active":"2023-11-28T17:41:16.262029+0000","last_became_peered":"2023-11-28T17:41:16.262029+0000","last_unstale":"2023-11-28T17:41:19.633476+0000","last_undegraded":"2023-11-28T17:41:19.633476+0000","last_fullsized":"2023-11-28T17:41:19.633476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:46:08.491380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207822+0000","last_change":"2023-11-28T17:41:17.235646+0000","last_active":"2023-11-28T17:41:18.207822+0000","last_peered":"2023-11-28T17:41:18.207822+0000","last_clean":"2023-11-28T17:41:18.207822+0000","last_became_active":"2023-11-28T17:41:17.235330+0000","last_became_peered":"2023-11-28T17:41:17.235330+0000","last_unstale":"2023-11-28T17:41:18.207822+0000","last_undegraded":"2023-11-28T17:41:18.207822+0000","last_fullsized":"2023-11-28T17:41:18.207822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:48:12.717031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207869+0000","last_change":"2023-11-28T17:41:16.262716+0000","last_active":"2023-11-28T17:41:18.207869+0000","last_peered":"2023-11-28T17:41:18.207869+0000","last_clean":"2023-11-28T17:41:18.207869+0000","last_became_active":"2023-11-28T17:41:16.262347+0000","last_became_peered":"2023-11-28T17:41:16.262347+0000","last_unstale":"2023-11-28T17:41:18.207869+0000","last_undegraded":"2023-11-28T17:41:18.207869+0000","last_fullsized":"2023-11-28T17:41:18.207869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:50.685367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650261+0000","last_change":"2023-11-28T17:41:17.245852+0000","last_active":"2023-11-28T17:41:18.650261+0000","last_peered":"2023-11-28T17:41:18.650261+0000","last_clean":"2023-11-28T17:41:18.650261+0000","last_became_active":"2023-11-28T17:41:17.245596+0000","last_became_peered":"2023-11-28T17:41:17.245596+0000","last_unstale":"2023-11-28T17:41:18.650261+0000","last_undegraded":"2023-11-28T17:41:18.650261+0000","last_fullsized":"2023-11-28T17:41:18.650261+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:19:03.484052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.204578+0000","last_change":"2023-11-28T17:41:16.254855+0000","last_active":"2023-11-28T17:41:17.204578+0000","last_peered":"2023-11-28T17:41:17.204578+0000","last_clean":"2023-11-28T17:41:17.204578+0000","last_became_active":"2023-11-28T17:41:16.254557+0000","last_became_peered":"2023-11-28T17:41:16.254557+0000","last_unstale":"2023-11-28T17:41:17.204578+0000","last_undegraded":"2023-11-28T17:41:17.204578+0000","last_fullsized":"2023-11-28T17:41:17.204578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:55:38.565240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208613+0000","last_change":"2023-11-28T17:41:17.249020+0000","last_active":"2023-11-28T17:41:18.208613+0000","last_peered":"2023-11-28T17:41:18.208613+0000","last_clean":"2023-11-28T17:41:18.208613+0000","last_became_active":"2023-11-28T17:41:17.248345+0000","last_became_peered":"2023-11-28T17:41:17.248345+0000","last_unstale":"2023-11-28T17:41:18.208613+0000","last_undegraded":"2023-11-28T17:41:18.208613+0000","last_fullsized":"2023-11-28T17:41:18.208613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:05:09.688116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633384+0000","last_change":"2023-11-28T17:41:16.256485+0000","last_active":"2023-11-28T17:41:19.633384+0000","last_peered":"2023-11-28T17:41:19.633384+0000","last_clean":"2023-11-28T17:41:19.633384+0000","last_became_active":"2023-11-28T17:41:16.256165+0000","last_became_peered":"2023-11-28T17:41:16.256165+0000","last_unstale":"2023-11-28T17:41:19.633384+0000","last_undegraded":"2023-11-28T17:41:19.633384+0000","last_fullsized":"2023-11-28T17:41:19.633384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:35:26.805435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338198+0000","last_change":"2023-11-28T17:41:17.230246+0000","last_active":"2023-11-28T17:41:18.338198+0000","last_peered":"2023-11-28T17:41:18.338198+0000","last_clean":"2023-11-28T17:41:18.338198+0000","last_became_active":"2023-11-28T17:41:17.229981+0000","last_became_peered":"2023-11-28T17:41:17.229981+0000","last_unstale":"2023-11-28T17:41:18.338198+0000","last_undegraded":"2023-11-28T17:41:18.338198+0000","last_fullsized":"2023-11-28T17:41:18.338198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:15:48.047641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.220706+0000","last_change":"2023-11-28T17:41:16.250894+0000","last_active":"2023-11-28T17:41:17.220706+0000","last_peered":"2023-11-28T17:41:17.220706+0000","last_clean":"2023-11-28T17:41:17.220706+0000","last_became_active":"2023-11-28T17:41:16.250547+0000","last_became_peered":"2023-11-28T17:41:16.250547+0000","last_unstale":"2023-11-28T17:41:17.220706+0000","last_undegraded":"2023-11-28T17:41:17.220706+0000","last_fullsized":"2023-11-28T17:41:17.220706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:58:47.412734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649511+0000","last_change":"2023-11-28T17:41:17.237500+0000","last_active":"2023-11-28T17:41:18.649511+0000","last_peered":"2023-11-28T17:41:18.649511+0000","last_clean":"2023-11-28T17:41:18.649511+0000","last_became_active":"2023-11-28T17:41:17.237211+0000","last_became_peered":"2023-11-28T17:41:17.237211+0000","last_unstale":"2023-11-28T17:41:18.649511+0000","last_undegraded":"2023-11-28T17:41:18.649511+0000","last_fullsized":"2023-11-28T17:41:18.649511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:13.675182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207345+0000","last_change":"2023-11-28T17:41:16.258147+0000","last_active":"2023-11-28T17:41:18.207345+0000","last_peered":"2023-11-28T17:41:18.207345+0000","last_clean":"2023-11-28T17:41:18.207345+0000","last_became_active":"2023-11-28T17:41:16.257540+0000","last_became_peered":"2023-11-28T17:41:16.257540+0000","last_unstale":"2023-11-28T17:41:18.207345+0000","last_undegraded":"2023-11-28T17:41:18.207345+0000","last_fullsized":"2023-11-28T17:41:18.207345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:51:11.394507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338684+0000","last_change":"2023-11-28T17:41:17.247768+0000","last_active":"2023-11-28T17:41:18.338684+0000","last_peered":"2023-11-28T17:41:18.338684+0000","last_clean":"2023-11-28T17:41:18.338684+0000","last_became_active":"2023-11-28T17:41:17.247475+0000","last_became_peered":"2023-11-28T17:41:17.247475+0000","last_unstale":"2023-11-28T17:41:18.338684+0000","last_undegraded":"2023-11-28T17:41:18.338684+0000","last_fullsized":"2023-11-28T17:41:18.338684+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:18.555573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632851+0000","last_change":"2023-11-28T17:41:16.262927+0000","last_active":"2023-11-28T17:41:19.632851+0000","last_peered":"2023-11-28T17:41:19.632851+0000","last_clean":"2023-11-28T17:41:19.632851+0000","last_became_active":"2023-11-28T17:41:16.262556+0000","last_became_peered":"2023-11-28T17:41:16.262556+0000","last_unstale":"2023-11-28T17:41:19.632851+0000","last_undegraded":"2023-11-28T17:41:19.632851+0000","last_fullsized":"2023-11-28T17:41:19.632851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:35:54.672978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632921+0000","last_change":"2023-11-28T17:41:17.239625+0000","last_active":"2023-11-28T17:41:19.632921+0000","last_peered":"2023-11-28T17:41:19.632921+0000","last_clean":"2023-11-28T17:41:19.632921+0000","last_became_active":"2023-11-28T17:41:17.238215+0000","last_became_peered":"2023-11-28T17:41:17.238215+0000","last_unstale":"2023-11-28T17:41:19.632921+0000","last_undegraded":"2023-11-28T17:41:19.632921+0000","last_fullsized":"2023-11-28T17:41:19.632921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:33:31.879592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.216317+0000","last_change":"2023-11-28T17:41:16.250674+0000","last_active":"2023-11-28T17:41:17.216317+0000","last_peered":"2023-11-28T17:41:17.216317+0000","last_clean":"2023-11-28T17:41:17.216317+0000","last_became_active":"2023-11-28T17:41:16.250312+0000","last_became_peered":"2023-11-28T17:41:16.250312+0000","last_unstale":"2023-11-28T17:41:17.216317+0000","last_undegraded":"2023-11-28T17:41:17.216317+0000","last_fullsized":"2023-11-28T17:41:17.216317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:04:05.342081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338315+0000","last_change":"2023-11-28T17:41:17.247675+0000","last_active":"2023-11-28T17:41:18.338315+0000","last_peered":"2023-11-28T17:41:18.338315+0000","last_clean":"2023-11-28T17:41:18.338315+0000","last_became_active":"2023-11-28T17:41:17.247346+0000","last_became_peered":"2023-11-28T17:41:17.247346+0000","last_unstale":"2023-11-28T17:41:18.338315+0000","last_undegraded":"2023-11-28T17:41:18.338315+0000","last_fullsized":"2023-11-28T17:41:18.338315+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:45:03.326583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.234280+0000","last_change":"2023-11-28T17:41:16.260893+0000","last_active":"2023-11-28T17:41:17.234280+0000","last_peered":"2023-11-28T17:41:17.234280+0000","last_clean":"2023-11-28T17:41:17.234280+0000","last_became_active":"2023-11-28T17:41:16.260637+0000","last_became_peered":"2023-11-28T17:41:16.260637+0000","last_unstale":"2023-11-28T17:41:17.234280+0000","last_undegraded":"2023-11-28T17:41:17.234280+0000","last_fullsized":"2023-11-28T17:41:17.234280+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:07:24.032989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338811+0000","last_change":"2023-11-28T17:41:17.249202+0000","last_active":"2023-11-28T17:41:18.338811+0000","last_peered":"2023-11-28T17:41:18.338811+0000","last_clean":"2023-11-28T17:41:18.338811+0000","last_became_active":"2023-11-28T17:41:17.248187+0000","last_became_peered":"2023-11-28T17:41:17.248187+0000","last_unstale":"2023-11-28T17:41:18.338811+0000","last_undegraded":"2023-11-28T17:41:18.338811+0000","last_fullsized":"2023-11-28T17:41:18.338811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:23.973069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.216810+0000","last_change":"2023-11-28T17:41:16.244938+0000","last_active":"2023-11-28T17:41:17.216810+0000","last_peered":"2023-11-28T17:41:17.216810+0000","last_clean":"2023-11-28T17:41:17.216810+0000","last_became_active":"2023-11-28T17:41:16.244586+0000","last_became_peered":"2023-11-28T17:41:16.244586+0000","last_unstale":"2023-11-28T17:41:17.216810+0000","last_undegraded":"2023-11-28T17:41:17.216810+0000","last_fullsized":"2023-11-28T17:41:17.216810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:38.421966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339409+0000","last_change":"2023-11-28T17:41:17.248488+0000","last_active":"2023-11-28T17:41:18.339409+0000","last_peered":"2023-11-28T17:41:18.339409+0000","last_clean":"2023-11-28T17:41:18.339409+0000","last_became_active":"2023-11-28T17:41:17.248170+0000","last_became_peered":"2023-11-28T17:41:17.248170+0000","last_unstale":"2023-11-28T17:41:18.339409+0000","last_undegraded":"2023-11-28T17:41:18.339409+0000","last_fullsized":"2023-11-28T17:41:18.339409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:41.188370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.204990+0000","last_change":"2023-11-28T17:41:16.246192+0000","last_active":"2023-11-28T17:41:17.204990+0000","last_peered":"2023-11-28T17:41:17.204990+0000","last_clean":"2023-11-28T17:41:17.204990+0000","last_became_active":"2023-11-28T17:41:16.245379+0000","last_became_peered":"2023-11-28T17:41:16.245379+0000","last_unstale":"2023-11-28T17:41:17.204990+0000","last_undegraded":"2023-11-28T17:41:17.204990+0000","last_fullsized":"2023-11-28T17:41:17.204990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:23:08.664344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207475+0000","last_change":"2023-11-28T17:41:17.234273+0000","last_active":"2023-11-28T17:41:18.207475+0000","last_peered":"2023-11-28T17:41:18.207475+0000","last_clean":"2023-11-28T17:41:18.207475+0000","last_became_active":"2023-11-28T17:41:17.233756+0000","last_became_peered":"2023-11-28T17:41:17.233756+0000","last_unstale":"2023-11-28T17:41:18.207475+0000","last_undegraded":"2023-11-28T17:41:18.207475+0000","last_fullsized":"2023-11-28T17:41:18.207475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:18:40.539329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.215640+0000","last_change":"2023-11-28T17:41:16.257952+0000","last_active":"2023-11-28T17:41:17.215640+0000","last_peered":"2023-11-28T17:41:17.215640+0000","last_clean":"2023-11-28T17:41:17.215640+0000","last_became_active":"2023-11-28T17:41:16.257659+0000","last_became_peered":"2023-11-28T17:41:16.257659+0000","last_unstale":"2023-11-28T17:41:17.215640+0000","last_undegraded":"2023-11-28T17:41:17.215640+0000","last_fullsized":"2023-11-28T17:41:17.215640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:28:12.131592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338433+0000","last_change":"2023-11-28T17:41:17.239069+0000","last_active":"2023-11-28T17:41:18.338433+0000","last_peered":"2023-11-28T17:41:18.338433+0000","last_clean":"2023-11-28T17:41:18.338433+0000","last_became_active":"2023-11-28T17:41:17.238548+0000","last_became_peered":"2023-11-28T17:41:17.238548+0000","last_unstale":"2023-11-28T17:41:18.338433+0000","last_undegraded":"2023-11-28T17:41:18.338433+0000","last_fullsized":"2023-11-28T17:41:18.338433+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:12:56.686814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206759+0000","last_change":"2023-11-28T17:41:12.444311+0000","last_active":"2023-11-28T17:41:17.206759+0000","last_peered":"2023-11-28T17:41:17.206759+0000","last_clean":"2023-11-28T17:41:17.206759+0000","last_became_active":"2023-11-28T17:41:12.442193+0000","last_became_peered":"2023-11-28T17:41:12.442193+0000","last_unstale":"2023-11-28T17:41:17.206759+0000","last_undegraded":"2023-11-28T17:41:17.206759+0000","last_fullsized":"2023-11-28T17:41:17.206759+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:10.865936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:10.865936+0000","last_clean_scrub_stamp":"2023-11-28T17:41:10.865936+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:12:48.912392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206457+0000","last_change":"2023-11-28T17:41:16.253023+0000","last_active":"2023-11-28T17:41:18.206457+0000","last_peered":"2023-11-28T17:41:18.206457+0000","last_clean":"2023-11-28T17:41:18.206457+0000","last_became_active":"2023-11-28T17:41:16.245730+0000","last_became_peered":"2023-11-28T17:41:16.245730+0000","last_unstale":"2023-11-28T17:41:18.206457+0000","last_undegraded":"2023-11-28T17:41:18.206457+0000","last_fullsized":"2023-11-28T17:41:18.206457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:05.970917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568226+0000","last_change":"2023-11-28T17:41:17.246677+0000","last_active":"2023-11-28T17:41:18.568226+0000","last_peered":"2023-11-28T17:41:18.568226+0000","last_clean":"2023-11-28T17:41:18.568226+0000","last_became_active":"2023-11-28T17:41:17.246373+0000","last_became_peered":"2023-11-28T17:41:17.246373+0000","last_unstale":"2023-11-28T17:41:18.568226+0000","last_undegraded":"2023-11-28T17:41:18.568226+0000","last_fullsized":"2023-11-28T17:41:18.568226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:22:18.623331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206504+0000","last_change":"2023-11-28T17:41:16.272843+0000","last_active":"2023-11-28T17:41:17.206504+0000","last_peered":"2023-11-28T17:41:17.206504+0000","last_clean":"2023-11-28T17:41:17.206504+0000","last_became_active":"2023-11-28T17:41:16.272527+0000","last_became_peered":"2023-11-28T17:41:16.272527+0000","last_unstale":"2023-11-28T17:41:17.206504+0000","last_undegraded":"2023-11-28T17:41:17.206504+0000","last_fullsized":"2023-11-28T17:41:17.206504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:55:52.185953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649479+0000","last_change":"2023-11-28T17:41:17.240890+0000","last_active":"2023-11-28T17:41:18.649479+0000","last_peered":"2023-11-28T17:41:18.649479+0000","last_clean":"2023-11-28T17:41:18.649479+0000","last_became_active":"2023-11-28T17:41:17.240571+0000","last_became_peered":"2023-11-28T17:41:17.240571+0000","last_unstale":"2023-11-28T17:41:18.649479+0000","last_undegraded":"2023-11-28T17:41:18.649479+0000","last_fullsized":"2023-11-28T17:41:18.649479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:58.086322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.216327+0000","last_change":"2023-11-28T17:41:16.251925+0000","last_active":"2023-11-28T17:41:17.216327+0000","last_peered":"2023-11-28T17:41:17.216327+0000","last_clean":"2023-11-28T17:41:17.216327+0000","last_became_active":"2023-11-28T17:41:16.251482+0000","last_became_peered":"2023-11-28T17:41:16.251482+0000","last_unstale":"2023-11-28T17:41:17.216327+0000","last_undegraded":"2023-11-28T17:41:17.216327+0000","last_fullsized":"2023-11-28T17:41:17.216327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:07:09.590250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632279+0000","last_change":"2023-11-28T17:41:17.223810+0000","last_active":"2023-11-28T17:41:19.632279+0000","last_peered":"2023-11-28T17:41:19.632279+0000","last_clean":"2023-11-28T17:41:19.632279+0000","last_became_active":"2023-11-28T17:41:17.223244+0000","last_became_peered":"2023-11-28T17:41:17.223244+0000","last_unstale":"2023-11-28T17:41:19.632279+0000","last_undegraded":"2023-11-28T17:41:19.632279+0000","last_fullsized":"2023-11-28T17:41:19.632279+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:31:22.132844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207190+0000","last_change":"2023-11-28T17:41:16.252402+0000","last_active":"2023-11-28T17:41:18.207190+0000","last_peered":"2023-11-28T17:41:18.207190+0000","last_clean":"2023-11-28T17:41:18.207190+0000","last_became_active":"2023-11-28T17:41:16.252061+0000","last_became_peered":"2023-11-28T17:41:16.252061+0000","last_unstale":"2023-11-28T17:41:18.207190+0000","last_undegraded":"2023-11-28T17:41:18.207190+0000","last_fullsized":"2023-11-28T17:41:18.207190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:20:11.890036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207754+0000","last_change":"2023-11-28T17:41:17.234320+0000","last_active":"2023-11-28T17:41:18.207754+0000","last_peered":"2023-11-28T17:41:18.207754+0000","last_clean":"2023-11-28T17:41:18.207754+0000","last_became_active":"2023-11-28T17:41:17.233738+0000","last_became_peered":"2023-11-28T17:41:17.233738+0000","last_unstale":"2023-11-28T17:41:18.207754+0000","last_undegraded":"2023-11-28T17:41:18.207754+0000","last_fullsized":"2023-11-28T17:41:18.207754+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:31:10.257906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.204406+0000","last_change":"2023-11-28T17:41:16.245394+0000","last_active":"2023-11-28T17:41:17.204406+0000","last_peered":"2023-11-28T17:41:17.204406+0000","last_clean":"2023-11-28T17:41:17.204406+0000","last_became_active":"2023-11-28T17:41:16.242985+0000","last_became_peered":"2023-11-28T17:41:16.242985+0000","last_unstale":"2023-11-28T17:41:17.204406+0000","last_undegraded":"2023-11-28T17:41:17.204406+0000","last_fullsized":"2023-11-28T17:41:17.204406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:48.728960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207612+0000","last_change":"2023-11-28T17:41:17.233987+0000","last_active":"2023-11-28T17:41:18.207612+0000","last_peered":"2023-11-28T17:41:18.207612+0000","last_clean":"2023-11-28T17:41:18.207612+0000","last_became_active":"2023-11-28T17:41:17.233430+0000","last_became_peered":"2023-11-28T17:41:17.233430+0000","last_unstale":"2023-11-28T17:41:18.207612+0000","last_undegraded":"2023-11-28T17:41:18.207612+0000","last_fullsized":"2023-11-28T17:41:18.207612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:51:44.816033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.213771+0000","last_change":"2023-11-28T17:41:16.273047+0000","last_active":"2023-11-28T17:41:17.213771+0000","last_peered":"2023-11-28T17:41:17.213771+0000","last_clean":"2023-11-28T17:41:17.213771+0000","last_became_active":"2023-11-28T17:41:16.272667+0000","last_became_peered":"2023-11-28T17:41:16.272667+0000","last_unstale":"2023-11-28T17:41:17.213771+0000","last_undegraded":"2023-11-28T17:41:17.213771+0000","last_fullsized":"2023-11-28T17:41:17.213771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:19:37.472121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632439+0000","last_change":"2023-11-28T17:41:17.232465+0000","last_active":"2023-11-28T17:41:19.632439+0000","last_peered":"2023-11-28T17:41:19.632439+0000","last_clean":"2023-11-28T17:41:19.632439+0000","last_became_active":"2023-11-28T17:41:17.230496+0000","last_became_peered":"2023-11-28T17:41:17.230496+0000","last_unstale":"2023-11-28T17:41:19.632439+0000","last_undegraded":"2023-11-28T17:41:19.632439+0000","last_fullsized":"2023-11-28T17:41:19.632439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:52:08.590259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206197+0000","last_change":"2023-11-28T17:41:16.258717+0000","last_active":"2023-11-28T17:41:17.206197+0000","last_peered":"2023-11-28T17:41:17.206197+0000","last_clean":"2023-11-28T17:41:17.206197+0000","last_became_active":"2023-11-28T17:41:16.258388+0000","last_became_peered":"2023-11-28T17:41:16.258388+0000","last_unstale":"2023-11-28T17:41:17.206197+0000","last_undegraded":"2023-11-28T17:41:17.206197+0000","last_fullsized":"2023-11-28T17:41:17.206197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:17.845137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568998+0000","last_change":"2023-11-28T17:41:17.239189+0000","last_active":"2023-11-28T17:41:18.568998+0000","last_peered":"2023-11-28T17:41:18.568998+0000","last_clean":"2023-11-28T17:41:18.568998+0000","last_became_active":"2023-11-28T17:41:17.238829+0000","last_became_peered":"2023-11-28T17:41:17.238829+0000","last_unstale":"2023-11-28T17:41:18.568998+0000","last_undegraded":"2023-11-28T17:41:18.568998+0000","last_fullsized":"2023-11-28T17:41:18.568998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:10.923572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633608+0000","last_change":"2023-11-28T17:41:16.268334+0000","last_active":"2023-11-28T17:41:19.633608+0000","last_peered":"2023-11-28T17:41:19.633608+0000","last_clean":"2023-11-28T17:41:19.633608+0000","last_became_active":"2023-11-28T17:41:16.268015+0000","last_became_peered":"2023-11-28T17:41:16.268015+0000","last_unstale":"2023-11-28T17:41:19.633608+0000","last_undegraded":"2023-11-28T17:41:19.633608+0000","last_fullsized":"2023-11-28T17:41:19.633608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:57:22.031553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650340+0000","last_change":"2023-11-28T17:41:17.225161+0000","last_active":"2023-11-28T17:41:18.650340+0000","last_peered":"2023-11-28T17:41:18.650340+0000","last_clean":"2023-11-28T17:41:18.650340+0000","last_became_active":"2023-11-28T17:41:17.224420+0000","last_became_peered":"2023-11-28T17:41:17.224420+0000","last_unstale":"2023-11-28T17:41:18.650340+0000","last_undegraded":"2023-11-28T17:41:18.650340+0000","last_fullsized":"2023-11-28T17:41:18.650340+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:25:11.426980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206160+0000","last_change":"2023-11-28T17:41:16.272884+0000","last_active":"2023-11-28T17:41:17.206160+0000","last_peered":"2023-11-28T17:41:17.206160+0000","last_clean":"2023-11-28T17:41:17.206160+0000","last_became_active":"2023-11-28T17:41:16.272557+0000","last_became_peered":"2023-11-28T17:41:16.272557+0000","last_unstale":"2023-11-28T17:41:17.206160+0000","last_undegraded":"2023-11-28T17:41:17.206160+0000","last_fullsized":"2023-11-28T17:41:17.206160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:56.634559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206149+0000","last_change":"2023-11-28T17:41:17.238145+0000","last_active":"2023-11-28T17:41:18.206149+0000","last_peered":"2023-11-28T17:41:18.206149+0000","last_clean":"2023-11-28T17:41:18.206149+0000","last_became_active":"2023-11-28T17:41:17.237774+0000","last_became_peered":"2023-11-28T17:41:17.237774+0000","last_unstale":"2023-11-28T17:41:18.206149+0000","last_undegraded":"2023-11-28T17:41:18.206149+0000","last_fullsized":"2023-11-28T17:41:18.206149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:44.723180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.204215+0000","last_change":"2023-11-28T17:41:16.262318+0000","last_active":"2023-11-28T17:41:17.204215+0000","last_peered":"2023-11-28T17:41:17.204215+0000","last_clean":"2023-11-28T17:41:17.204215+0000","last_became_active":"2023-11-28T17:41:16.262014+0000","last_became_peered":"2023-11-28T17:41:16.262014+0000","last_unstale":"2023-11-28T17:41:17.204215+0000","last_undegraded":"2023-11-28T17:41:17.204215+0000","last_fullsized":"2023-11-28T17:41:17.204215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:17:15.470393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568031+0000","last_change":"2023-11-28T17:41:17.229233+0000","last_active":"2023-11-28T17:41:18.568031+0000","last_peered":"2023-11-28T17:41:18.568031+0000","last_clean":"2023-11-28T17:41:18.568031+0000","last_became_active":"2023-11-28T17:41:17.228885+0000","last_became_peered":"2023-11-28T17:41:17.228885+0000","last_unstale":"2023-11-28T17:41:18.568031+0000","last_undegraded":"2023-11-28T17:41:18.568031+0000","last_fullsized":"2023-11-28T17:41:18.568031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:10:24.247307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.203545+0000","last_change":"2023-11-28T17:41:16.246533+0000","last_active":"2023-11-28T17:41:17.203545+0000","last_peered":"2023-11-28T17:41:17.203545+0000","last_clean":"2023-11-28T17:41:17.203545+0000","last_became_active":"2023-11-28T17:41:16.244067+0000","last_became_peered":"2023-11-28T17:41:16.244067+0000","last_unstale":"2023-11-28T17:41:17.203545+0000","last_undegraded":"2023-11-28T17:41:17.203545+0000","last_fullsized":"2023-11-28T17:41:17.203545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:02:20.862297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568303+0000","last_change":"2023-11-28T17:41:17.236159+0000","last_active":"2023-11-28T17:41:18.568303+0000","last_peered":"2023-11-28T17:41:18.568303+0000","last_clean":"2023-11-28T17:41:18.568303+0000","last_became_active":"2023-11-28T17:41:17.235700+0000","last_became_peered":"2023-11-28T17:41:17.235700+0000","last_unstale":"2023-11-28T17:41:18.568303+0000","last_undegraded":"2023-11-28T17:41:18.568303+0000","last_fullsized":"2023-11-28T17:41:18.568303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:21.900805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.209240+0000","last_change":"2023-11-28T17:41:16.268969+0000","last_active":"2023-11-28T17:41:17.209240+0000","last_peered":"2023-11-28T17:41:17.209240+0000","last_clean":"2023-11-28T17:41:17.209240+0000","last_became_active":"2023-11-28T17:41:16.268717+0000","last_became_peered":"2023-11-28T17:41:16.268717+0000","last_unstale":"2023-11-28T17:41:17.209240+0000","last_undegraded":"2023-11-28T17:41:17.209240+0000","last_fullsized":"2023-11-28T17:41:17.209240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:03:29.120624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633192+0000","last_change":"2023-11-28T17:41:17.244376+0000","last_active":"2023-11-28T17:41:19.633192+0000","last_peered":"2023-11-28T17:41:19.633192+0000","last_clean":"2023-11-28T17:41:19.633192+0000","last_became_active":"2023-11-28T17:41:17.244034+0000","last_became_peered":"2023-11-28T17:41:17.244034+0000","last_unstale":"2023-11-28T17:41:19.633192+0000","last_undegraded":"2023-11-28T17:41:19.633192+0000","last_fullsized":"2023-11-28T17:41:19.633192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:15:56.198732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208198+0000","last_change":"2023-11-28T17:41:17.233749+0000","last_active":"2023-11-28T17:41:18.208198+0000","last_peered":"2023-11-28T17:41:18.208198+0000","last_clean":"2023-11-28T17:41:18.208198+0000","last_became_active":"2023-11-28T17:41:17.232992+0000","last_became_peered":"2023-11-28T17:41:17.232992+0000","last_unstale":"2023-11-28T17:41:18.208198+0000","last_undegraded":"2023-11-28T17:41:18.208198+0000","last_fullsized":"2023-11-28T17:41:18.208198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:00:43.698244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.207649+0000","last_change":"2023-11-28T17:41:16.269465+0000","last_active":"2023-11-28T17:41:17.207649+0000","last_peered":"2023-11-28T17:41:17.207649+0000","last_clean":"2023-11-28T17:41:17.207649+0000","last_became_active":"2023-11-28T17:41:16.269147+0000","last_became_peered":"2023-11-28T17:41:16.269147+0000","last_unstale":"2023-11-28T17:41:17.207649+0000","last_undegraded":"2023-11-28T17:41:17.207649+0000","last_fullsized":"2023-11-28T17:41:17.207649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:57:58.121232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207346+0000","last_change":"2023-11-28T17:41:17.243498+0000","last_active":"2023-11-28T17:41:18.207346+0000","last_peered":"2023-11-28T17:41:18.207346+0000","last_clean":"2023-11-28T17:41:18.207346+0000","last_became_active":"2023-11-28T17:41:17.243169+0000","last_became_peered":"2023-11-28T17:41:17.243169+0000","last_unstale":"2023-11-28T17:41:18.207346+0000","last_undegraded":"2023-11-28T17:41:18.207346+0000","last_fullsized":"2023-11-28T17:41:18.207346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:11:31.509921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.214303+0000","last_change":"2023-11-28T17:41:16.270231+0000","last_active":"2023-11-28T17:41:17.214303+0000","last_peered":"2023-11-28T17:41:17.214303+0000","last_clean":"2023-11-28T17:41:17.214303+0000","last_became_active":"2023-11-28T17:41:16.269938+0000","last_became_peered":"2023-11-28T17:41:16.269938+0000","last_unstale":"2023-11-28T17:41:17.214303+0000","last_undegraded":"2023-11-28T17:41:17.214303+0000","last_fullsized":"2023-11-28T17:41:17.214303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:45.654658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650357+0000","last_change":"2023-11-28T17:41:17.245856+0000","last_active":"2023-11-28T17:41:18.650357+0000","last_peered":"2023-11-28T17:41:18.650357+0000","last_clean":"2023-11-28T17:41:18.650357+0000","last_became_active":"2023-11-28T17:41:17.245624+0000","last_became_peered":"2023-11-28T17:41:17.245624+0000","last_unstale":"2023-11-28T17:41:18.650357+0000","last_undegraded":"2023-11-28T17:41:18.650357+0000","last_fullsized":"2023-11-28T17:41:18.650357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:03:10.206506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.228705+0000","last_change":"2023-11-28T17:41:16.260989+0000","last_active":"2023-11-28T17:41:17.228705+0000","last_peered":"2023-11-28T17:41:17.228705+0000","last_clean":"2023-11-28T17:41:17.228705+0000","last_became_active":"2023-11-28T17:41:16.260686+0000","last_became_peered":"2023-11-28T17:41:16.260686+0000","last_unstale":"2023-11-28T17:41:17.228705+0000","last_undegraded":"2023-11-28T17:41:17.228705+0000","last_fullsized":"2023-11-28T17:41:17.228705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:58:44.193791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568672+0000","last_change":"2023-11-28T17:41:17.240803+0000","last_active":"2023-11-28T17:41:18.568672+0000","last_peered":"2023-11-28T17:41:18.568672+0000","last_clean":"2023-11-28T17:41:18.568672+0000","last_became_active":"2023-11-28T17:41:17.240442+0000","last_became_peered":"2023-11-28T17:41:17.240442+0000","last_unstale":"2023-11-28T17:41:18.568672+0000","last_undegraded":"2023-11-28T17:41:18.568672+0000","last_fullsized":"2023-11-28T17:41:18.568672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:28:21.223788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.217458+0000","last_change":"2023-11-28T17:41:16.266520+0000","last_active":"2023-11-28T17:41:17.217458+0000","last_peered":"2023-11-28T17:41:17.217458+0000","last_clean":"2023-11-28T17:41:17.217458+0000","last_became_active":"2023-11-28T17:41:16.266245+0000","last_became_peered":"2023-11-28T17:41:16.266245+0000","last_unstale":"2023-11-28T17:41:17.217458+0000","last_undegraded":"2023-11-28T17:41:17.217458+0000","last_fullsized":"2023-11-28T17:41:17.217458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:18:28.732421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65062,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509445,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":286720,"data_stored":666900,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.51}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.72}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1070000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.839}]}]},{"osd":6,"up_from":15,"seq":64424509443,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":70232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1619999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509444,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":76532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.24}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4470000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.377}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.679}]}]},{"osd":4,"up_from":15,"seq":64424509444,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":76532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.377}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.482}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.756}]}]},{"osd":3,"up_from":15,"seq":64424509443,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":76532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.397}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5409999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.605}]}]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":76532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.603}]}]},{"osd":0,"up_from":15,"seq":64424509445,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":286720,"data_stored":666900,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.514}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7869999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5409999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":63457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6040000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0329999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7869999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7450000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T17:41:23.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:41:23.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.909+0000 7fc3bece0640 1 Processor -- start 2023-11-28T17:41:23.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.909+0000 7fc3bece0640 1 -- start start 2023-11-28T17:41:23.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.911+0000 7fc3bece0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3b80c4b20 0x7fc3b815dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:23.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.911+0000 7fc3bece0640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3b80c5660 0x7fc3b80c5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:23.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.911+0000 7fc3bece0640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b815e200 con 0x7fc3b8153fe0 2023-11-28T17:41:23.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.911+0000 7fc3bece0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b80c4800 con 0x7fc3b80c4b20 2023-11-28T17:41:23.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.911+0000 7fc3bece0640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b80c51d0 con 0x7fc3b80c5660 2023-11-28T17:41:23.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.911+0000 7fc3b7fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3b80c4b20 0x7fc3b815dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:23.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.911+0000 7fc3b7fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3b80c4b20 0x7fc3b815dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49266/0 (socket says 172.21.15.98:49266) 2023-11-28T17:41:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.911+0000 7fc3b7fff640 1 -- 172.21.15.98:0/582528185 learned_addr learned my addr 172.21.15.98:0/582528185 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.912+0000 7fc3bd27d640 1 --2- 172.21.15.98:0/582528185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3b80c5660 0x7fc3b80c5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.912+0000 7fc3b7fff640 1 -- 172.21.15.98:0/582528185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3b80c5660 msgr2=0x7fc3b80c5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.912+0000 7fc3b7fff640 1 --2- 172.21.15.98:0/582528185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3b80c5660 0x7fc3b80c5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.912+0000 7fc3b7fff640 1 -- 172.21.15.98:0/582528185 >> v1:172.21.15.113:6789/0 conn(0x7fc3b8153fe0 legacy=0x7fc3b80ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.912+0000 7fc3b7fff640 1 -- 172.21.15.98:0/582528185 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b80c6030 con 0x7fc3b80c4b20 2023-11-28T17:41:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.913+0000 7fc3b7fff640 1 --2- 172.21.15.98:0/582528185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b80c4b20 0x7fc3b815dcc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4009870 tx=0x7fc3a402ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b529b6c33697a608 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.913+0000 7fc3b77fe640 1 -- 172.21.15.98:0/582528185 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a402f830 con 0x7fc3b80c4b20 2023-11-28T17:41:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.913+0000 7fc3b77fe640 1 -- 172.21.15.98:0/582528185 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a402f990 con 0x7fc3b80c4b20 2023-11-28T17:41:23.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.913+0000 7fc3b77fe640 1 -- 172.21.15.98:0/582528185 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a402fd20 con 0x7fc3b80c4b20 2023-11-28T17:41:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.914+0000 7fc3bece0640 1 -- 172.21.15.98:0/582528185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b80c4b20 msgr2=0x7fc3b815dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.914+0000 7fc3bece0640 1 --2- 172.21.15.98:0/582528185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b80c4b20 0x7fc3b815dcc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4009870 tx=0x7fc3a402ba10 comp rx=0 tx=0).stop 2023-11-28T17:41:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.914+0000 7fc3bece0640 1 -- 172.21.15.98:0/582528185 shutdown_connections 2023-11-28T17:41:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.914+0000 7fc3bece0640 1 --2- 172.21.15.98:0/582528185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3b80c5660 0x7fc3b80c5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.914+0000 7fc3bece0640 1 --2- 172.21.15.98:0/582528185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b80c4b20 0x7fc3b815dcc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.914+0000 7fc3bece0640 1 -- 172.21.15.98:0/582528185 >> 172.21.15.98:0/582528185 conn(0x7fc3b8066230 msgr2=0x7fc3b8066640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.914+0000 7fc3bece0640 1 -- 172.21.15.98:0/582528185 shutdown_connections 2023-11-28T17:41:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.914+0000 7fc3bece0640 1 -- 172.21.15.98:0/582528185 wait complete. 2023-11-28T17:41:23.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.916+0000 7fc3bece0640 1 Processor -- start 2023-11-28T17:41:23.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.916+0000 7fc3bece0640 1 -- start start 2023-11-28T17:41:23.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.917+0000 7fc3bece0640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3b80c4b20 0x7fc3b81f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:23.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.917+0000 7fc3bece0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3b80c5660 0x7fc3b8156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:23.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.917+0000 7fc3bece0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b8153fe0 0x7fc3b81e6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:23.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.917+0000 7fc3bece0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b8159010 con 0x7fc3b8153fe0 2023-11-28T17:41:23.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.917+0000 7fc3bece0640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b8150a80 con 0x7fc3b80c5660 2023-11-28T17:41:23.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.917+0000 7fc3bece0640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b815bba0 con 0x7fc3b80c4b20 2023-11-28T17:41:23.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.917+0000 7fc3bd27d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b8153fe0 0x7fc3b81e6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:23.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.917+0000 7fc3bd27d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b8153fe0 0x7fc3b81e6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49268/0 (socket says 172.21.15.98:49268) 2023-11-28T17:41:23.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.917+0000 7fc3bd27d640 1 -- 172.21.15.98:0/2965634100 learned_addr learned my addr 172.21.15.98:0/2965634100 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:23.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.917+0000 7fc3b7fff640 1 --2- 172.21.15.98:0/2965634100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3b80c5660 0x7fc3b8156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:23.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.917+0000 7fc3bca7c640 1 --2- 172.21.15.98:0/2965634100 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3b80c4b20 0x7fc3b81f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:23.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.918+0000 7fc3bd27d640 1 -- 172.21.15.98:0/2965634100 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3b80c4b20 msgr2=0x7fc3b81f3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:23.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.918+0000 7fc3bd27d640 1 --2- 172.21.15.98:0/2965634100 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3b80c4b20 0x7fc3b81f3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.918+0000 7fc3bd27d640 1 -- 172.21.15.98:0/2965634100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3b80c5660 msgr2=0x7fc3b8156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:23.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.918+0000 7fc3bd27d640 1 --2- 172.21.15.98:0/2965634100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3b80c5660 0x7fc3b8156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:23.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.918+0000 7fc3bd27d640 1 -- 172.21.15.98:0/2965634100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a40094d0 con 0x7fc3b8153fe0 2023-11-28T17:41:23.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.918+0000 7fc3b7fff640 1 --2- 172.21.15.98:0/2965634100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3b80c5660 0x7fc3b8156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:23.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.918+0000 7fc3bd27d640 1 --2- 172.21.15.98:0/2965634100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b8153fe0 0x7fc3b81e6ab0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0007e40 tx=0x7fc3b000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:23.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.918+0000 7fc3b57fa640 1 -- 172.21.15.98:0/2965634100 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0014620 con 0x7fc3b8153fe0 2023-11-28T17:41:23.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.918+0000 7fc3b57fa640 1 -- 172.21.15.98:0/2965634100 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b000ee90 con 0x7fc3b8153fe0 2023-11-28T17:41:23.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.918+0000 7fc3b57fa640 1 -- 172.21.15.98:0/2965634100 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b00149e0 con 0x7fc3b8153fe0 2023-11-28T17:41:23.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.918+0000 7fc3bece0640 1 -- 172.21.15.98:0/2965634100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b81f32a0 con 0x7fc3b8153fe0 2023-11-28T17:41:23.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.918+0000 7fc3bece0640 1 -- 172.21.15.98:0/2965634100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b815b8c0 con 0x7fc3b8153fe0 2023-11-28T17:41:23.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.920+0000 7fc3bece0640 1 -- 172.21.15.98:0/2965634100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b81f3430 con 0x7fc3b8153fe0 2023-11-28T17:41:23.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.920+0000 7fc3b57fa640 1 -- 172.21.15.98:0/2965634100 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fc3b0014b40 con 0x7fc3b8153fe0 2023-11-28T17:41:23.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.921+0000 7fc3b57fa640 1 --2- 172.21.15.98:0/2965634100 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc39c06b2a0 0x7fc39c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:23.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.921+0000 7fc3b57fa640 1 -- 172.21.15.98:0/2965634100 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3b008a500 con 0x7fc3b8153fe0 2023-11-28T17:41:23.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.922+0000 7fc3bca7c640 1 --2- 172.21.15.98:0/2965634100 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc39c06b2a0 0x7fc39c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:23.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.922+0000 7fc3bca7c640 1 --2- 172.21.15.98:0/2965634100 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc39c06b2a0 0x7fc39c06d760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac000f60 tx=0x7fc3ac009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:23.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:23.926+0000 7fc3b57fa640 1 -- 172.21.15.98:0/2965634100 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3b0055400 con 0x7fc3b8153fe0 2023-11-28T17:41:24.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.098+0000 7fc3bece0640 1 -- 172.21.15.98:0/2965634100 --> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc3b815a370 con 0x7fc39c06b2a0 2023-11-28T17:41:24.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.118+0000 7fc3b57fa640 1 -- 172.21.15.98:0/2965634100 <== mgr.4098 v2:172.21.15.98:6832/2274392793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264402 (secure 0 0 0) 0x7fc3ac0012f0 con 0x7fc39c06b2a0 2023-11-28T17:41:24.122 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:24.125 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-11-28T17:41:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.124+0000 7fc3bece0640 1 -- 172.21.15.98:0/2965634100 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc39c06b2a0 msgr2=0x7fc39c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.124+0000 7fc3bece0640 1 --2- 172.21.15.98:0/2965634100 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc39c06b2a0 0x7fc39c06d760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac000f60 tx=0x7fc3ac009040 comp rx=0 tx=0).stop 2023-11-28T17:41:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.124+0000 7fc3bece0640 1 -- 172.21.15.98:0/2965634100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b8153fe0 msgr2=0x7fc3b81e6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:24.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.124+0000 7fc3bece0640 1 --2- 172.21.15.98:0/2965634100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b8153fe0 0x7fc3b81e6ab0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0007e40 tx=0x7fc3b000e4b0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.125+0000 7fc3bece0640 1 -- 172.21.15.98:0/2965634100 shutdown_connections 2023-11-28T17:41:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.125+0000 7fc3bece0640 1 --2- 172.21.15.98:0/2965634100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3b8153fe0 0x7fc3b81e6ab0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.125+0000 7fc3bece0640 1 --2- 172.21.15.98:0/2965634100 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc39c06b2a0 0x7fc39c06d760 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.125+0000 7fc3bece0640 1 --2- 172.21.15.98:0/2965634100 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3b80c5660 0x7fc3b8156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.125+0000 7fc3bece0640 1 --2- 172.21.15.98:0/2965634100 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3b80c4b20 0x7fc3b81f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.125+0000 7fc3bece0640 1 -- 172.21.15.98:0/2965634100 >> 172.21.15.98:0/2965634100 conn(0x7fc3b8066230 msgr2=0x7fc3b8152e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.125+0000 7fc3bece0640 1 -- 172.21.15.98:0/2965634100 shutdown_connections 2023-11-28T17:41:24.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.125+0000 7fc3bece0640 1 -- 172.21.15.98:0/2965634100 wait complete. 2023-11-28T17:41:24.144 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-11-28T17:41:22.235317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218012,"kb_used_data":1820,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756708,"statfs":{"total":773094113280,"available":772870868992,"internally_reserved":0,"allocated":1863680,"data_stored":1773617,"data_compressed":65062,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.011876"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339217+0000","last_change":"2023-11-28T17:41:17.238145+0000","last_active":"2023-11-28T17:41:18.339217+0000","last_peered":"2023-11-28T17:41:18.339217+0000","last_clean":"2023-11-28T17:41:18.339217+0000","last_became_active":"2023-11-28T17:41:17.237776+0000","last_became_peered":"2023-11-28T17:41:17.237776+0000","last_unstale":"2023-11-28T17:41:18.339217+0000","last_undegraded":"2023-11-28T17:41:18.339217+0000","last_fullsized":"2023-11-28T17:41:18.339217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:41:06.902665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632799+0000","last_change":"2023-11-28T17:41:17.231913+0000","last_active":"2023-11-28T17:41:19.632799+0000","last_peered":"2023-11-28T17:41:19.632799+0000","last_clean":"2023-11-28T17:41:19.632799+0000","last_became_active":"2023-11-28T17:41:17.231622+0000","last_became_peered":"2023-11-28T17:41:17.231622+0000","last_unstale":"2023-11-28T17:41:19.632799+0000","last_undegraded":"2023-11-28T17:41:19.632799+0000","last_fullsized":"2023-11-28T17:41:19.632799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:06.911223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649842+0000","last_change":"2023-11-28T17:41:17.230232+0000","last_active":"2023-11-28T17:41:18.649842+0000","last_peered":"2023-11-28T17:41:18.649842+0000","last_clean":"2023-11-28T17:41:18.649842+0000","last_became_active":"2023-11-28T17:41:17.229795+0000","last_became_peered":"2023-11-28T17:41:17.229795+0000","last_unstale":"2023-11-28T17:41:18.649842+0000","last_undegraded":"2023-11-28T17:41:18.649842+0000","last_fullsized":"2023-11-28T17:41:18.649842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:09:09.507547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207139+0000","last_change":"2023-11-28T17:41:17.234948+0000","last_active":"2023-11-28T17:41:18.207139+0000","last_peered":"2023-11-28T17:41:18.207139+0000","last_clean":"2023-11-28T17:41:18.207139+0000","last_became_active":"2023-11-28T17:41:17.233674+0000","last_became_peered":"2023-11-28T17:41:17.233674+0000","last_unstale":"2023-11-28T17:41:18.207139+0000","last_undegraded":"2023-11-28T17:41:18.207139+0000","last_fullsized":"2023-11-28T17:41:18.207139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:45:10.831719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338225+0000","last_change":"2023-11-28T17:41:17.233908+0000","last_active":"2023-11-28T17:41:18.338225+0000","last_peered":"2023-11-28T17:41:18.338225+0000","last_clean":"2023-11-28T17:41:18.338225+0000","last_became_active":"2023-11-28T17:41:17.233589+0000","last_became_peered":"2023-11-28T17:41:17.233589+0000","last_unstale":"2023-11-28T17:41:18.338225+0000","last_undegraded":"2023-11-28T17:41:18.338225+0000","last_fullsized":"2023-11-28T17:41:18.338225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:21:37.379103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206516+0000","last_change":"2023-11-28T17:41:17.239186+0000","last_active":"2023-11-28T17:41:18.206516+0000","last_peered":"2023-11-28T17:41:18.206516+0000","last_clean":"2023-11-28T17:41:18.206516+0000","last_became_active":"2023-11-28T17:41:17.233898+0000","last_became_peered":"2023-11-28T17:41:17.233898+0000","last_unstale":"2023-11-28T17:41:18.206516+0000","last_undegraded":"2023-11-28T17:41:18.206516+0000","last_fullsized":"2023-11-28T17:41:18.206516+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:54.918589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338198+0000","last_change":"2023-11-28T17:41:17.227923+0000","last_active":"2023-11-28T17:41:18.338198+0000","last_peered":"2023-11-28T17:41:18.338198+0000","last_clean":"2023-11-28T17:41:18.338198+0000","last_became_active":"2023-11-28T17:41:17.227642+0000","last_became_peered":"2023-11-28T17:41:17.227642+0000","last_unstale":"2023-11-28T17:41:18.338198+0000","last_undegraded":"2023-11-28T17:41:18.338198+0000","last_fullsized":"2023-11-28T17:41:18.338198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:33:29.400533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567710+0000","last_change":"2023-11-28T17:41:17.239818+0000","last_active":"2023-11-28T17:41:18.567710+0000","last_peered":"2023-11-28T17:41:18.567710+0000","last_clean":"2023-11-28T17:41:18.567710+0000","last_became_active":"2023-11-28T17:41:17.239594+0000","last_became_peered":"2023-11-28T17:41:17.239594+0000","last_unstale":"2023-11-28T17:41:18.567710+0000","last_undegraded":"2023-11-28T17:41:18.567710+0000","last_fullsized":"2023-11-28T17:41:18.567710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:23.969471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339318+0000","last_change":"2023-11-28T17:41:17.243214+0000","last_active":"2023-11-28T17:41:18.339318+0000","last_peered":"2023-11-28T17:41:18.339318+0000","last_clean":"2023-11-28T17:41:18.339318+0000","last_became_active":"2023-11-28T17:41:17.242933+0000","last_became_peered":"2023-11-28T17:41:17.242933+0000","last_unstale":"2023-11-28T17:41:18.339318+0000","last_undegraded":"2023-11-28T17:41:18.339318+0000","last_fullsized":"2023-11-28T17:41:18.339318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:47:47.697137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633703+0000","last_change":"2023-11-28T17:41:17.242507+0000","last_active":"2023-11-28T17:41:19.633703+0000","last_peered":"2023-11-28T17:41:19.633703+0000","last_clean":"2023-11-28T17:41:19.633703+0000","last_became_active":"2023-11-28T17:41:17.242196+0000","last_became_peered":"2023-11-28T17:41:17.242196+0000","last_unstale":"2023-11-28T17:41:19.633703+0000","last_undegraded":"2023-11-28T17:41:19.633703+0000","last_fullsized":"2023-11-28T17:41:19.633703+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:06.103318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633386+0000","last_change":"2023-11-28T17:41:17.244694+0000","last_active":"2023-11-28T17:41:19.633386+0000","last_peered":"2023-11-28T17:41:19.633386+0000","last_clean":"2023-11-28T17:41:19.633386+0000","last_became_active":"2023-11-28T17:41:17.244301+0000","last_became_peered":"2023-11-28T17:41:17.244301+0000","last_unstale":"2023-11-28T17:41:19.633386+0000","last_undegraded":"2023-11-28T17:41:19.633386+0000","last_fullsized":"2023-11-28T17:41:19.633386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:40.030685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208504+0000","last_change":"2023-11-28T17:41:17.242800+0000","last_active":"2023-11-28T17:41:18.208504+0000","last_peered":"2023-11-28T17:41:18.208504+0000","last_clean":"2023-11-28T17:41:18.208504+0000","last_became_active":"2023-11-28T17:41:17.242463+0000","last_became_peered":"2023-11-28T17:41:17.242463+0000","last_unstale":"2023-11-28T17:41:18.208504+0000","last_undegraded":"2023-11-28T17:41:18.208504+0000","last_fullsized":"2023-11-28T17:41:18.208504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:57:34.802348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568335+0000","last_change":"2023-11-28T17:41:17.232485+0000","last_active":"2023-11-28T17:41:18.568335+0000","last_peered":"2023-11-28T17:41:18.568335+0000","last_clean":"2023-11-28T17:41:18.568335+0000","last_became_active":"2023-11-28T17:41:17.232132+0000","last_became_peered":"2023-11-28T17:41:17.232132+0000","last_unstale":"2023-11-28T17:41:18.568335+0000","last_undegraded":"2023-11-28T17:41:18.568335+0000","last_fullsized":"2023-11-28T17:41:18.568335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:49.292565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649479+0000","last_change":"2023-11-28T17:41:17.234514+0000","last_active":"2023-11-28T17:41:18.649479+0000","last_peered":"2023-11-28T17:41:18.649479+0000","last_clean":"2023-11-28T17:41:18.649479+0000","last_became_active":"2023-11-28T17:41:17.234154+0000","last_became_peered":"2023-11-28T17:41:17.234154+0000","last_unstale":"2023-11-28T17:41:18.649479+0000","last_undegraded":"2023-11-28T17:41:18.649479+0000","last_fullsized":"2023-11-28T17:41:18.649479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:22.430159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632129+0000","last_change":"2023-11-28T17:41:17.225905+0000","last_active":"2023-11-28T17:41:19.632129+0000","last_peered":"2023-11-28T17:41:19.632129+0000","last_clean":"2023-11-28T17:41:19.632129+0000","last_became_active":"2023-11-28T17:41:17.223533+0000","last_became_peered":"2023-11-28T17:41:17.223533+0000","last_unstale":"2023-11-28T17:41:19.632129+0000","last_undegraded":"2023-11-28T17:41:19.632129+0000","last_fullsized":"2023-11-28T17:41:19.632129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:32.314586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.205612+0000","last_change":"2023-11-28T17:41:17.236242+0000","last_active":"2023-11-28T17:41:18.205612+0000","last_peered":"2023-11-28T17:41:18.205612+0000","last_clean":"2023-11-28T17:41:18.205612+0000","last_became_active":"2023-11-28T17:41:17.235890+0000","last_became_peered":"2023-11-28T17:41:17.235890+0000","last_unstale":"2023-11-28T17:41:18.205612+0000","last_undegraded":"2023-11-28T17:41:18.205612+0000","last_fullsized":"2023-11-28T17:41:18.205612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:20:09.004702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208885+0000","last_change":"2023-11-28T17:41:17.234610+0000","last_active":"2023-11-28T17:41:18.208885+0000","last_peered":"2023-11-28T17:41:18.208885+0000","last_clean":"2023-11-28T17:41:18.208885+0000","last_became_active":"2023-11-28T17:41:17.234255+0000","last_became_peered":"2023-11-28T17:41:17.234255+0000","last_unstale":"2023-11-28T17:41:18.208885+0000","last_undegraded":"2023-11-28T17:41:18.208885+0000","last_fullsized":"2023-11-28T17:41:18.208885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:25:51.137148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.234674+0000","last_change":"2023-11-28T17:41:17.231838+0000","last_active":"2023-11-28T17:41:17.234674+0000","last_peered":"2023-11-28T17:41:17.234674+0000","last_clean":"2023-11-28T17:41:17.234674+0000","last_became_active":"2023-11-28T17:41:17.231533+0000","last_became_peered":"2023-11-28T17:41:17.231533+0000","last_unstale":"2023-11-28T17:41:17.234674+0000","last_undegraded":"2023-11-28T17:41:17.234674+0000","last_fullsized":"2023-11-28T17:41:17.234674+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:21:03.904863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339038+0000","last_change":"2023-11-28T17:41:17.238888+0000","last_active":"2023-11-28T17:41:18.339038+0000","last_peered":"2023-11-28T17:41:18.339038+0000","last_clean":"2023-11-28T17:41:18.339038+0000","last_became_active":"2023-11-28T17:41:17.236140+0000","last_became_peered":"2023-11-28T17:41:17.236140+0000","last_unstale":"2023-11-28T17:41:18.339038+0000","last_undegraded":"2023-11-28T17:41:18.339038+0000","last_fullsized":"2023-11-28T17:41:18.339038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:20:57.936710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633201+0000","last_change":"2023-11-28T17:41:17.242150+0000","last_active":"2023-11-28T17:41:19.633201+0000","last_peered":"2023-11-28T17:41:19.633201+0000","last_clean":"2023-11-28T17:41:19.633201+0000","last_became_active":"2023-11-28T17:41:17.241832+0000","last_became_peered":"2023-11-28T17:41:17.241832+0000","last_unstale":"2023-11-28T17:41:19.633201+0000","last_undegraded":"2023-11-28T17:41:19.633201+0000","last_fullsized":"2023-11-28T17:41:19.633201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:30:37.837711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649928+0000","last_change":"2023-11-28T17:41:17.233651+0000","last_active":"2023-11-28T17:41:18.649928+0000","last_peered":"2023-11-28T17:41:18.649928+0000","last_clean":"2023-11-28T17:41:18.649928+0000","last_became_active":"2023-11-28T17:41:17.233393+0000","last_became_peered":"2023-11-28T17:41:17.233393+0000","last_unstale":"2023-11-28T17:41:18.649928+0000","last_undegraded":"2023-11-28T17:41:18.649928+0000","last_fullsized":"2023-11-28T17:41:18.649928+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:52:08.152046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206688+0000","last_change":"2023-11-28T17:41:17.239696+0000","last_active":"2023-11-28T17:41:18.206688+0000","last_peered":"2023-11-28T17:41:18.206688+0000","last_clean":"2023-11-28T17:41:18.206688+0000","last_became_active":"2023-11-28T17:41:17.239449+0000","last_became_peered":"2023-11-28T17:41:17.239449+0000","last_unstale":"2023-11-28T17:41:18.206688+0000","last_undegraded":"2023-11-28T17:41:18.206688+0000","last_fullsized":"2023-11-28T17:41:18.206688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:08:43.019689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567770+0000","last_change":"2023-11-28T17:41:17.244917+0000","last_active":"2023-11-28T17:41:18.567770+0000","last_peered":"2023-11-28T17:41:18.567770+0000","last_clean":"2023-11-28T17:41:18.567770+0000","last_became_active":"2023-11-28T17:41:17.244676+0000","last_became_peered":"2023-11-28T17:41:17.244676+0000","last_unstale":"2023-11-28T17:41:18.567770+0000","last_undegraded":"2023-11-28T17:41:18.567770+0000","last_fullsized":"2023-11-28T17:41:18.567770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:08:10.818556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206063+0000","last_change":"2023-11-28T17:41:17.236488+0000","last_active":"2023-11-28T17:41:18.206063+0000","last_peered":"2023-11-28T17:41:18.206063+0000","last_clean":"2023-11-28T17:41:18.206063+0000","last_became_active":"2023-11-28T17:41:17.235554+0000","last_became_peered":"2023-11-28T17:41:17.235554+0000","last_unstale":"2023-11-28T17:41:18.206063+0000","last_undegraded":"2023-11-28T17:41:18.206063+0000","last_fullsized":"2023-11-28T17:41:18.206063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:29:35.500772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568927+0000","last_change":"2023-11-28T17:41:17.245042+0000","last_active":"2023-11-28T17:41:18.568927+0000","last_peered":"2023-11-28T17:41:18.568927+0000","last_clean":"2023-11-28T17:41:18.568927+0000","last_became_active":"2023-11-28T17:41:17.244729+0000","last_became_peered":"2023-11-28T17:41:17.244729+0000","last_unstale":"2023-11-28T17:41:18.568927+0000","last_undegraded":"2023-11-28T17:41:18.568927+0000","last_fullsized":"2023-11-28T17:41:18.568927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:57:00.728653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633914+0000","last_change":"2023-11-28T17:41:17.249367+0000","last_active":"2023-11-28T17:41:19.633914+0000","last_peered":"2023-11-28T17:41:19.633914+0000","last_clean":"2023-11-28T17:41:19.633914+0000","last_became_active":"2023-11-28T17:41:17.249093+0000","last_became_peered":"2023-11-28T17:41:17.249093+0000","last_unstale":"2023-11-28T17:41:19.633914+0000","last_undegraded":"2023-11-28T17:41:19.633914+0000","last_fullsized":"2023-11-28T17:41:19.633914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:12:41.994354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632733+0000","last_change":"2023-11-28T17:41:17.230256+0000","last_active":"2023-11-28T17:41:19.632733+0000","last_peered":"2023-11-28T17:41:19.632733+0000","last_clean":"2023-11-28T17:41:19.632733+0000","last_became_active":"2023-11-28T17:41:17.229551+0000","last_became_peered":"2023-11-28T17:41:17.229551+0000","last_unstale":"2023-11-28T17:41:19.632733+0000","last_undegraded":"2023-11-28T17:41:19.632733+0000","last_fullsized":"2023-11-28T17:41:19.632733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:54:09.421121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568751+0000","last_change":"2023-11-28T17:41:17.241672+0000","last_active":"2023-11-28T17:41:18.568751+0000","last_peered":"2023-11-28T17:41:18.568751+0000","last_clean":"2023-11-28T17:41:18.568751+0000","last_became_active":"2023-11-28T17:41:17.241366+0000","last_became_peered":"2023-11-28T17:41:17.241366+0000","last_unstale":"2023-11-28T17:41:18.568751+0000","last_undegraded":"2023-11-28T17:41:18.568751+0000","last_fullsized":"2023-11-28T17:41:18.568751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:29:01.718362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650047+0000","last_change":"2023-11-28T17:41:17.229925+0000","last_active":"2023-11-28T17:41:18.650047+0000","last_peered":"2023-11-28T17:41:18.650047+0000","last_clean":"2023-11-28T17:41:18.650047+0000","last_became_active":"2023-11-28T17:41:17.229617+0000","last_became_peered":"2023-11-28T17:41:17.229617+0000","last_unstale":"2023-11-28T17:41:18.650047+0000","last_undegraded":"2023-11-28T17:41:18.650047+0000","last_fullsized":"2023-11-28T17:41:18.650047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:40:14.646558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206950+0000","last_change":"2023-11-28T17:41:17.241933+0000","last_active":"2023-11-28T17:41:18.206950+0000","last_peered":"2023-11-28T17:41:18.206950+0000","last_clean":"2023-11-28T17:41:18.206950+0000","last_became_active":"2023-11-28T17:41:17.241634+0000","last_became_peered":"2023-11-28T17:41:17.241634+0000","last_unstale":"2023-11-28T17:41:18.206950+0000","last_undegraded":"2023-11-28T17:41:18.206950+0000","last_fullsized":"2023-11-28T17:41:18.206950+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:44:01.525920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.238580+0000","last_change":"2023-11-28T17:41:17.233872+0000","last_active":"2023-11-28T17:41:17.238580+0000","last_peered":"2023-11-28T17:41:17.238580+0000","last_clean":"2023-11-28T17:41:17.238580+0000","last_became_active":"2023-11-28T17:41:17.233586+0000","last_became_peered":"2023-11-28T17:41:17.233586+0000","last_unstale":"2023-11-28T17:41:17.238580+0000","last_undegraded":"2023-11-28T17:41:17.238580+0000","last_fullsized":"2023-11-28T17:41:17.238580+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:57:07.694916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633270+0000","last_change":"2023-11-28T17:41:17.236164+0000","last_active":"2023-11-28T17:41:19.633270+0000","last_peered":"2023-11-28T17:41:19.633270+0000","last_clean":"2023-11-28T17:41:19.633270+0000","last_became_active":"2023-11-28T17:41:17.235821+0000","last_became_peered":"2023-11-28T17:41:17.235821+0000","last_unstale":"2023-11-28T17:41:19.633270+0000","last_undegraded":"2023-11-28T17:41:19.633270+0000","last_fullsized":"2023-11-28T17:41:19.633270+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:54:16.568394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206920+0000","last_change":"2023-11-28T17:41:17.235816+0000","last_active":"2023-11-28T17:41:18.206920+0000","last_peered":"2023-11-28T17:41:18.206920+0000","last_clean":"2023-11-28T17:41:18.206920+0000","last_became_active":"2023-11-28T17:41:17.235506+0000","last_became_peered":"2023-11-28T17:41:17.235506+0000","last_unstale":"2023-11-28T17:41:18.206920+0000","last_undegraded":"2023-11-28T17:41:18.206920+0000","last_fullsized":"2023-11-28T17:41:18.206920+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:18:00.597727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.228065+0000","last_change":"2023-11-28T17:41:16.266076+0000","last_active":"2023-11-28T17:41:17.228065+0000","last_peered":"2023-11-28T17:41:17.228065+0000","last_clean":"2023-11-28T17:41:17.228065+0000","last_became_active":"2023-11-28T17:41:16.265837+0000","last_became_peered":"2023-11-28T17:41:16.265837+0000","last_unstale":"2023-11-28T17:41:17.228065+0000","last_undegraded":"2023-11-28T17:41:17.228065+0000","last_fullsized":"2023-11-28T17:41:17.228065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:01:51.477975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650250+0000","last_change":"2023-11-28T17:41:17.238915+0000","last_active":"2023-11-28T17:41:18.650250+0000","last_peered":"2023-11-28T17:41:18.650250+0000","last_clean":"2023-11-28T17:41:18.650250+0000","last_became_active":"2023-11-28T17:41:17.238581+0000","last_became_peered":"2023-11-28T17:41:17.238581+0000","last_unstale":"2023-11-28T17:41:18.650250+0000","last_undegraded":"2023-11-28T17:41:18.650250+0000","last_fullsized":"2023-11-28T17:41:18.650250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:52:31.640239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632930+0000","last_change":"2023-11-28T17:41:16.261044+0000","last_active":"2023-11-28T17:41:19.632930+0000","last_peered":"2023-11-28T17:41:19.632930+0000","last_clean":"2023-11-28T17:41:19.632930+0000","last_became_active":"2023-11-28T17:41:16.260752+0000","last_became_peered":"2023-11-28T17:41:16.260752+0000","last_unstale":"2023-11-28T17:41:19.632930+0000","last_undegraded":"2023-11-28T17:41:19.632930+0000","last_fullsized":"2023-11-28T17:41:19.632930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:07:01.443140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.227924+0000","last_change":"2023-11-28T17:41:16.261095+0000","last_active":"2023-11-28T17:41:17.227924+0000","last_peered":"2023-11-28T17:41:17.227924+0000","last_clean":"2023-11-28T17:41:17.227924+0000","last_became_active":"2023-11-28T17:41:16.260793+0000","last_became_peered":"2023-11-28T17:41:16.260793+0000","last_unstale":"2023-11-28T17:41:17.227924+0000","last_undegraded":"2023-11-28T17:41:17.227924+0000","last_fullsized":"2023-11-28T17:41:17.227924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:54:16.821314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567897+0000","last_change":"2023-11-28T17:41:17.236517+0000","last_active":"2023-11-28T17:41:18.567897+0000","last_peered":"2023-11-28T17:41:18.567897+0000","last_clean":"2023-11-28T17:41:18.567897+0000","last_became_active":"2023-11-28T17:41:17.235925+0000","last_became_peered":"2023-11-28T17:41:17.235925+0000","last_unstale":"2023-11-28T17:41:18.567897+0000","last_undegraded":"2023-11-28T17:41:18.567897+0000","last_fullsized":"2023-11-28T17:41:18.567897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:45:09.817844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.205947+0000","last_change":"2023-11-28T17:41:16.258176+0000","last_active":"2023-11-28T17:41:18.205947+0000","last_peered":"2023-11-28T17:41:18.205947+0000","last_clean":"2023-11-28T17:41:18.205947+0000","last_became_active":"2023-11-28T17:41:16.257838+0000","last_became_peered":"2023-11-28T17:41:16.257838+0000","last_unstale":"2023-11-28T17:41:18.205947+0000","last_undegraded":"2023-11-28T17:41:18.205947+0000","last_fullsized":"2023-11-28T17:41:18.205947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:31:01.443661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.631980+0000","last_change":"2023-11-28T17:41:17.227823+0000","last_active":"2023-11-28T17:41:19.631980+0000","last_peered":"2023-11-28T17:41:19.631980+0000","last_clean":"2023-11-28T17:41:19.631980+0000","last_became_active":"2023-11-28T17:41:17.227545+0000","last_became_peered":"2023-11-28T17:41:17.227545+0000","last_unstale":"2023-11-28T17:41:19.631980+0000","last_undegraded":"2023-11-28T17:41:19.631980+0000","last_fullsized":"2023-11-28T17:41:19.631980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:20:33.253155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633476+0000","last_change":"2023-11-28T17:41:16.262372+0000","last_active":"2023-11-28T17:41:19.633476+0000","last_peered":"2023-11-28T17:41:19.633476+0000","last_clean":"2023-11-28T17:41:19.633476+0000","last_became_active":"2023-11-28T17:41:16.262029+0000","last_became_peered":"2023-11-28T17:41:16.262029+0000","last_unstale":"2023-11-28T17:41:19.633476+0000","last_undegraded":"2023-11-28T17:41:19.633476+0000","last_fullsized":"2023-11-28T17:41:19.633476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:46:08.491380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207822+0000","last_change":"2023-11-28T17:41:17.235646+0000","last_active":"2023-11-28T17:41:18.207822+0000","last_peered":"2023-11-28T17:41:18.207822+0000","last_clean":"2023-11-28T17:41:18.207822+0000","last_became_active":"2023-11-28T17:41:17.235330+0000","last_became_peered":"2023-11-28T17:41:17.235330+0000","last_unstale":"2023-11-28T17:41:18.207822+0000","last_undegraded":"2023-11-28T17:41:18.207822+0000","last_fullsized":"2023-11-28T17:41:18.207822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:48:12.717031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207869+0000","last_change":"2023-11-28T17:41:16.262716+0000","last_active":"2023-11-28T17:41:18.207869+0000","last_peered":"2023-11-28T17:41:18.207869+0000","last_clean":"2023-11-28T17:41:18.207869+0000","last_became_active":"2023-11-28T17:41:16.262347+0000","last_became_peered":"2023-11-28T17:41:16.262347+0000","last_unstale":"2023-11-28T17:41:18.207869+0000","last_undegraded":"2023-11-28T17:41:18.207869+0000","last_fullsized":"2023-11-28T17:41:18.207869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:50.685367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650261+0000","last_change":"2023-11-28T17:41:17.245852+0000","last_active":"2023-11-28T17:41:18.650261+0000","last_peered":"2023-11-28T17:41:18.650261+0000","last_clean":"2023-11-28T17:41:18.650261+0000","last_became_active":"2023-11-28T17:41:17.245596+0000","last_became_peered":"2023-11-28T17:41:17.245596+0000","last_unstale":"2023-11-28T17:41:18.650261+0000","last_undegraded":"2023-11-28T17:41:18.650261+0000","last_fullsized":"2023-11-28T17:41:18.650261+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:19:03.484052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.204578+0000","last_change":"2023-11-28T17:41:16.254855+0000","last_active":"2023-11-28T17:41:17.204578+0000","last_peered":"2023-11-28T17:41:17.204578+0000","last_clean":"2023-11-28T17:41:17.204578+0000","last_became_active":"2023-11-28T17:41:16.254557+0000","last_became_peered":"2023-11-28T17:41:16.254557+0000","last_unstale":"2023-11-28T17:41:17.204578+0000","last_undegraded":"2023-11-28T17:41:17.204578+0000","last_fullsized":"2023-11-28T17:41:17.204578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:55:38.565240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208613+0000","last_change":"2023-11-28T17:41:17.249020+0000","last_active":"2023-11-28T17:41:18.208613+0000","last_peered":"2023-11-28T17:41:18.208613+0000","last_clean":"2023-11-28T17:41:18.208613+0000","last_became_active":"2023-11-28T17:41:17.248345+0000","last_became_peered":"2023-11-28T17:41:17.248345+0000","last_unstale":"2023-11-28T17:41:18.208613+0000","last_undegraded":"2023-11-28T17:41:18.208613+0000","last_fullsized":"2023-11-28T17:41:18.208613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:05:09.688116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633384+0000","last_change":"2023-11-28T17:41:16.256485+0000","last_active":"2023-11-28T17:41:19.633384+0000","last_peered":"2023-11-28T17:41:19.633384+0000","last_clean":"2023-11-28T17:41:19.633384+0000","last_became_active":"2023-11-28T17:41:16.256165+0000","last_became_peered":"2023-11-28T17:41:16.256165+0000","last_unstale":"2023-11-28T17:41:19.633384+0000","last_undegraded":"2023-11-28T17:41:19.633384+0000","last_fullsized":"2023-11-28T17:41:19.633384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:35:26.805435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338198+0000","last_change":"2023-11-28T17:41:17.230246+0000","last_active":"2023-11-28T17:41:18.338198+0000","last_peered":"2023-11-28T17:41:18.338198+0000","last_clean":"2023-11-28T17:41:18.338198+0000","last_became_active":"2023-11-28T17:41:17.229981+0000","last_became_peered":"2023-11-28T17:41:17.229981+0000","last_unstale":"2023-11-28T17:41:18.338198+0000","last_undegraded":"2023-11-28T17:41:18.338198+0000","last_fullsized":"2023-11-28T17:41:18.338198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:15:48.047641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.220706+0000","last_change":"2023-11-28T17:41:16.250894+0000","last_active":"2023-11-28T17:41:17.220706+0000","last_peered":"2023-11-28T17:41:17.220706+0000","last_clean":"2023-11-28T17:41:17.220706+0000","last_became_active":"2023-11-28T17:41:16.250547+0000","last_became_peered":"2023-11-28T17:41:16.250547+0000","last_unstale":"2023-11-28T17:41:17.220706+0000","last_undegraded":"2023-11-28T17:41:17.220706+0000","last_fullsized":"2023-11-28T17:41:17.220706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:58:47.412734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649511+0000","last_change":"2023-11-28T17:41:17.237500+0000","last_active":"2023-11-28T17:41:18.649511+0000","last_peered":"2023-11-28T17:41:18.649511+0000","last_clean":"2023-11-28T17:41:18.649511+0000","last_became_active":"2023-11-28T17:41:17.237211+0000","last_became_peered":"2023-11-28T17:41:17.237211+0000","last_unstale":"2023-11-28T17:41:18.649511+0000","last_undegraded":"2023-11-28T17:41:18.649511+0000","last_fullsized":"2023-11-28T17:41:18.649511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:13.675182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207345+0000","last_change":"2023-11-28T17:41:16.258147+0000","last_active":"2023-11-28T17:41:18.207345+0000","last_peered":"2023-11-28T17:41:18.207345+0000","last_clean":"2023-11-28T17:41:18.207345+0000","last_became_active":"2023-11-28T17:41:16.257540+0000","last_became_peered":"2023-11-28T17:41:16.257540+0000","last_unstale":"2023-11-28T17:41:18.207345+0000","last_undegraded":"2023-11-28T17:41:18.207345+0000","last_fullsized":"2023-11-28T17:41:18.207345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:51:11.394507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338684+0000","last_change":"2023-11-28T17:41:17.247768+0000","last_active":"2023-11-28T17:41:18.338684+0000","last_peered":"2023-11-28T17:41:18.338684+0000","last_clean":"2023-11-28T17:41:18.338684+0000","last_became_active":"2023-11-28T17:41:17.247475+0000","last_became_peered":"2023-11-28T17:41:17.247475+0000","last_unstale":"2023-11-28T17:41:18.338684+0000","last_undegraded":"2023-11-28T17:41:18.338684+0000","last_fullsized":"2023-11-28T17:41:18.338684+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:18.555573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632851+0000","last_change":"2023-11-28T17:41:16.262927+0000","last_active":"2023-11-28T17:41:19.632851+0000","last_peered":"2023-11-28T17:41:19.632851+0000","last_clean":"2023-11-28T17:41:19.632851+0000","last_became_active":"2023-11-28T17:41:16.262556+0000","last_became_peered":"2023-11-28T17:41:16.262556+0000","last_unstale":"2023-11-28T17:41:19.632851+0000","last_undegraded":"2023-11-28T17:41:19.632851+0000","last_fullsized":"2023-11-28T17:41:19.632851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:35:54.672978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632921+0000","last_change":"2023-11-28T17:41:17.239625+0000","last_active":"2023-11-28T17:41:19.632921+0000","last_peered":"2023-11-28T17:41:19.632921+0000","last_clean":"2023-11-28T17:41:19.632921+0000","last_became_active":"2023-11-28T17:41:17.238215+0000","last_became_peered":"2023-11-28T17:41:17.238215+0000","last_unstale":"2023-11-28T17:41:19.632921+0000","last_undegraded":"2023-11-28T17:41:19.632921+0000","last_fullsized":"2023-11-28T17:41:19.632921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:33:31.879592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.216317+0000","last_change":"2023-11-28T17:41:16.250674+0000","last_active":"2023-11-28T17:41:17.216317+0000","last_peered":"2023-11-28T17:41:17.216317+0000","last_clean":"2023-11-28T17:41:17.216317+0000","last_became_active":"2023-11-28T17:41:16.250312+0000","last_became_peered":"2023-11-28T17:41:16.250312+0000","last_unstale":"2023-11-28T17:41:17.216317+0000","last_undegraded":"2023-11-28T17:41:17.216317+0000","last_fullsized":"2023-11-28T17:41:17.216317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:04:05.342081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338315+0000","last_change":"2023-11-28T17:41:17.247675+0000","last_active":"2023-11-28T17:41:18.338315+0000","last_peered":"2023-11-28T17:41:18.338315+0000","last_clean":"2023-11-28T17:41:18.338315+0000","last_became_active":"2023-11-28T17:41:17.247346+0000","last_became_peered":"2023-11-28T17:41:17.247346+0000","last_unstale":"2023-11-28T17:41:18.338315+0000","last_undegraded":"2023-11-28T17:41:18.338315+0000","last_fullsized":"2023-11-28T17:41:18.338315+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:45:03.326583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.234280+0000","last_change":"2023-11-28T17:41:16.260893+0000","last_active":"2023-11-28T17:41:17.234280+0000","last_peered":"2023-11-28T17:41:17.234280+0000","last_clean":"2023-11-28T17:41:17.234280+0000","last_became_active":"2023-11-28T17:41:16.260637+0000","last_became_peered":"2023-11-28T17:41:16.260637+0000","last_unstale":"2023-11-28T17:41:17.234280+0000","last_undegraded":"2023-11-28T17:41:17.234280+0000","last_fullsized":"2023-11-28T17:41:17.234280+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:07:24.032989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338811+0000","last_change":"2023-11-28T17:41:17.249202+0000","last_active":"2023-11-28T17:41:18.338811+0000","last_peered":"2023-11-28T17:41:18.338811+0000","last_clean":"2023-11-28T17:41:18.338811+0000","last_became_active":"2023-11-28T17:41:17.248187+0000","last_became_peered":"2023-11-28T17:41:17.248187+0000","last_unstale":"2023-11-28T17:41:18.338811+0000","last_undegraded":"2023-11-28T17:41:18.338811+0000","last_fullsized":"2023-11-28T17:41:18.338811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:23.973069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.216810+0000","last_change":"2023-11-28T17:41:16.244938+0000","last_active":"2023-11-28T17:41:17.216810+0000","last_peered":"2023-11-28T17:41:17.216810+0000","last_clean":"2023-11-28T17:41:17.216810+0000","last_became_active":"2023-11-28T17:41:16.244586+0000","last_became_peered":"2023-11-28T17:41:16.244586+0000","last_unstale":"2023-11-28T17:41:17.216810+0000","last_undegraded":"2023-11-28T17:41:17.216810+0000","last_fullsized":"2023-11-28T17:41:17.216810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:38.421966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339409+0000","last_change":"2023-11-28T17:41:17.248488+0000","last_active":"2023-11-28T17:41:18.339409+0000","last_peered":"2023-11-28T17:41:18.339409+0000","last_clean":"2023-11-28T17:41:18.339409+0000","last_became_active":"2023-11-28T17:41:17.248170+0000","last_became_peered":"2023-11-28T17:41:17.248170+0000","last_unstale":"2023-11-28T17:41:18.339409+0000","last_undegraded":"2023-11-28T17:41:18.339409+0000","last_fullsized":"2023-11-28T17:41:18.339409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:41.188370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.204990+0000","last_change":"2023-11-28T17:41:16.246192+0000","last_active":"2023-11-28T17:41:17.204990+0000","last_peered":"2023-11-28T17:41:17.204990+0000","last_clean":"2023-11-28T17:41:17.204990+0000","last_became_active":"2023-11-28T17:41:16.245379+0000","last_became_peered":"2023-11-28T17:41:16.245379+0000","last_unstale":"2023-11-28T17:41:17.204990+0000","last_undegraded":"2023-11-28T17:41:17.204990+0000","last_fullsized":"2023-11-28T17:41:17.204990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:23:08.664344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207475+0000","last_change":"2023-11-28T17:41:17.234273+0000","last_active":"2023-11-28T17:41:18.207475+0000","last_peered":"2023-11-28T17:41:18.207475+0000","last_clean":"2023-11-28T17:41:18.207475+0000","last_became_active":"2023-11-28T17:41:17.233756+0000","last_became_peered":"2023-11-28T17:41:17.233756+0000","last_unstale":"2023-11-28T17:41:18.207475+0000","last_undegraded":"2023-11-28T17:41:18.207475+0000","last_fullsized":"2023-11-28T17:41:18.207475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:18:40.539329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.215640+0000","last_change":"2023-11-28T17:41:16.257952+0000","last_active":"2023-11-28T17:41:17.215640+0000","last_peered":"2023-11-28T17:41:17.215640+0000","last_clean":"2023-11-28T17:41:17.215640+0000","last_became_active":"2023-11-28T17:41:16.257659+0000","last_became_peered":"2023-11-28T17:41:16.257659+0000","last_unstale":"2023-11-28T17:41:17.215640+0000","last_undegraded":"2023-11-28T17:41:17.215640+0000","last_fullsized":"2023-11-28T17:41:17.215640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:28:12.131592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338433+0000","last_change":"2023-11-28T17:41:17.239069+0000","last_active":"2023-11-28T17:41:18.338433+0000","last_peered":"2023-11-28T17:41:18.338433+0000","last_clean":"2023-11-28T17:41:18.338433+0000","last_became_active":"2023-11-28T17:41:17.238548+0000","last_became_peered":"2023-11-28T17:41:17.238548+0000","last_unstale":"2023-11-28T17:41:18.338433+0000","last_undegraded":"2023-11-28T17:41:18.338433+0000","last_fullsized":"2023-11-28T17:41:18.338433+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:12:56.686814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206759+0000","last_change":"2023-11-28T17:41:12.444311+0000","last_active":"2023-11-28T17:41:17.206759+0000","last_peered":"2023-11-28T17:41:17.206759+0000","last_clean":"2023-11-28T17:41:17.206759+0000","last_became_active":"2023-11-28T17:41:12.442193+0000","last_became_peered":"2023-11-28T17:41:12.442193+0000","last_unstale":"2023-11-28T17:41:17.206759+0000","last_undegraded":"2023-11-28T17:41:17.206759+0000","last_fullsized":"2023-11-28T17:41:17.206759+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:10.865936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:10.865936+0000","last_clean_scrub_stamp":"2023-11-28T17:41:10.865936+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:12:48.912392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206457+0000","last_change":"2023-11-28T17:41:16.253023+0000","last_active":"2023-11-28T17:41:18.206457+0000","last_peered":"2023-11-28T17:41:18.206457+0000","last_clean":"2023-11-28T17:41:18.206457+0000","last_became_active":"2023-11-28T17:41:16.245730+0000","last_became_peered":"2023-11-28T17:41:16.245730+0000","last_unstale":"2023-11-28T17:41:18.206457+0000","last_undegraded":"2023-11-28T17:41:18.206457+0000","last_fullsized":"2023-11-28T17:41:18.206457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:05.970917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568226+0000","last_change":"2023-11-28T17:41:17.246677+0000","last_active":"2023-11-28T17:41:18.568226+0000","last_peered":"2023-11-28T17:41:18.568226+0000","last_clean":"2023-11-28T17:41:18.568226+0000","last_became_active":"2023-11-28T17:41:17.246373+0000","last_became_peered":"2023-11-28T17:41:17.246373+0000","last_unstale":"2023-11-28T17:41:18.568226+0000","last_undegraded":"2023-11-28T17:41:18.568226+0000","last_fullsized":"2023-11-28T17:41:18.568226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:22:18.623331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206504+0000","last_change":"2023-11-28T17:41:16.272843+0000","last_active":"2023-11-28T17:41:17.206504+0000","last_peered":"2023-11-28T17:41:17.206504+0000","last_clean":"2023-11-28T17:41:17.206504+0000","last_became_active":"2023-11-28T17:41:16.272527+0000","last_became_peered":"2023-11-28T17:41:16.272527+0000","last_unstale":"2023-11-28T17:41:17.206504+0000","last_undegraded":"2023-11-28T17:41:17.206504+0000","last_fullsized":"2023-11-28T17:41:17.206504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:55:52.185953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649479+0000","last_change":"2023-11-28T17:41:17.240890+0000","last_active":"2023-11-28T17:41:18.649479+0000","last_peered":"2023-11-28T17:41:18.649479+0000","last_clean":"2023-11-28T17:41:18.649479+0000","last_became_active":"2023-11-28T17:41:17.240571+0000","last_became_peered":"2023-11-28T17:41:17.240571+0000","last_unstale":"2023-11-28T17:41:18.649479+0000","last_undegraded":"2023-11-28T17:41:18.649479+0000","last_fullsized":"2023-11-28T17:41:18.649479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:58.086322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.216327+0000","last_change":"2023-11-28T17:41:16.251925+0000","last_active":"2023-11-28T17:41:17.216327+0000","last_peered":"2023-11-28T17:41:17.216327+0000","last_clean":"2023-11-28T17:41:17.216327+0000","last_became_active":"2023-11-28T17:41:16.251482+0000","last_became_peered":"2023-11-28T17:41:16.251482+0000","last_unstale":"2023-11-28T17:41:17.216327+0000","last_undegraded":"2023-11-28T17:41:17.216327+0000","last_fullsized":"2023-11-28T17:41:17.216327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:07:09.590250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632279+0000","last_change":"2023-11-28T17:41:17.223810+0000","last_active":"2023-11-28T17:41:19.632279+0000","last_peered":"2023-11-28T17:41:19.632279+0000","last_clean":"2023-11-28T17:41:19.632279+0000","last_became_active":"2023-11-28T17:41:17.223244+0000","last_became_peered":"2023-11-28T17:41:17.223244+0000","last_unstale":"2023-11-28T17:41:19.632279+0000","last_undegraded":"2023-11-28T17:41:19.632279+0000","last_fullsized":"2023-11-28T17:41:19.632279+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:31:22.132844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207190+0000","last_change":"2023-11-28T17:41:16.252402+0000","last_active":"2023-11-28T17:41:18.207190+0000","last_peered":"2023-11-28T17:41:18.207190+0000","last_clean":"2023-11-28T17:41:18.207190+0000","last_became_active":"2023-11-28T17:41:16.252061+0000","last_became_peered":"2023-11-28T17:41:16.252061+0000","last_unstale":"2023-11-28T17:41:18.207190+0000","last_undegraded":"2023-11-28T17:41:18.207190+0000","last_fullsized":"2023-11-28T17:41:18.207190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:20:11.890036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207754+0000","last_change":"2023-11-28T17:41:17.234320+0000","last_active":"2023-11-28T17:41:18.207754+0000","last_peered":"2023-11-28T17:41:18.207754+0000","last_clean":"2023-11-28T17:41:18.207754+0000","last_became_active":"2023-11-28T17:41:17.233738+0000","last_became_peered":"2023-11-28T17:41:17.233738+0000","last_unstale":"2023-11-28T17:41:18.207754+0000","last_undegraded":"2023-11-28T17:41:18.207754+0000","last_fullsized":"2023-11-28T17:41:18.207754+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:31:10.257906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.204406+0000","last_change":"2023-11-28T17:41:16.245394+0000","last_active":"2023-11-28T17:41:17.204406+0000","last_peered":"2023-11-28T17:41:17.204406+0000","last_clean":"2023-11-28T17:41:17.204406+0000","last_became_active":"2023-11-28T17:41:16.242985+0000","last_became_peered":"2023-11-28T17:41:16.242985+0000","last_unstale":"2023-11-28T17:41:17.204406+0000","last_undegraded":"2023-11-28T17:41:17.204406+0000","last_fullsized":"2023-11-28T17:41:17.204406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:48.728960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207612+0000","last_change":"2023-11-28T17:41:17.233987+0000","last_active":"2023-11-28T17:41:18.207612+0000","last_peered":"2023-11-28T17:41:18.207612+0000","last_clean":"2023-11-28T17:41:18.207612+0000","last_became_active":"2023-11-28T17:41:17.233430+0000","last_became_peered":"2023-11-28T17:41:17.233430+0000","last_unstale":"2023-11-28T17:41:18.207612+0000","last_undegraded":"2023-11-28T17:41:18.207612+0000","last_fullsized":"2023-11-28T17:41:18.207612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:51:44.816033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.213771+0000","last_change":"2023-11-28T17:41:16.273047+0000","last_active":"2023-11-28T17:41:17.213771+0000","last_peered":"2023-11-28T17:41:17.213771+0000","last_clean":"2023-11-28T17:41:17.213771+0000","last_became_active":"2023-11-28T17:41:16.272667+0000","last_became_peered":"2023-11-28T17:41:16.272667+0000","last_unstale":"2023-11-28T17:41:17.213771+0000","last_undegraded":"2023-11-28T17:41:17.213771+0000","last_fullsized":"2023-11-28T17:41:17.213771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:19:37.472121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632439+0000","last_change":"2023-11-28T17:41:17.232465+0000","last_active":"2023-11-28T17:41:19.632439+0000","last_peered":"2023-11-28T17:41:19.632439+0000","last_clean":"2023-11-28T17:41:19.632439+0000","last_became_active":"2023-11-28T17:41:17.230496+0000","last_became_peered":"2023-11-28T17:41:17.230496+0000","last_unstale":"2023-11-28T17:41:19.632439+0000","last_undegraded":"2023-11-28T17:41:19.632439+0000","last_fullsized":"2023-11-28T17:41:19.632439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:52:08.590259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206197+0000","last_change":"2023-11-28T17:41:16.258717+0000","last_active":"2023-11-28T17:41:17.206197+0000","last_peered":"2023-11-28T17:41:17.206197+0000","last_clean":"2023-11-28T17:41:17.206197+0000","last_became_active":"2023-11-28T17:41:16.258388+0000","last_became_peered":"2023-11-28T17:41:16.258388+0000","last_unstale":"2023-11-28T17:41:17.206197+0000","last_undegraded":"2023-11-28T17:41:17.206197+0000","last_fullsized":"2023-11-28T17:41:17.206197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:17.845137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568998+0000","last_change":"2023-11-28T17:41:17.239189+0000","last_active":"2023-11-28T17:41:18.568998+0000","last_peered":"2023-11-28T17:41:18.568998+0000","last_clean":"2023-11-28T17:41:18.568998+0000","last_became_active":"2023-11-28T17:41:17.238829+0000","last_became_peered":"2023-11-28T17:41:17.238829+0000","last_unstale":"2023-11-28T17:41:18.568998+0000","last_undegraded":"2023-11-28T17:41:18.568998+0000","last_fullsized":"2023-11-28T17:41:18.568998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:10.923572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633608+0000","last_change":"2023-11-28T17:41:16.268334+0000","last_active":"2023-11-28T17:41:19.633608+0000","last_peered":"2023-11-28T17:41:19.633608+0000","last_clean":"2023-11-28T17:41:19.633608+0000","last_became_active":"2023-11-28T17:41:16.268015+0000","last_became_peered":"2023-11-28T17:41:16.268015+0000","last_unstale":"2023-11-28T17:41:19.633608+0000","last_undegraded":"2023-11-28T17:41:19.633608+0000","last_fullsized":"2023-11-28T17:41:19.633608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:57:22.031553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650340+0000","last_change":"2023-11-28T17:41:17.225161+0000","last_active":"2023-11-28T17:41:18.650340+0000","last_peered":"2023-11-28T17:41:18.650340+0000","last_clean":"2023-11-28T17:41:18.650340+0000","last_became_active":"2023-11-28T17:41:17.224420+0000","last_became_peered":"2023-11-28T17:41:17.224420+0000","last_unstale":"2023-11-28T17:41:18.650340+0000","last_undegraded":"2023-11-28T17:41:18.650340+0000","last_fullsized":"2023-11-28T17:41:18.650340+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:25:11.426980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206160+0000","last_change":"2023-11-28T17:41:16.272884+0000","last_active":"2023-11-28T17:41:17.206160+0000","last_peered":"2023-11-28T17:41:17.206160+0000","last_clean":"2023-11-28T17:41:17.206160+0000","last_became_active":"2023-11-28T17:41:16.272557+0000","last_became_peered":"2023-11-28T17:41:16.272557+0000","last_unstale":"2023-11-28T17:41:17.206160+0000","last_undegraded":"2023-11-28T17:41:17.206160+0000","last_fullsized":"2023-11-28T17:41:17.206160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:56.634559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206149+0000","last_change":"2023-11-28T17:41:17.238145+0000","last_active":"2023-11-28T17:41:18.206149+0000","last_peered":"2023-11-28T17:41:18.206149+0000","last_clean":"2023-11-28T17:41:18.206149+0000","last_became_active":"2023-11-28T17:41:17.237774+0000","last_became_peered":"2023-11-28T17:41:17.237774+0000","last_unstale":"2023-11-28T17:41:18.206149+0000","last_undegraded":"2023-11-28T17:41:18.206149+0000","last_fullsized":"2023-11-28T17:41:18.206149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:44.723180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.204215+0000","last_change":"2023-11-28T17:41:16.262318+0000","last_active":"2023-11-28T17:41:17.204215+0000","last_peered":"2023-11-28T17:41:17.204215+0000","last_clean":"2023-11-28T17:41:17.204215+0000","last_became_active":"2023-11-28T17:41:16.262014+0000","last_became_peered":"2023-11-28T17:41:16.262014+0000","last_unstale":"2023-11-28T17:41:17.204215+0000","last_undegraded":"2023-11-28T17:41:17.204215+0000","last_fullsized":"2023-11-28T17:41:17.204215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:17:15.470393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568031+0000","last_change":"2023-11-28T17:41:17.229233+0000","last_active":"2023-11-28T17:41:18.568031+0000","last_peered":"2023-11-28T17:41:18.568031+0000","last_clean":"2023-11-28T17:41:18.568031+0000","last_became_active":"2023-11-28T17:41:17.228885+0000","last_became_peered":"2023-11-28T17:41:17.228885+0000","last_unstale":"2023-11-28T17:41:18.568031+0000","last_undegraded":"2023-11-28T17:41:18.568031+0000","last_fullsized":"2023-11-28T17:41:18.568031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:10:24.247307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.203545+0000","last_change":"2023-11-28T17:41:16.246533+0000","last_active":"2023-11-28T17:41:17.203545+0000","last_peered":"2023-11-28T17:41:17.203545+0000","last_clean":"2023-11-28T17:41:17.203545+0000","last_became_active":"2023-11-28T17:41:16.244067+0000","last_became_peered":"2023-11-28T17:41:16.244067+0000","last_unstale":"2023-11-28T17:41:17.203545+0000","last_undegraded":"2023-11-28T17:41:17.203545+0000","last_fullsized":"2023-11-28T17:41:17.203545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:02:20.862297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568303+0000","last_change":"2023-11-28T17:41:17.236159+0000","last_active":"2023-11-28T17:41:18.568303+0000","last_peered":"2023-11-28T17:41:18.568303+0000","last_clean":"2023-11-28T17:41:18.568303+0000","last_became_active":"2023-11-28T17:41:17.235700+0000","last_became_peered":"2023-11-28T17:41:17.235700+0000","last_unstale":"2023-11-28T17:41:18.568303+0000","last_undegraded":"2023-11-28T17:41:18.568303+0000","last_fullsized":"2023-11-28T17:41:18.568303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:21.900805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.209240+0000","last_change":"2023-11-28T17:41:16.268969+0000","last_active":"2023-11-28T17:41:17.209240+0000","last_peered":"2023-11-28T17:41:17.209240+0000","last_clean":"2023-11-28T17:41:17.209240+0000","last_became_active":"2023-11-28T17:41:16.268717+0000","last_became_peered":"2023-11-28T17:41:16.268717+0000","last_unstale":"2023-11-28T17:41:17.209240+0000","last_undegraded":"2023-11-28T17:41:17.209240+0000","last_fullsized":"2023-11-28T17:41:17.209240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:03:29.120624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633192+0000","last_change":"2023-11-28T17:41:17.244376+0000","last_active":"2023-11-28T17:41:19.633192+0000","last_peered":"2023-11-28T17:41:19.633192+0000","last_clean":"2023-11-28T17:41:19.633192+0000","last_became_active":"2023-11-28T17:41:17.244034+0000","last_became_peered":"2023-11-28T17:41:17.244034+0000","last_unstale":"2023-11-28T17:41:19.633192+0000","last_undegraded":"2023-11-28T17:41:19.633192+0000","last_fullsized":"2023-11-28T17:41:19.633192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:15:56.198732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208198+0000","last_change":"2023-11-28T17:41:17.233749+0000","last_active":"2023-11-28T17:41:18.208198+0000","last_peered":"2023-11-28T17:41:18.208198+0000","last_clean":"2023-11-28T17:41:18.208198+0000","last_became_active":"2023-11-28T17:41:17.232992+0000","last_became_peered":"2023-11-28T17:41:17.232992+0000","last_unstale":"2023-11-28T17:41:18.208198+0000","last_undegraded":"2023-11-28T17:41:18.208198+0000","last_fullsized":"2023-11-28T17:41:18.208198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:00:43.698244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.207649+0000","last_change":"2023-11-28T17:41:16.269465+0000","last_active":"2023-11-28T17:41:17.207649+0000","last_peered":"2023-11-28T17:41:17.207649+0000","last_clean":"2023-11-28T17:41:17.207649+0000","last_became_active":"2023-11-28T17:41:16.269147+0000","last_became_peered":"2023-11-28T17:41:16.269147+0000","last_unstale":"2023-11-28T17:41:17.207649+0000","last_undegraded":"2023-11-28T17:41:17.207649+0000","last_fullsized":"2023-11-28T17:41:17.207649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:57:58.121232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207346+0000","last_change":"2023-11-28T17:41:17.243498+0000","last_active":"2023-11-28T17:41:18.207346+0000","last_peered":"2023-11-28T17:41:18.207346+0000","last_clean":"2023-11-28T17:41:18.207346+0000","last_became_active":"2023-11-28T17:41:17.243169+0000","last_became_peered":"2023-11-28T17:41:17.243169+0000","last_unstale":"2023-11-28T17:41:18.207346+0000","last_undegraded":"2023-11-28T17:41:18.207346+0000","last_fullsized":"2023-11-28T17:41:18.207346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:11:31.509921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.214303+0000","last_change":"2023-11-28T17:41:16.270231+0000","last_active":"2023-11-28T17:41:17.214303+0000","last_peered":"2023-11-28T17:41:17.214303+0000","last_clean":"2023-11-28T17:41:17.214303+0000","last_became_active":"2023-11-28T17:41:16.269938+0000","last_became_peered":"2023-11-28T17:41:16.269938+0000","last_unstale":"2023-11-28T17:41:17.214303+0000","last_undegraded":"2023-11-28T17:41:17.214303+0000","last_fullsized":"2023-11-28T17:41:17.214303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:45.654658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650357+0000","last_change":"2023-11-28T17:41:17.245856+0000","last_active":"2023-11-28T17:41:18.650357+0000","last_peered":"2023-11-28T17:41:18.650357+0000","last_clean":"2023-11-28T17:41:18.650357+0000","last_became_active":"2023-11-28T17:41:17.245624+0000","last_became_peered":"2023-11-28T17:41:17.245624+0000","last_unstale":"2023-11-28T17:41:18.650357+0000","last_undegraded":"2023-11-28T17:41:18.650357+0000","last_fullsized":"2023-11-28T17:41:18.650357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:03:10.206506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.228705+0000","last_change":"2023-11-28T17:41:16.260989+0000","last_active":"2023-11-28T17:41:17.228705+0000","last_peered":"2023-11-28T17:41:17.228705+0000","last_clean":"2023-11-28T17:41:17.228705+0000","last_became_active":"2023-11-28T17:41:16.260686+0000","last_became_peered":"2023-11-28T17:41:16.260686+0000","last_unstale":"2023-11-28T17:41:17.228705+0000","last_undegraded":"2023-11-28T17:41:17.228705+0000","last_fullsized":"2023-11-28T17:41:17.228705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:58:44.193791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568672+0000","last_change":"2023-11-28T17:41:17.240803+0000","last_active":"2023-11-28T17:41:18.568672+0000","last_peered":"2023-11-28T17:41:18.568672+0000","last_clean":"2023-11-28T17:41:18.568672+0000","last_became_active":"2023-11-28T17:41:17.240442+0000","last_became_peered":"2023-11-28T17:41:17.240442+0000","last_unstale":"2023-11-28T17:41:18.568672+0000","last_undegraded":"2023-11-28T17:41:18.568672+0000","last_fullsized":"2023-11-28T17:41:18.568672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:28:21.223788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.217458+0000","last_change":"2023-11-28T17:41:16.266520+0000","last_active":"2023-11-28T17:41:17.217458+0000","last_peered":"2023-11-28T17:41:17.217458+0000","last_clean":"2023-11-28T17:41:17.217458+0000","last_became_active":"2023-11-28T17:41:16.266245+0000","last_became_peered":"2023-11-28T17:41:16.266245+0000","last_unstale":"2023-11-28T17:41:17.217458+0000","last_undegraded":"2023-11-28T17:41:17.217458+0000","last_fullsized":"2023-11-28T17:41:17.217458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:18:28.732421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65062,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509445,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":286720,"data_stored":666900,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.51}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.72}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1070000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.839}]}]},{"osd":6,"up_from":15,"seq":64424509443,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":70232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1619999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509444,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":76532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.24}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4470000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.377}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.679}]}]},{"osd":4,"up_from":15,"seq":64424509444,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":76532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.377}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.482}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.756}]}]},{"osd":3,"up_from":15,"seq":64424509443,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":76532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.397}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5409999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.605}]}]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":76532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.603}]}]},{"osd":0,"up_from":15,"seq":64424509445,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":286720,"data_stored":666900,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.514}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7869999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5409999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":63457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6040000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0329999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7869999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7450000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T17:41:24.148 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T17:41:24.149 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-28T17:41:24.149 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-28T17:41:24.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-28T17:41:24.149 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-28T17:41:24.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.397+0000 7fe6d2b01640 1 Processor -- start 2023-11-28T17:41:24.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.397+0000 7fe6d2b01640 1 -- start start 2023-11-28T17:41:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.399+0000 7fe6d2b01640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe6cc153fe0 0x7fe6cc15dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.399+0000 7fe6d2b01640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6cc0c4b20 0x7fe6cc0ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.399+0000 7fe6d2b01640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6cc15e200 con 0x7fe6cc0ca8e0 2023-11-28T17:41:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.399+0000 7fe6d2b01640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6cc0c4800 con 0x7fe6cc0c4b20 2023-11-28T17:41:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.399+0000 7fe6d2b01640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6cc0c51d0 con 0x7fe6cc153fe0 2023-11-28T17:41:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.399+0000 7fe6cbfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6cc0c4b20 0x7fe6cc0ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.399+0000 7fe6d109e640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe6cc0ca8e0 0x7fe6cc0c5660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37474/0 (socket says 172.21.15.98:37474) 2023-11-28T17:41:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.399+0000 7fe6d109e640 1 -- 172.21.15.98:0/2085783259 learned_addr learned my addr 172.21.15.98:0/2085783259 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.400+0000 7fe6d089d640 1 --2- 172.21.15.98:0/2085783259 >> v2:172.21.15.113:3300/0 conn(0x7fe6cc153fe0 0x7fe6cc15dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.400+0000 7fe6cb7fe640 1 -- 172.21.15.98:0/2085783259 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862124313 0 0) 0x7fe6c4001490 con 0x7fe6cc0ca8e0 2023-11-28T17:41:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.400+0000 7fe6cbfff640 1 -- 172.21.15.98:0/2085783259 >> v2:172.21.15.113:3300/0 conn(0x7fe6cc153fe0 msgr2=0x7fe6cc15dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.400+0000 7fe6cbfff640 1 --2- 172.21.15.98:0/2085783259 >> v2:172.21.15.113:3300/0 conn(0x7fe6cc153fe0 0x7fe6cc15dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.400+0000 7fe6cbfff640 1 -- 172.21.15.98:0/2085783259 >> v1:172.21.15.98:6789/0 conn(0x7fe6cc0ca8e0 legacy=0x7fe6cc0c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.400+0000 7fe6cbfff640 1 -- 172.21.15.98:0/2085783259 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc1543c0 con 0x7fe6cc0c4b20 2023-11-28T17:41:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.401+0000 7fe6cbfff640 1 --2- 172.21.15.98:0/2085783259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0c4b20 0x7fe6cc0ca3a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80097a0 tx=0x7fe6b802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5154af5cd838a1e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.402+0000 7fe6cb7fe640 1 -- 172.21.15.98:0/2085783259 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8030610 con 0x7fe6cc0c4b20 2023-11-28T17:41:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.402+0000 7fe6cb7fe640 1 -- 172.21.15.98:0/2085783259 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b8030770 con 0x7fe6cc0c4b20 2023-11-28T17:41:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.402+0000 7fe6cb7fe640 1 -- 172.21.15.98:0/2085783259 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8030b00 con 0x7fe6cc0c4b20 2023-11-28T17:41:24.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.402+0000 7fe6d2b01640 1 -- 172.21.15.98:0/2085783259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0c4b20 msgr2=0x7fe6cc0ca3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:24.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.402+0000 7fe6d2b01640 1 --2- 172.21.15.98:0/2085783259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0c4b20 0x7fe6cc0ca3a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80097a0 tx=0x7fe6b802b960 comp rx=0 tx=0).stop 2023-11-28T17:41:24.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.402+0000 7fe6d2b01640 1 -- 172.21.15.98:0/2085783259 shutdown_connections 2023-11-28T17:41:24.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.402+0000 7fe6d2b01640 1 --2- 172.21.15.98:0/2085783259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0c4b20 0x7fe6cc0ca3a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.402+0000 7fe6d2b01640 1 --2- 172.21.15.98:0/2085783259 >> v2:172.21.15.113:3300/0 conn(0x7fe6cc153fe0 0x7fe6cc15dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.402+0000 7fe6d2b01640 1 -- 172.21.15.98:0/2085783259 >> 172.21.15.98:0/2085783259 conn(0x7fe6cc066230 msgr2=0x7fe6cc066640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:24.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.403+0000 7fe6d2b01640 1 -- 172.21.15.98:0/2085783259 shutdown_connections 2023-11-28T17:41:24.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.403+0000 7fe6d2b01640 1 -- 172.21.15.98:0/2085783259 wait complete. 2023-11-28T17:41:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.404+0000 7fe6d2b01640 1 Processor -- start 2023-11-28T17:41:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.404+0000 7fe6d2b01640 1 -- start start 2023-11-28T17:41:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.405+0000 7fe6d2b01640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6cc0c4b20 0x7fe6cc158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.405+0000 7fe6d2b01640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0ca8e0 0x7fe6cc0b5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.405+0000 7fe6d2b01640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6cc153fe0 0x7fe6cc0b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.405+0000 7fe6d2b01640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc1ef990 con 0x7fe6cc0ca8e0 2023-11-28T17:41:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.405+0000 7fe6d2b01640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc159010 con 0x7fe6cc153fe0 2023-11-28T17:41:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.405+0000 7fe6d2b01640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc1f0960 con 0x7fe6cc0c4b20 2023-11-28T17:41:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.405+0000 7fe6cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0ca8e0 0x7fe6cc0b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.406+0000 7fe6cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0ca8e0 0x7fe6cc0b5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49294/0 (socket says 172.21.15.98:49294) 2023-11-28T17:41:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.406+0000 7fe6cbfff640 1 -- 172.21.15.98:0/4106137998 learned_addr learned my addr 172.21.15.98:0/4106137998 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.406+0000 7fe6d109e640 1 --2- 172.21.15.98:0/4106137998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6cc153fe0 0x7fe6cc0b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:24.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.406+0000 7fe6d089d640 1 --2- 172.21.15.98:0/4106137998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6cc0c4b20 0x7fe6cc158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:24.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.406+0000 7fe6cbfff640 1 -- 172.21.15.98:0/4106137998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6cc0c4b20 msgr2=0x7fe6cc158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:24.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.406+0000 7fe6cbfff640 1 --2- 172.21.15.98:0/4106137998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6cc0c4b20 0x7fe6cc158ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.406+0000 7fe6cbfff640 1 -- 172.21.15.98:0/4106137998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6cc153fe0 msgr2=0x7fe6cc0b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:24.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.406+0000 7fe6cbfff640 1 --2- 172.21.15.98:0/4106137998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6cc153fe0 0x7fe6cc0b3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.406+0000 7fe6cbfff640 1 -- 172.21.15.98:0/4106137998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b8009400 con 0x7fe6cc0ca8e0 2023-11-28T17:41:24.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.406+0000 7fe6cbfff640 1 --2- 172.21.15.98:0/4106137998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0ca8e0 0x7fe6cc0b5170 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009890 tx=0x7fe6b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:24.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.407+0000 7fe6c97fa640 1 -- 172.21.15.98:0/4106137998 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b803e040 con 0x7fe6cc0ca8e0 2023-11-28T17:41:24.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.407+0000 7fe6c97fa640 1 -- 172.21.15.98:0/4106137998 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b8039ce0 con 0x7fe6cc0ca8e0 2023-11-28T17:41:24.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.407+0000 7fe6c97fa640 1 -- 172.21.15.98:0/4106137998 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8043680 con 0x7fe6cc0ca8e0 2023-11-28T17:41:24.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.407+0000 7fe6d2b01640 1 -- 172.21.15.98:0/4106137998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6cc1f04d0 con 0x7fe6cc0ca8e0 2023-11-28T17:41:24.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.407+0000 7fe6d2b01640 1 -- 172.21.15.98:0/4106137998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6cc159230 con 0x7fe6cc0ca8e0 2023-11-28T17:41:24.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.408+0000 7fe6d2b01640 1 -- 172.21.15.98:0/4106137998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6cc1f0660 con 0x7fe6cc0ca8e0 2023-11-28T17:41:24.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.409+0000 7fe6c97fa640 1 -- 172.21.15.98:0/4106137998 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe6b80423e0 con 0x7fe6cc0ca8e0 2023-11-28T17:41:24.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.410+0000 7fe6c97fa640 1 --2- 172.21.15.98:0/4106137998 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe6b006b2a0 0x7fe6b006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:24.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.410+0000 7fe6c97fa640 1 -- 172.21.15.98:0/4106137998 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe6b80abdf0 con 0x7fe6cc0ca8e0 2023-11-28T17:41:24.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.410+0000 7fe6d089d640 1 --2- 172.21.15.98:0/4106137998 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe6b006b2a0 0x7fe6b006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:24.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.411+0000 7fe6d089d640 1 --2- 172.21.15.98:0/4106137998 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe6b006b2a0 0x7fe6b006d760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0006fd0 tx=0x7fe6c0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:24.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.414+0000 7fe6c97fa640 1 -- 172.21.15.98:0/4106137998 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6b8076b30 con 0x7fe6cc0ca8e0 2023-11-28T17:41:24.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.655+0000 7fe6d2b01640 1 -- 172.21.15.98:0/4106137998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe6cc074f60 con 0x7fe6cc0ca8e0 2023-11-28T17:41:24.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.663+0000 7fe6c97fa640 1 -- 172.21.15.98:0/4106137998 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171569 (secure 0 0 0) 0x7fe6b8006e10 con 0x7fe6cc0ca8e0 2023-11-28T17:41:24.667 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:24.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.668+0000 7fe6d2b01640 1 -- 172.21.15.98:0/4106137998 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe6b006b2a0 msgr2=0x7fe6b006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:24.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.668+0000 7fe6d2b01640 1 --2- 172.21.15.98:0/4106137998 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe6b006b2a0 0x7fe6b006d760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0006fd0 tx=0x7fe6c0008040 comp rx=0 tx=0).stop 2023-11-28T17:41:24.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.668+0000 7fe6d2b01640 1 -- 172.21.15.98:0/4106137998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0ca8e0 msgr2=0x7fe6cc0b5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:24.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.668+0000 7fe6d2b01640 1 --2- 172.21.15.98:0/4106137998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0ca8e0 0x7fe6cc0b5170 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009890 tx=0x7fe6b8004730 comp rx=0 tx=0).stop 2023-11-28T17:41:24.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.668+0000 7fe6d2b01640 1 -- 172.21.15.98:0/4106137998 shutdown_connections 2023-11-28T17:41:24.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.668+0000 7fe6d2b01640 1 --2- 172.21.15.98:0/4106137998 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe6b006b2a0 0x7fe6b006d760 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.668+0000 7fe6d2b01640 1 --2- 172.21.15.98:0/4106137998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6cc153fe0 0x7fe6cc0b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.668+0000 7fe6d2b01640 1 --2- 172.21.15.98:0/4106137998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6cc0ca8e0 0x7fe6cc0b5170 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.668+0000 7fe6d2b01640 1 --2- 172.21.15.98:0/4106137998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6cc0c4b20 0x7fe6cc158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.668+0000 7fe6d2b01640 1 -- 172.21.15.98:0/4106137998 >> 172.21.15.98:0/4106137998 conn(0x7fe6cc066230 msgr2=0x7fe6cc0cbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:24.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.668+0000 7fe6d2b01640 1 -- 172.21.15.98:0/4106137998 shutdown_connections 2023-11-28T17:41:24.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.668+0000 7fe6d2b01640 1 -- 172.21.15.98:0/4106137998 wait complete. 2023-11-28T17:41:24.688 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"active_gid":4098,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2274392793},{"type":"v1","addr":"172.21.15.98:6833","nonce":2274392793}]},"active_addr":"172.21.15.98:6833/2274392793","active_change":"2023-11-28T17:41:02.213990+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4106,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1749385956}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":2758054914}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1082065653}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":596090353}]}]} 2023-11-28T17:41:24.693 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-28T17:41:24.693 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-28T17:41:24.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:41:24.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.887+0000 7f737db01640 1 Processor -- start 2023-11-28T17:41:24.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.887+0000 7f737db01640 1 -- start start 2023-11-28T17:41:24.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.889+0000 7f737db01640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7378153f60 0x7f73780cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:24.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.889+0000 7f737db01640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f737815d240 0x7f737815d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:24.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.889+0000 7f737db01640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f737806b010 con 0x7f737806bd40 2023-11-28T17:41:24.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.889+0000 7f737db01640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f737815dd60 con 0x7f737815d240 2023-11-28T17:41:24.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.889+0000 7f737db01640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f737815e040 con 0x7f7378153f60 2023-11-28T17:41:24.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.889+0000 7f736ffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f737806bd40 0x7f73780c4460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37478/0 (socket says 172.21.15.98:37478) 2023-11-28T17:41:24.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.889+0000 7f736ffff640 1 -- 172.21.15.98:0/1073654861 learned_addr learned my addr 172.21.15.98:0/1073654861 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:24.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.889+0000 7f7377fff640 1 --2- 172.21.15.98:0/1073654861 >> v2:172.21.15.98:3300/0 conn(0x7f737815d240 0x7f737815d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:24.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.890+0000 7f73777fe640 1 --2- 172.21.15.98:0/1073654861 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7378153f60 0x7f73780cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:24.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.890+0000 7f7376ffd640 1 -- 172.21.15.98:0/1073654861 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619080916 0 0) 0x7f737806b010 con 0x7f737806bd40 2023-11-28T17:41:24.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.890+0000 7f7376ffd640 1 -- 172.21.15.98:0/1073654861 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7364003520 con 0x7f737806bd40 2023-11-28T17:41:24.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.890+0000 7f7377fff640 1 -- 172.21.15.98:0/1073654861 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7378153f60 msgr2=0x7f73780cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:24.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.890+0000 7f7377fff640 1 --2- 172.21.15.98:0/1073654861 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7378153f60 0x7f73780cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.890+0000 7f7377fff640 1 -- 172.21.15.98:0/1073654861 >> v1:172.21.15.98:6789/0 conn(0x7f737806bd40 legacy=0x7f73780c4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:24.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.890+0000 7f7377fff640 1 -- 172.21.15.98:0/1073654861 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737806b260 con 0x7f737815d240 2023-11-28T17:41:24.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.890+0000 7f73777fe640 1 --2- 172.21.15.98:0/1073654861 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7378153f60 0x7f73780cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:41:24.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.891+0000 7f7377fff640 1 --2- 172.21.15.98:0/1073654861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737815d240 0x7f737815d690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7370009d90 tx=0x7f737002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=825bf4adbccbd3e9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:24.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.891+0000 7f7376ffd640 1 -- 172.21.15.98:0/1073654861 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737002cce0 con 0x7f737815d240 2023-11-28T17:41:24.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.891+0000 7f7376ffd640 1 -- 172.21.15.98:0/1073654861 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737002ce40 con 0x7f737815d240 2023-11-28T17:41:24.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.891+0000 7f7376ffd640 1 -- 172.21.15.98:0/1073654861 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370034660 con 0x7f737815d240 2023-11-28T17:41:24.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.892+0000 7f737db01640 1 -- 172.21.15.98:0/1073654861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737815d240 msgr2=0x7f737815d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:24.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.892+0000 7f737db01640 1 --2- 172.21.15.98:0/1073654861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737815d240 0x7f737815d690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7370009d90 tx=0x7f737002bf30 comp rx=0 tx=0).stop 2023-11-28T17:41:24.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.892+0000 7f737db01640 1 -- 172.21.15.98:0/1073654861 shutdown_connections 2023-11-28T17:41:24.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.892+0000 7f737db01640 1 --2- 172.21.15.98:0/1073654861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737815d240 0x7f737815d690 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.892+0000 7f737db01640 1 --2- 172.21.15.98:0/1073654861 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7378153f60 0x7f73780cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.892+0000 7f737db01640 1 -- 172.21.15.98:0/1073654861 >> 172.21.15.98:0/1073654861 conn(0x7f7378066410 msgr2=0x7f7378066820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:24.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.892+0000 7f737db01640 1 -- 172.21.15.98:0/1073654861 shutdown_connections 2023-11-28T17:41:24.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.892+0000 7f737db01640 1 -- 172.21.15.98:0/1073654861 wait complete. 2023-11-28T17:41:24.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.894+0000 7f737db01640 1 Processor -- start 2023-11-28T17:41:24.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.894+0000 7f737db01640 1 -- start start 2023-11-28T17:41:24.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.895+0000 7f737db01640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737806bd40 0x7f73781f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:24.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.895+0000 7f737db01640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7378153f60 0x7f737814c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:24.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.895+0000 7f737db01640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f737815d240 0x7f737815c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:24.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.895+0000 7f737db01640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737814b230 con 0x7f737806bd40 2023-11-28T17:41:24.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.895+0000 7f737db01640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737814b720 con 0x7f737815d240 2023-11-28T17:41:24.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.895+0000 7f737db01640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f737814dd90 con 0x7f7378153f60 2023-11-28T17:41:24.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.895+0000 7f73777fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737806bd40 0x7f73781f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:24.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.895+0000 7f73777fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737806bd40 0x7f73781f2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49302/0 (socket says 172.21.15.98:49302) 2023-11-28T17:41:24.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.895+0000 7f73777fe640 1 -- 172.21.15.98:0/798264161 learned_addr learned my addr 172.21.15.98:0/798264161 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:24.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.896+0000 7f7377fff640 1 --2- 172.21.15.98:0/798264161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f737815d240 0x7f737815c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:24.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.896+0000 7f73777fe640 1 -- 172.21.15.98:0/798264161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7378153f60 msgr2=0x7f737814c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:24.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.896+0000 7f736ffff640 1 --2- 172.21.15.98:0/798264161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7378153f60 0x7f737814c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:24.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.896+0000 7f73777fe640 1 --2- 172.21.15.98:0/798264161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7378153f60 0x7f737814c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.896+0000 7f73777fe640 1 -- 172.21.15.98:0/798264161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f737815d240 msgr2=0x7f737815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:24.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.896+0000 7f73777fe640 1 --2- 172.21.15.98:0/798264161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f737815d240 0x7f737815c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:24.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.896+0000 7f73777fe640 1 -- 172.21.15.98:0/798264161 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7370009a70 con 0x7f737806bd40 2023-11-28T17:41:24.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.896+0000 7f73777fe640 1 --2- 172.21.15.98:0/798264161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737806bd40 0x7f73781f2740 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f736800e7e0 tx=0x7f736800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:24.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.896+0000 7f7374ff9640 1 -- 172.21.15.98:0/798264161 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7368004030 con 0x7f737806bd40 2023-11-28T17:41:24.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.896+0000 7f7374ff9640 1 -- 172.21.15.98:0/798264161 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7368004190 con 0x7f737806bd40 2023-11-28T17:41:24.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.896+0000 7f7374ff9640 1 -- 172.21.15.98:0/798264161 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7368004520 con 0x7f737806bd40 2023-11-28T17:41:24.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.896+0000 7f737db01640 1 -- 172.21.15.98:0/798264161 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f737814b440 con 0x7f737806bd40 2023-11-28T17:41:24.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.897+0000 7f737db01640 1 -- 172.21.15.98:0/798264161 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f737814bb40 con 0x7f737806bd40 2023-11-28T17:41:24.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.898+0000 7f737db01640 1 -- 172.21.15.98:0/798264161 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73780cc330 con 0x7f737806bd40 2023-11-28T17:41:24.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.899+0000 7f7374ff9640 1 -- 172.21.15.98:0/798264161 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f736800c6d0 con 0x7f737806bd40 2023-11-28T17:41:24.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.899+0000 7f7374ff9640 1 --2- 172.21.15.98:0/798264161 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f735806b2a0 0x7f735806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:24.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.900+0000 7f7374ff9640 1 -- 172.21.15.98:0/798264161 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73680899c0 con 0x7f737806bd40 2023-11-28T17:41:24.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.900+0000 7f736ffff640 1 --2- 172.21.15.98:0/798264161 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f735806b2a0 0x7f735806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:24.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.900+0000 7f736ffff640 1 --2- 172.21.15.98:0/798264161 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f735806b2a0 0x7f735806d760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f73780b1fc0 tx=0x7f7360009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:24.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:24.904+0000 7f7374ff9640 1 -- 172.21.15.98:0/798264161 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73680547a0 con 0x7f737806bd40 2023-11-28T17:41:25.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.095+0000 7f737db01640 1 -- 172.21.15.98:0/798264161 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7378074f60 con 0x7f737806bd40 2023-11-28T17:41:25.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.097+0000 7f7374ff9640 1 -- 172.21.15.98:0/798264161 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16517 (secure 0 0 0) 0x7f7368020020 con 0x7f737806bd40 2023-11-28T17:41:25.099 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:25.099 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:41:18.195933+0000","last_up_change":"2023-11-28T17:41:09.556993+0000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:41:10.303280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:41:14.890871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:41:15.704103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6810","nonce":2188704385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6813","nonce":2188704385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6819","nonce":2188704385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6816","nonce":2188704385}]},"public_addr":"172.21.15.98:6810/2188704385","cluster_addr":"172.21.15.98:6813/2188704385","heartbeat_back_addr":"172.21.15.98:6819/2188704385","heartbeat_front_addr":"172.21.15.98:6816/2188704385","state":["exists","up"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6801","nonce":3918226860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6803","nonce":3918226860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6807","nonce":3918226860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6805","nonce":3918226860}]},"public_addr":"172.21.15.98:6801/3918226860","cluster_addr":"172.21.15.98:6803/3918226860","heartbeat_back_addr":"172.21.15.98:6807/3918226860","heartbeat_front_addr":"172.21.15.98:6805/3918226860","state":["exists","up"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6812","nonce":2691532400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6815","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6818","nonce":2691532400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6823","nonce":2691532400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6821","nonce":2691532400}]},"public_addr":"172.21.15.98:6812/2691532400","cluster_addr":"172.21.15.98:6818/2691532400","heartbeat_back_addr":"172.21.15.98:6823/2691532400","heartbeat_front_addr":"172.21.15.98:6821/2691532400","state":["exists","up"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6825","nonce":1905487008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6827","nonce":1905487008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6831","nonce":1905487008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6829","nonce":1905487008}]},"public_addr":"172.21.15.98:6825/1905487008","cluster_addr":"172.21.15.98:6827/1905487008","heartbeat_back_addr":"172.21.15.98:6831/1905487008","heartbeat_front_addr":"172.21.15.98:6829/1905487008","state":["exists","up"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6801","nonce":2974677215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6803","nonce":2974677215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6807","nonce":2974677215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6805","nonce":2974677215}]},"public_addr":"172.21.15.113:6801/2974677215","cluster_addr":"172.21.15.113:6803/2974677215","heartbeat_back_addr":"172.21.15.113:6807/2974677215","heartbeat_front_addr":"172.21.15.113:6805/2974677215","state":["exists","up"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6817","nonce":2685000355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6821","nonce":2685000355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6829","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6831","nonce":2685000355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6823","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6826","nonce":2685000355}]},"public_addr":"172.21.15.113:6817/2685000355","cluster_addr":"172.21.15.113:6821/2685000355","heartbeat_back_addr":"172.21.15.113:6831/2685000355","heartbeat_front_addr":"172.21.15.113:6826/2685000355","state":["exists","up"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6809","nonce":2764300831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6811","nonce":2764300831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6815","nonce":2764300831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6813","nonce":2764300831}]},"public_addr":"172.21.15.113:6809/2764300831","cluster_addr":"172.21.15.113:6811/2764300831","heartbeat_back_addr":"172.21.15.113:6815/2764300831","heartbeat_front_addr":"172.21.15.113:6813/2764300831","state":["exists","up"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6819","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6820","nonce":1656248400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6824","nonce":1656248400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6830","nonce":1656248400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6825","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6827","nonce":1656248400}]},"public_addr":"172.21.15.113:6820/1656248400","cluster_addr":"172.21.15.113:6824/1656248400","heartbeat_back_addr":"172.21.15.113:6830/1656248400","heartbeat_front_addr":"172.21.15.113:6827/1656248400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.379549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.312196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.117494+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.388128+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.360608+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.411101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.355995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.390362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:41:25.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.100+0000 7f737db01640 1 -- 172.21.15.98:0/798264161 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f735806b2a0 msgr2=0x7f735806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.100+0000 7f737db01640 1 --2- 172.21.15.98:0/798264161 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f735806b2a0 0x7f735806d760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f73780b1fc0 tx=0x7f7360009040 comp rx=0 tx=0).stop 2023-11-28T17:41:25.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.100+0000 7f737db01640 1 -- 172.21.15.98:0/798264161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737806bd40 msgr2=0x7f73781f2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.100+0000 7f737db01640 1 --2- 172.21.15.98:0/798264161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737806bd40 0x7f73781f2740 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f736800e7e0 tx=0x7f736800eca0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.100+0000 7f737db01640 1 -- 172.21.15.98:0/798264161 shutdown_connections 2023-11-28T17:41:25.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.100+0000 7f737db01640 1 --2- 172.21.15.98:0/798264161 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f735806b2a0 0x7f735806d760 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.100+0000 7f737db01640 1 --2- 172.21.15.98:0/798264161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f737815d240 0x7f737815c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.100+0000 7f737db01640 1 --2- 172.21.15.98:0/798264161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7378153f60 0x7f737814c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.100+0000 7f737db01640 1 --2- 172.21.15.98:0/798264161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f737806bd40 0x7f73781f2740 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.100+0000 7f737db01640 1 -- 172.21.15.98:0/798264161 >> 172.21.15.98:0/798264161 conn(0x7f7378066410 msgr2=0x7f73780667f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.100+0000 7f737db01640 1 -- 172.21.15.98:0/798264161 shutdown_connections 2023-11-28T17:41:25.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.100+0000 7f737db01640 1 -- 172.21.15.98:0/798264161 wait complete. 2023-11-28T17:41:25.119 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-28T17:41:25.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:41:25.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.332+0000 7f954ec9b640 1 Processor -- start 2023-11-28T17:41:25.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.332+0000 7f954ec9b640 1 -- start start 2023-11-28T17:41:25.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.334+0000 7f954ec9b640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9548151d80 0x7f9548157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.334+0000 7f954ec9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f954815bc10 0x7f95480c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.334+0000 7f954ec9b640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f95480c6140 0x7f95480c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.334+0000 7f954ec9b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f954815b150 con 0x7f954815bc10 2023-11-28T17:41:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.334+0000 7f954ec9b640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f954806e1f0 con 0x7f95480c6140 2023-11-28T17:41:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.334+0000 7f954ec9b640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954806e4d0 con 0x7f9548151d80 2023-11-28T17:41:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.334+0000 7f9547fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f954815bc10 0x7f95480c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.334+0000 7f9547fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f954815bc10 0x7f95480c5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49318/0 (socket says 172.21.15.98:49318) 2023-11-28T17:41:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.334+0000 7f9547fff640 1 -- 172.21.15.98:0/3016821256 learned_addr learned my addr 172.21.15.98:0/3016821256 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.334+0000 7f954d238640 1 --2- 172.21.15.98:0/3016821256 >> v2:172.21.15.113:3300/0 conn(0x7f95480c6140 0x7f95480c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.334+0000 7f954ca37640 1 --2- 172.21.15.98:0/3016821256 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9548151d80 0x7f9548157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.335+0000 7f9547fff640 1 -- 172.21.15.98:0/3016821256 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9548151d80 msgr2=0x7f9548157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.335+0000 7f9547fff640 1 --2- 172.21.15.98:0/3016821256 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9548151d80 0x7f9548157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.335+0000 7f9547fff640 1 -- 172.21.15.98:0/3016821256 >> v2:172.21.15.113:3300/0 conn(0x7f95480c6140 msgr2=0x7f95480c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.335+0000 7f9547fff640 1 --2- 172.21.15.98:0/3016821256 >> v2:172.21.15.113:3300/0 conn(0x7f95480c6140 0x7f95480c4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.335+0000 7f9547fff640 1 -- 172.21.15.98:0/3016821256 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954806e7b0 con 0x7f954815bc10 2023-11-28T17:41:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.336+0000 7f9547fff640 1 --2- 172.21.15.98:0/3016821256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f954815bc10 0x7f95480c5c00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f95480c78a0 tx=0x7f953002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c9de7439305397 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.336+0000 7f95477fe640 1 -- 172.21.15.98:0/3016821256 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9530030860 con 0x7f954815bc10 2023-11-28T17:41:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.336+0000 7f95477fe640 1 -- 172.21.15.98:0/3016821256 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95300309c0 con 0x7f954815bc10 2023-11-28T17:41:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.336+0000 7f95477fe640 1 -- 172.21.15.98:0/3016821256 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9530030d50 con 0x7f954815bc10 2023-11-28T17:41:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.337+0000 7f954ec9b640 1 -- 172.21.15.98:0/3016821256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f954815bc10 msgr2=0x7f95480c5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.337+0000 7f954ec9b640 1 --2- 172.21.15.98:0/3016821256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f954815bc10 0x7f95480c5c00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f95480c78a0 tx=0x7f953002b9a0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.337+0000 7f954ec9b640 1 -- 172.21.15.98:0/3016821256 shutdown_connections 2023-11-28T17:41:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.337+0000 7f954ec9b640 1 --2- 172.21.15.98:0/3016821256 >> v2:172.21.15.113:3300/0 conn(0x7f95480c6140 0x7f95480c4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.337+0000 7f954ec9b640 1 --2- 172.21.15.98:0/3016821256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f954815bc10 0x7f95480c5c00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.337+0000 7f954ec9b640 1 --2- 172.21.15.98:0/3016821256 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9548151d80 0x7f9548157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.337+0000 7f954ec9b640 1 -- 172.21.15.98:0/3016821256 >> 172.21.15.98:0/3016821256 conn(0x7f954806a4d0 msgr2=0x7f954806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.337+0000 7f954ec9b640 1 -- 172.21.15.98:0/3016821256 shutdown_connections 2023-11-28T17:41:25.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.337+0000 7f954ec9b640 1 -- 172.21.15.98:0/3016821256 wait complete. 2023-11-28T17:41:25.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.339+0000 7f954ec9b640 1 Processor -- start 2023-11-28T17:41:25.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.339+0000 7f954ec9b640 1 -- start start 2023-11-28T17:41:25.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.340+0000 7f954ec9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95480c6140 0x7f954815a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.340+0000 7f954ec9b640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9548151d80 0x7f95481e4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.340+0000 7f954ec9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f954815bc10 0x7f95481487f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.340+0000 7f954ec9b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95481f3e50 con 0x7f95480c6140 2023-11-28T17:41:25.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.340+0000 7f954ec9b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9548149360 con 0x7f954815bc10 2023-11-28T17:41:25.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.340+0000 7f954ec9b640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95481f2ff0 con 0x7f9548151d80 2023-11-28T17:41:25.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.340+0000 7f954ca37640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95480c6140 0x7f954815a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.340+0000 7f954ca37640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95480c6140 0x7f954815a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49324/0 (socket says 172.21.15.98:49324) 2023-11-28T17:41:25.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.340+0000 7f954ca37640 1 -- 172.21.15.98:0/214452536 learned_addr learned my addr 172.21.15.98:0/214452536 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:25.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.340+0000 7f954ca37640 1 -- 172.21.15.98:0/214452536 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9548151d80 msgr2=0x7f95481e4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.340+0000 7f954ca37640 1 --2- 172.21.15.98:0/214452536 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9548151d80 0x7f95481e4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.340+0000 7f954ca37640 1 -- 172.21.15.98:0/214452536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f954815bc10 msgr2=0x7f95481487f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.340+0000 7f954ca37640 1 --2- 172.21.15.98:0/214452536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f954815bc10 0x7f95481487f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.340+0000 7f954ca37640 1 -- 172.21.15.98:0/214452536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95300094d0 con 0x7f95480c6140 2023-11-28T17:41:25.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.340+0000 7f954ca37640 1 --2- 172.21.15.98:0/214452536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95480c6140 0x7f954815a990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9540007ef0 tx=0x7f954000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.341+0000 7f95457fa640 1 -- 172.21.15.98:0/214452536 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540014620 con 0x7f95480c6140 2023-11-28T17:41:25.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.341+0000 7f95457fa640 1 -- 172.21.15.98:0/214452536 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9540014780 con 0x7f95480c6140 2023-11-28T17:41:25.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.341+0000 7f95457fa640 1 -- 172.21.15.98:0/214452536 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540014b30 con 0x7f95480c6140 2023-11-28T17:41:25.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.341+0000 7f954ec9b640 1 -- 172.21.15.98:0/214452536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95481f32d0 con 0x7f95480c6140 2023-11-28T17:41:25.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.341+0000 7f954ec9b640 1 -- 172.21.15.98:0/214452536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95481f3890 con 0x7f95480c6140 2023-11-28T17:41:25.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.342+0000 7f954ec9b640 1 -- 172.21.15.98:0/214452536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9548067bf0 con 0x7f95480c6140 2023-11-28T17:41:25.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.343+0000 7f95457fa640 1 -- 172.21.15.98:0/214452536 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f9540014c90 con 0x7f95480c6140 2023-11-28T17:41:25.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.344+0000 7f95457fa640 1 --2- 172.21.15.98:0/214452536 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f952806b2a0 0x7f952806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.344+0000 7f95457fa640 1 -- 172.21.15.98:0/214452536 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f954008a740 con 0x7f95480c6140 2023-11-28T17:41:25.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.344+0000 7f9547fff640 1 --2- 172.21.15.98:0/214452536 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f952806b2a0 0x7f952806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.345+0000 7f9547fff640 1 --2- 172.21.15.98:0/214452536 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f952806b2a0 0x7f952806d760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9530036000 tx=0x7f9530030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.348+0000 7f95457fa640 1 -- 172.21.15.98:0/214452536 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9540055480 con 0x7f95480c6140 2023-11-28T17:41:25.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.537+0000 7f954ec9b640 1 -- 172.21.15.98:0/214452536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f954806b2d0 con 0x7f95480c6140 2023-11-28T17:41:25.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.539+0000 7f95457fa640 1 -- 172.21.15.98:0/214452536 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16517 (secure 0 0 0) 0x7f9540058aa0 con 0x7f95480c6140 2023-11-28T17:41:25.542 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:25.542 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:41:18.195933+0000","last_up_change":"2023-11-28T17:41:09.556993+0000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:41:10.303280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:41:14.890871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:41:15.704103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6810","nonce":2188704385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6813","nonce":2188704385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6819","nonce":2188704385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6816","nonce":2188704385}]},"public_addr":"172.21.15.98:6810/2188704385","cluster_addr":"172.21.15.98:6813/2188704385","heartbeat_back_addr":"172.21.15.98:6819/2188704385","heartbeat_front_addr":"172.21.15.98:6816/2188704385","state":["exists","up"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6801","nonce":3918226860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6803","nonce":3918226860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6807","nonce":3918226860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6805","nonce":3918226860}]},"public_addr":"172.21.15.98:6801/3918226860","cluster_addr":"172.21.15.98:6803/3918226860","heartbeat_back_addr":"172.21.15.98:6807/3918226860","heartbeat_front_addr":"172.21.15.98:6805/3918226860","state":["exists","up"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6812","nonce":2691532400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6815","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6818","nonce":2691532400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6823","nonce":2691532400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6821","nonce":2691532400}]},"public_addr":"172.21.15.98:6812/2691532400","cluster_addr":"172.21.15.98:6818/2691532400","heartbeat_back_addr":"172.21.15.98:6823/2691532400","heartbeat_front_addr":"172.21.15.98:6821/2691532400","state":["exists","up"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6825","nonce":1905487008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6827","nonce":1905487008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6831","nonce":1905487008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6829","nonce":1905487008}]},"public_addr":"172.21.15.98:6825/1905487008","cluster_addr":"172.21.15.98:6827/1905487008","heartbeat_back_addr":"172.21.15.98:6831/1905487008","heartbeat_front_addr":"172.21.15.98:6829/1905487008","state":["exists","up"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6801","nonce":2974677215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6803","nonce":2974677215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6807","nonce":2974677215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6805","nonce":2974677215}]},"public_addr":"172.21.15.113:6801/2974677215","cluster_addr":"172.21.15.113:6803/2974677215","heartbeat_back_addr":"172.21.15.113:6807/2974677215","heartbeat_front_addr":"172.21.15.113:6805/2974677215","state":["exists","up"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6817","nonce":2685000355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6821","nonce":2685000355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6829","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6831","nonce":2685000355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6823","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6826","nonce":2685000355}]},"public_addr":"172.21.15.113:6817/2685000355","cluster_addr":"172.21.15.113:6821/2685000355","heartbeat_back_addr":"172.21.15.113:6831/2685000355","heartbeat_front_addr":"172.21.15.113:6826/2685000355","state":["exists","up"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6809","nonce":2764300831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6811","nonce":2764300831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6815","nonce":2764300831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6813","nonce":2764300831}]},"public_addr":"172.21.15.113:6809/2764300831","cluster_addr":"172.21.15.113:6811/2764300831","heartbeat_back_addr":"172.21.15.113:6815/2764300831","heartbeat_front_addr":"172.21.15.113:6813/2764300831","state":["exists","up"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6819","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6820","nonce":1656248400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6824","nonce":1656248400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6830","nonce":1656248400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6825","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6827","nonce":1656248400}]},"public_addr":"172.21.15.113:6820/1656248400","cluster_addr":"172.21.15.113:6824/1656248400","heartbeat_back_addr":"172.21.15.113:6830/1656248400","heartbeat_front_addr":"172.21.15.113:6827/1656248400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.379549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.312196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.117494+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.388128+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.360608+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.411101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.355995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.390362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:41:25.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.541+0000 7f954ec9b640 1 -- 172.21.15.98:0/214452536 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f952806b2a0 msgr2=0x7f952806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.541+0000 7f954ec9b640 1 --2- 172.21.15.98:0/214452536 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f952806b2a0 0x7f952806d760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9530036000 tx=0x7f9530030000 comp rx=0 tx=0).stop 2023-11-28T17:41:25.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.541+0000 7f954ec9b640 1 -- 172.21.15.98:0/214452536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95480c6140 msgr2=0x7f954815a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.541+0000 7f954ec9b640 1 --2- 172.21.15.98:0/214452536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95480c6140 0x7f954815a990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9540007ef0 tx=0x7f954000e4b0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.542+0000 7f954ec9b640 1 -- 172.21.15.98:0/214452536 shutdown_connections 2023-11-28T17:41:25.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.542+0000 7f954ec9b640 1 --2- 172.21.15.98:0/214452536 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f952806b2a0 0x7f952806d760 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.542+0000 7f954ec9b640 1 --2- 172.21.15.98:0/214452536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f954815bc10 0x7f95481487f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.542+0000 7f954ec9b640 1 --2- 172.21.15.98:0/214452536 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9548151d80 0x7f95481e4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.542+0000 7f954ec9b640 1 --2- 172.21.15.98:0/214452536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95480c6140 0x7f954815a990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.542+0000 7f954ec9b640 1 -- 172.21.15.98:0/214452536 >> 172.21.15.98:0/214452536 conn(0x7f954806a4d0 msgr2=0x7f95480c4910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.542+0000 7f954ec9b640 1 -- 172.21.15.98:0/214452536 shutdown_connections 2023-11-28T17:41:25.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.542+0000 7f954ec9b640 1 -- 172.21.15.98:0/214452536 wait complete. 2023-11-28T17:41:25.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-28T17:41:25.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-28T17:41:25.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-28T17:41:25.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-11-28T17:41:25.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-11-28T17:41:25.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-11-28T17:41:25.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-11-28T17:41:25.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-11-28T17:41:25.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.868+0000 7fac07fff640 1 Processor -- start 2023-11-28T17:41:25.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.868+0000 7f9540ad1640 1 Processor -- start 2023-11-28T17:41:25.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.869+0000 7f9540ad1640 1 -- start start 2023-11-28T17:41:25.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.870+0000 7fac07fff640 1 -- start start 2023-11-28T17:41:25.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.871+0000 7f9540ad1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f953c153c80 0x7f953c06b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.872+0000 7f9540ad1640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f953c0c7e30 0x7f953c06be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.872+0000 7f9540ad1640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f953c15d230 con 0x7f953c0c69b0 2023-11-28T17:41:25.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.872+0000 7f9540ad1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f953c15e070 con 0x7f953c153c80 2023-11-28T17:41:25.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.872+0000 7f9540ad1640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f953c15d4f0 con 0x7f953c0c7e30 2023-11-28T17:41:25.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.872+0000 7f953b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f953c153c80 0x7f953c06b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.872+0000 7ff6bf7de640 1 Processor -- start 2023-11-28T17:41:25.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.872+0000 7f953b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f953c153c80 0x7f953c06b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49332/0 (socket says 172.21.15.98:49332) 2023-11-28T17:41:25.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.872+0000 7f953b7fe640 1 -- 172.21.15.98:0/2527891655 learned_addr learned my addr 172.21.15.98:0/2527891655 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:25.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.873+0000 7fac07fff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fac08153d60 0x7fac0815dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.872+0000 7f953bfff640 1 --2- 172.21.15.98:0/2527891655 >> v2:172.21.15.113:3300/0 conn(0x7f953c0c7e30 0x7f953c06be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.873+0000 7fac07fff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0815aa80 con 0x7fac080c4c60 2023-11-28T17:41:25.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.873+0000 7fac07fff640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac08159bf0 con 0x7fac080ca380 2023-11-28T17:41:25.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.873+0000 7fac07fff640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac08159e70 con 0x7fac08153d60 2023-11-28T17:41:25.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.874+0000 7fabfe7fc640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fac080ca380 0x7fac080c4550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.98:41326/0 (socket says 172.21.15.98:41326) 2023-11-28T17:41:25.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.874+0000 7fabfe7fc640 1 -- 172.21.15.98:0/2605627336 learned_addr learned my addr 172.21.15.98:0/2605627336 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:25.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.874+0000 7fac06ffd640 1 --2- 172.21.15.98:0/2605627336 >> v2:172.21.15.113:3300/0 conn(0x7fac08153d60 0x7fac0815dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.873+0000 7f953bfff640 1 -- 172.21.15.98:0/2527891655 >> v2:172.21.15.98:3300/0 conn(0x7f953c153c80 msgr2=0x7f953c06b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:25.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.874+0000 7f953bfff640 1 --2- 172.21.15.98:0/2527891655 >> v2:172.21.15.98:3300/0 conn(0x7f953c153c80 0x7f953c06b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.874+0000 7f953bfff640 1 -- 172.21.15.98:0/2527891655 >> v1:172.21.15.113:6789/0 conn(0x7f953c0c69b0 legacy=0x7f953c0c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.874+0000 7f953bfff640 1 -- 172.21.15.98:0/2527891655 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953c15d7d0 con 0x7f953c0c7e30 2023-11-28T17:41:25.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.874+0000 7f953b7fe640 1 --2- 172.21.15.98:0/2527891655 >> v2:172.21.15.98:3300/0 conn(0x7f953c153c80 0x7f953c06b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:41:25.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.873+0000 7ff6bf7de640 1 -- start start 2023-11-28T17:41:25.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.875+0000 7ff6bf7de640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff6b003a010 0x7ff6b0007190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.875+0000 7ff6bf7de640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6b00550f0 0x7ff6b00076d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.875+0000 7ff6bf7de640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b0006180 con 0x7ff6b004f590 2023-11-28T17:41:25.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.875+0000 7ff6bf7de640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b003a3f0 con 0x7ff6b003a010 2023-11-28T17:41:25.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.875+0000 7ff6bf7de640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b00e3220 con 0x7ff6b00550f0 2023-11-28T17:41:25.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.875+0000 7ff6bcd79640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6b00550f0 0x7ff6b00076d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.875+0000 7ff6bcd79640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6b00550f0 0x7ff6b00076d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:55478/0 (socket says 172.21.15.98:55478) 2023-11-28T17:41:25.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.875+0000 7ff6bcd79640 1 -- 172.21.15.98:0/3794748785 learned_addr learned my addr 172.21.15.98:0/3794748785 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:25.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.876+0000 7ff6bcd79640 1 -- 172.21.15.98:0/3794748785 >> v2:172.21.15.98:3300/0 conn(0x7ff6b003a010 msgr2=0x7ff6b0007190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:41:25.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.875+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623489926 0 0) 0x7fac08159bf0 con 0x7fac080ca380 2023-11-28T17:41:25.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf4003520 con 0x7fac080ca380 2023-11-28T17:41:25.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7ff6affff640 1 -- 172.21.15.98:0/3794748785 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436467799 0 0) 0x7ff6b0006180 con 0x7ff6b004f590 2023-11-28T17:41:25.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7fa178b9e640 1 Processor -- start 2023-11-28T17:41:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413161256 0 0) 0x7fac0815aa80 con 0x7fac080c4c60 2023-11-28T17:41:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac08159bf0 con 0x7fac080c4c60 2023-11-28T17:41:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 <== mon.1 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 805419414 0 0) 0x7fabf4003520 con 0x7fac080ca380 2023-11-28T17:41:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 >> v2:172.21.15.113:3300/0 conn(0x7fac08153d60 msgr2=0x7fac0815dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7fac067fc640 1 --2- 172.21.15.98:0/2605627336 >> v2:172.21.15.113:3300/0 conn(0x7fac08153d60 0x7fac0815dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 >> v1:172.21.15.98:6789/0 conn(0x7fac080c4c60 legacy=0x7fac080cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0815a150 con 0x7fac080ca380 2023-11-28T17:41:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 <== mon.1 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7fabf80019f0 con 0x7fac080ca380 2023-11-28T17:41:25.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 >> v1:172.21.15.113:6789/0 conn(0x7fac080ca380 legacy=0x7fac080c4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7fac067fc640 1 --2- 172.21.15.98:0/2605627336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabf4003b50 0x7fabf4003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7fac067fc640 1 --2- 172.21.15.98:0/2605627336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf40047c0 0x7fabf4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7fac067fc640 1 --2- 172.21.15.98:0/2605627336 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf4005180 0x7fabf400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.878+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac08159e70 con 0x7fabf4003b50 2023-11-28T17:41:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.878+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf400c060 con 0x7fabf40047c0 2023-11-28T17:41:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.878+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf400c220 con 0x7fabf4005180 2023-11-28T17:41:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.878+0000 7fabfe7fc640 1 --2- 172.21.15.98:0/2605627336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf40047c0 0x7fabf4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.878+0000 7fac077fe640 1 --2- 172.21.15.98:0/2605627336 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf4005180 0x7fabf400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.879+0000 7fabfe7fc640 1 -- 172.21.15.98:0/2605627336 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf4005180 msgr2=0x7fabf400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.879+0000 7fabfe7fc640 1 --2- 172.21.15.98:0/2605627336 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabf4005180 0x7fabf400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.879+0000 7fabfe7fc640 1 -- 172.21.15.98:0/2605627336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabf4003b50 msgr2=0x7fabf4003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.879+0000 7fabfe7fc640 1 --2- 172.21.15.98:0/2605627336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabf4003b50 0x7fabf4003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.879+0000 7fabfe7fc640 1 -- 172.21.15.98:0/2605627336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf4003520 con 0x7fabf40047c0 2023-11-28T17:41:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7f953bfff640 1 --2- 172.21.15.98:0/2527891655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f953c0c7e30 0x7f953c06be80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f953c067c00 tx=0x7f952c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=278c043d14a51c7f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.878+0000 7f953a7fc640 1 -- 172.21.15.98:0/2527891655 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952c030610 con 0x7f953c0c7e30 2023-11-28T17:41:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.878+0000 7f953a7fc640 1 -- 172.21.15.98:0/2527891655 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952c030770 con 0x7f953c0c7e30 2023-11-28T17:41:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.878+0000 7f953a7fc640 1 -- 172.21.15.98:0/2527891655 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952c030b00 con 0x7f953c0c7e30 2023-11-28T17:41:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.879+0000 7f9540ad1640 1 -- 172.21.15.98:0/2527891655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f953c0c7e30 msgr2=0x7f953c06be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.879+0000 7f9540ad1640 1 --2- 172.21.15.98:0/2527891655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f953c0c7e30 0x7f953c06be80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f953c067c00 tx=0x7f952c02b960 comp rx=0 tx=0).stop 2023-11-28T17:41:25.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.879+0000 7f9540ad1640 1 -- 172.21.15.98:0/2527891655 shutdown_connections 2023-11-28T17:41:25.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.879+0000 7f9540ad1640 1 --2- 172.21.15.98:0/2527891655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f953c0c7e30 0x7f953c06be80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.878+0000 7ff6bd57a640 1 --2- 172.21.15.98:0/3794748785 >> v2:172.21.15.98:3300/0 conn(0x7ff6b003a010 0x7ff6b0007190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.879+0000 7f9540ad1640 1 --2- 172.21.15.98:0/2527891655 >> v2:172.21.15.98:3300/0 conn(0x7f953c153c80 0x7f953c06b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.880+0000 7f9540ad1640 1 -- 172.21.15.98:0/2527891655 >> 172.21.15.98:0/2527891655 conn(0x7f953c0cf300 msgr2=0x7f953c0cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.878+0000 7ff6bcd79640 1 --2- 172.21.15.98:0/3794748785 >> v2:172.21.15.98:3300/0 conn(0x7ff6b003a010 0x7ff6b0007190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.878+0000 7ff6bcd79640 1 -- 172.21.15.98:0/3794748785 >> v1:172.21.15.113:6789/0 conn(0x7ff6b004f590 legacy=0x7ff6b0050140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.878+0000 7ff6bcd79640 1 -- 172.21.15.98:0/3794748785 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b00063d0 con 0x7ff6b00550f0 2023-11-28T17:41:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.880+0000 7fac077fe640 1 -- 172.21.15.98:0/2605627336 reap_dead start 2023-11-28T17:41:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.880+0000 7f9540ad1640 1 -- 172.21.15.98:0/2527891655 shutdown_connections 2023-11-28T17:41:25.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.880+0000 7f9540ad1640 1 -- 172.21.15.98:0/2527891655 wait complete. 2023-11-28T17:41:25.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.877+0000 7fa178b9e640 1 -- start start 2023-11-28T17:41:25.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.879+0000 7fa178b9e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa174071b50 0x7fa17415e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.879+0000 7fa178b9e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1740c6630 con 0x7fa174071770 2023-11-28T17:41:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.879+0000 7fa178b9e640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17406e710 con 0x7fa1740c5a60 2023-11-28T17:41:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.879+0000 7fa178b9e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa17406de70 con 0x7fa174071b50 2023-11-28T17:41:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.880+0000 7fa172d9d640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fa1740c5a60 0x7fa174168a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.98:41348/0 (socket says 172.21.15.98:41348) 2023-11-28T17:41:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.881+0000 7fa172d9d640 1 -- 172.21.15.98:0/1817189691 learned_addr learned my addr 172.21.15.98:0/1817189691 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.881+0000 7ff6bcd79640 1 --2- 172.21.15.98:0/3794748785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6b00550f0 0x7ff6b00076d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4009870 tx=0x7ff6b402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=753a60e7427839e9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.881+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143651306 0 0) 0x7fa16c001490 con 0x7fa1740c5a60 2023-11-28T17:41:25.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.881+0000 7ff6affff640 1 -- 172.21.15.98:0/3794748785 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4030840 con 0x7ff6b00550f0 2023-11-28T17:41:25.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.882+0000 7ff6affff640 1 -- 172.21.15.98:0/3794748785 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b40309a0 con 0x7ff6b00550f0 2023-11-28T17:41:25.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.881+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa160003520 con 0x7fa1740c5a60 2023-11-28T17:41:25.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.882+0000 7fa16bfff640 1 --2- 172.21.15.98:0/1817189691 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa174071b50 0x7fa17415e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.882+0000 7ff6affff640 1 -- 172.21.15.98:0/3794748785 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4030d30 con 0x7ff6b00550f0 2023-11-28T17:41:25.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.882+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005481495 0 0) 0x7fa1740c6630 con 0x7fa174071770 2023-11-28T17:41:25.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.882+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa16c001490 con 0x7fa174071770 2023-11-28T17:41:25.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.882+0000 7f9540ad1640 1 Processor -- start 2023-11-28T17:41:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.882+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 <== mon.1 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2757035954 0 0) 0x7fa160003520 con 0x7fa1740c5a60 2023-11-28T17:41:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.882+0000 7f6e547c9640 1 Processor -- start 2023-11-28T17:41:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.882+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa174071b50 msgr2=0x7fa17415e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.882+0000 7fa171d9b640 1 --2- 172.21.15.98:0/1817189691 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa174071b50 0x7fa17415e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 >> v1:172.21.15.98:6789/0 conn(0x7fa174071770 legacy=0x7fa174157fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17406e150 con 0x7fa1740c5a60 2023-11-28T17:41:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 <== mon.1 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7fa16c001b80 con 0x7fa1740c5a60 2023-11-28T17:41:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 >> v1:172.21.15.113:6789/0 conn(0x7fa1740c5a60 legacy=0x7fa174168a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7fa171d9b640 1 --2- 172.21.15.98:0/1817189691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa160003b30 0x7fa160003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7fa171d9b640 1 --2- 172.21.15.98:0/1817189691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1600047c0 0x7fa160004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7fa171d9b640 1 --2- 172.21.15.98:0/1817189691 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa160005180 0x7fa16000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa164001210 con 0x7fa160003b30 2023-11-28T17:41:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17406de70 con 0x7fa1600047c0 2023-11-28T17:41:25.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa16000c080 con 0x7fa160005180 2023-11-28T17:41:25.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7f9540ad1640 1 -- start start 2023-11-28T17:41:25.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7ff6bf7de640 1 -- 172.21.15.98:0/3794748785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6b00550f0 msgr2=0x7ff6b00076d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7ff6bf7de640 1 --2- 172.21.15.98:0/3794748785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6b00550f0 0x7ff6b00076d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4009870 tx=0x7ff6b402b980 comp rx=0 tx=0).stop 2023-11-28T17:41:25.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7ff6bf7de640 1 -- 172.21.15.98:0/3794748785 shutdown_connections 2023-11-28T17:41:25.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7ff6bf7de640 1 --2- 172.21.15.98:0/3794748785 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6b00550f0 0x7ff6b00076d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7ff6bf7de640 1 --2- 172.21.15.98:0/3794748785 >> v2:172.21.15.98:3300/0 conn(0x7ff6b003a010 0x7ff6b0007190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7ff6bf7de640 1 -- 172.21.15.98:0/3794748785 >> 172.21.15.98:0/3794748785 conn(0x7ff6b00545d0 msgr2=0x7ff6b00549e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7fa172d9d640 1 --2- 172.21.15.98:0/1817189691 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa160005180 0x7fa16000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7f9540ad1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c0c69b0 0x7f953c1f7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7fa16bfff640 1 --2- 172.21.15.98:0/1817189691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1600047c0 0x7fa160004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7f9540ad1640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f953c0c7e30 0x7f953c1f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7f9540ad1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f953c153c80 0x7f953c0b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7fa17259c640 1 --2- 172.21.15.98:0/1817189691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa160003b30 0x7fa160003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7fabfe7fc640 1 --2- 172.21.15.98:0/2605627336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf40047c0 0x7fabf4004c40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fabf800a710 tx=0x7fabf40057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34520e279cf706e2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7f9540ad1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953c0c3100 con 0x7f953c0c69b0 2023-11-28T17:41:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7f953b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c0c69b0 0x7f953c1f7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7f9540ad1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953c1548a0 con 0x7f953c153c80 2023-11-28T17:41:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7f9540ad1640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f953c1f8150 con 0x7f953c0c7e30 2023-11-28T17:41:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7f953b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c0c69b0 0x7f953c1f7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49386/0 (socket says 172.21.15.98:49386) 2023-11-28T17:41:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7f953b7fe640 1 -- 172.21.15.98:0/394266859 learned_addr learned my addr 172.21.15.98:0/394266859 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7f953affd640 1 --2- 172.21.15.98:0/394266859 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f953c0c7e30 0x7f953c1f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7f953bfff640 1 --2- 172.21.15.98:0/394266859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f953c153c80 0x7f953c0b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.883+0000 7f6e547c9640 1 -- start start 2023-11-28T17:41:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7f6e547c9640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e4c1540d0 0x7f6e4c15dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.884+0000 7f6e547c9640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6e4c0cb070 0x7f6e4c06b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7f6e547c9640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4c06a660 con 0x7f6e4c0ca280 2023-11-28T17:41:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7f6e547c9640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e4c0cbdd0 con 0x7f6e4c0cb070 2023-11-28T17:41:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7f6e547c9640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c1544b0 con 0x7f6e4c1540d0 2023-11-28T17:41:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7f953b7fe640 1 -- 172.21.15.98:0/394266859 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f953c0c7e30 msgr2=0x7f953c1f7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7f953b7fe640 1 --2- 172.21.15.98:0/394266859 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f953c0c7e30 0x7f953c1f7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7ff6bf7de640 1 -- 172.21.15.98:0/3794748785 shutdown_connections 2023-11-28T17:41:25.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7ff6bf7de640 1 -- 172.21.15.98:0/3794748785 wait complete. 2023-11-28T17:41:25.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7f953b7fe640 1 -- 172.21.15.98:0/394266859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f953c153c80 msgr2=0x7f953c0b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7f953b7fe640 1 --2- 172.21.15.98:0/394266859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f953c153c80 0x7f953c0b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7f953b7fe640 1 -- 172.21.15.98:0/394266859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95300094d0 con 0x7f953c0c69b0 2023-11-28T17:41:25.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7fa17259c640 1 -- 172.21.15.98:0/1817189691 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa160005180 msgr2=0x7fa16000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7f953b7fe640 1 --2- 172.21.15.98:0/394266859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c0c69b0 0x7f953c1f7560 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f953000e9b0 tx=0x7f953000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.886+0000 7f9523fff640 1 -- 172.21.15.98:0/394266859 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953000c9a0 con 0x7f953c0c69b0 2023-11-28T17:41:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.886+0000 7f9540ad1640 1 -- 172.21.15.98:0/394266859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f952c009400 con 0x7f953c0c69b0 2023-11-28T17:41:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.886+0000 7f6e52d66640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f6e4c0ca280 0x7f6e4c159c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.98:41364/0 (socket says 172.21.15.98:41364) 2023-11-28T17:41:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.886+0000 7f6e52d66640 1 -- 172.21.15.98:0/1140303381 learned_addr learned my addr 172.21.15.98:0/1140303381 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.886+0000 7f6e51d64640 1 --2- 172.21.15.98:0/1140303381 >> v2:172.21.15.113:3300/0 conn(0x7f6e4c0cb070 0x7f6e4c06b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.886+0000 7f9540ad1640 1 -- 172.21.15.98:0/394266859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f953c156670 con 0x7f953c0c69b0 2023-11-28T17:41:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.887+0000 7ff6bf7de640 1 Processor -- start 2023-11-28T17:41:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7fa17259c640 1 --2- 172.21.15.98:0/1817189691 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa160005180 0x7fa16000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7fa17259c640 1 -- 172.21.15.98:0/1817189691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1600047c0 msgr2=0x7fa160004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:25.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7fa17259c640 1 --2- 172.21.15.98:0/1817189691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1600047c0 0x7fa160004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.885+0000 7fa17259c640 1 -- 172.21.15.98:0/1817189691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa160003520 con 0x7fa160003b30 2023-11-28T17:41:25.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.887+0000 7f372ac72640 1 Processor -- start 2023-11-28T17:41:25.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.887+0000 7f6e51d64640 1 -- 172.21.15.98:0/1140303381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e4c1540d0 msgr2=0x7f6e4c15dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T17:41:25.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.887+0000 7f6e51d64640 1 --2- 172.21.15.98:0/1140303381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e4c1540d0 0x7f6e4c15dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.887+0000 7f6e51d64640 1 -- 172.21.15.98:0/1140303381 >> v1:172.21.15.113:6789/0 conn(0x7f6e4c0ca280 legacy=0x7f6e4c159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.887+0000 7f6e51d64640 1 -- 172.21.15.98:0/1140303381 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c15e1b0 con 0x7f6e4c0cb070 2023-11-28T17:41:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.887+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8034630 con 0x7fabf40047c0 2023-11-28T17:41:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.887+0000 7fa16bfff640 1 --2- 172.21.15.98:0/1817189691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1600047c0 0x7fa160004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:41:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.887+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf802f070 con 0x7fabf40047c0 2023-11-28T17:41:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.887+0000 7fac067fc640 1 -- 172.21.15.98:0/2605627336 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8034a70 con 0x7fabf40047c0 2023-11-28T17:41:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.888+0000 7fac07fff640 1 -- 172.21.15.98:0/2605627336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf40047c0 msgr2=0x7fabf4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.888+0000 7fac07fff640 1 --2- 172.21.15.98:0/2605627336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf40047c0 0x7fabf4004c40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fabf800a710 tx=0x7fabf40057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.887+0000 7f9523fff640 1 -- 172.21.15.98:0/394266859 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953000cb00 con 0x7f953c0c69b0 2023-11-28T17:41:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.887+0000 7f9523fff640 1 -- 172.21.15.98:0/394266859 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953000c9a0 con 0x7f953c0c69b0 2023-11-28T17:41:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.888+0000 7f9540ad1640 1 -- 172.21.15.98:0/394266859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f951c000fc0 con 0x7f953c0c69b0 2023-11-28T17:41:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.888+0000 7fa17259c640 1 --2- 172.21.15.98:0/1817189691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa160003b30 0x7fa160003f10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa16400c1e0 tx=0x7fa1600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=957e17a8bc90b6bd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.888+0000 7f372ac72640 1 -- start start 2023-11-28T17:41:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.887+0000 7ff6bf7de640 1 -- start start 2023-11-28T17:41:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.888+0000 7f6e51d64640 1 --2- 172.21.15.98:0/1140303381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e4c0cb070 0x7f6e4c06b950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6e44009d90 tx=0x7f6e4402bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a974efe9d2bbcd4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.888+0000 7ff6bf7de640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6b003a010 0x7ff6b000c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.888+0000 7ff6bf7de640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6b004f590 0x7ff6b000cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7ff6bf7de640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6b00550f0 0x7ff6b0048ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7ff6bf7de640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b000c470 con 0x7ff6b003a010 2023-11-28T17:41:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7ff6bf7de640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b000af30 con 0x7ff6b00550f0 2023-11-28T17:41:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7ff6bf7de640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b0048150 con 0x7ff6b004f590 2023-11-28T17:41:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7fac07fff640 1 -- 172.21.15.98:0/2605627336 shutdown_connections 2023-11-28T17:41:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7fac07fff640 1 --2- 172.21.15.98:0/2605627336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf40047c0 0x7fabf4004c40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7fac07fff640 1 -- 172.21.15.98:0/2605627336 >> 172.21.15.98:0/2605627336 conn(0x7fac0806ade0 msgr2=0x7fac0806b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7f6e51563640 1 -- 172.21.15.98:0/1140303381 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4402cc70 con 0x7f6e4c0cb070 2023-11-28T17:41:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16402cac0 con 0x7fa160003b30 2023-11-28T17:41:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7f6e51563640 1 -- 172.21.15.98:0/1140303381 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4402cdd0 con 0x7f6e4c0cb070 2023-11-28T17:41:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7fac07fff640 1 -- 172.21.15.98:0/2605627336 shutdown_connections 2023-11-28T17:41:25.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16402cc40 con 0x7fa160003b30 2023-11-28T17:41:25.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7fa171d9b640 1 -- 172.21.15.98:0/1817189691 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa164034680 con 0x7fa160003b30 2023-11-28T17:41:25.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7ff6bcd79640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6b004f590 0x7ff6b000cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7ff6bdd7b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6b00550f0 0x7ff6b0048ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7ff6bdd7b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6b00550f0 0x7ff6b0048ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:42670/0 (socket says 172.21.15.98:42670) 2023-11-28T17:41:25.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7ff6bdd7b640 1 -- 172.21.15.98:0/680139394 learned_addr learned my addr 172.21.15.98:0/680139394 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:25.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.890+0000 7f6e51563640 1 -- 172.21.15.98:0/1140303381 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e44034660 con 0x7f6e4c0cb070 2023-11-28T17:41:25.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.889+0000 7fac07fff640 1 -- 172.21.15.98:0/2605627336 wait complete. 2023-11-28T17:41:25.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.890+0000 7f372ac72640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f372414fdd0 0x7f3724071120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.890+0000 7ff6bdd7b640 1 -- 172.21.15.98:0/680139394 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6b004f590 msgr2=0x7ff6b000cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.890+0000 7ff6bdd7b640 1 --2- 172.21.15.98:0/680139394 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff6b004f590 0x7ff6b000cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.890+0000 7ff6bdd7b640 1 -- 172.21.15.98:0/680139394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6b003a010 msgr2=0x7ff6b000c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.890+0000 7ff6bdd7b640 1 --2- 172.21.15.98:0/680139394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6b003a010 0x7ff6b000c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.890+0000 7f6e547c9640 1 -- 172.21.15.98:0/1140303381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e4c0cb070 msgr2=0x7f6e4c06b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.890+0000 7f6e547c9640 1 --2- 172.21.15.98:0/1140303381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e4c0cb070 0x7f6e4c06b950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6e44009d90 tx=0x7f6e4402bec0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.890+0000 7f372ac72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3724071940 0x7f372406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.891+0000 7ff6bdd7b640 1 -- 172.21.15.98:0/680139394 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b40094d0 con 0x7ff6b00550f0 2023-11-28T17:41:25.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.891+0000 7f372ac72640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37241501b0 con 0x7f372406ddf0 2023-11-28T17:41:25.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.891+0000 7f372ac72640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37241592e0 con 0x7f3724071940 2023-11-28T17:41:25.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.891+0000 7f372ac72640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37240c7620 con 0x7f372414fdd0 2023-11-28T17:41:25.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.891+0000 7fa178b9e640 1 -- 172.21.15.98:0/1817189691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa160003b30 msgr2=0x7fa160003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.891+0000 7fa178b9e640 1 --2- 172.21.15.98:0/1817189691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa160003b30 0x7fa160003f10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa16400c1e0 tx=0x7fa1600057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.891+0000 7fa172d9d640 1 -- 172.21.15.98:0/1817189691 reap_dead start 2023-11-28T17:41:25.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.891+0000 7f3728a0e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f372414fdd0 0x7f3724071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.891+0000 7f3728a0e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f372414fdd0 0x7f3724071120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:55558/0 (socket says 172.21.15.98:55558) 2023-11-28T17:41:25.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.891+0000 7f3728a0e640 1 -- 172.21.15.98:0/1766953126 learned_addr learned my addr 172.21.15.98:0/1766953126 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:25.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.891+0000 7ff6bdd7b640 1 --2- 172.21.15.98:0/680139394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6b00550f0 0x7ff6b0048ff0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff6a800c430 tx=0x7ff6a800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7fa178b9e640 1 -- 172.21.15.98:0/1817189691 shutdown_connections 2023-11-28T17:41:25.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7fa178b9e640 1 -- 172.21.15.98:0/1817189691 >> 172.21.15.98:0/1817189691 conn(0x7fa17406c0f0 msgr2=0x7fa17406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7ff6adffb640 1 -- 172.21.15.98:0/680139394 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a80035f0 con 0x7ff6b00550f0 2023-11-28T17:41:25.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7ff6adffb640 1 -- 172.21.15.98:0/680139394 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a8003770 con 0x7ff6b00550f0 2023-11-28T17:41:25.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7f3723fff640 1 --2- 172.21.15.98:0/1766953126 >> v2:172.21.15.113:3300/0 conn(0x7f3724071940 0x7f372406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7ff6bf7de640 1 -- 172.21.15.98:0/680139394 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b004aad0 con 0x7ff6b00550f0 2023-11-28T17:41:25.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7ff6adffb640 1 -- 172.21.15.98:0/680139394 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8003b20 con 0x7ff6b00550f0 2023-11-28T17:41:25.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7fac07fff640 1 Processor -- start 2023-11-28T17:41:25.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7fa178b9e640 1 -- 172.21.15.98:0/1817189691 shutdown_connections 2023-11-28T17:41:25.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7fac07fff640 1 -- start start 2023-11-28T17:41:25.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7f6e547c9640 1 -- 172.21.15.98:0/1140303381 shutdown_connections 2023-11-28T17:41:25.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7f6e547c9640 1 --2- 172.21.15.98:0/1140303381 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e4c0cb070 0x7f6e4c06b950 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7f6e547c9640 1 --2- 172.21.15.98:0/1140303381 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e4c1540d0 0x7f6e4c15dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7f6e547c9640 1 -- 172.21.15.98:0/1140303381 >> 172.21.15.98:0/1140303381 conn(0x7f6e4c0cf490 msgr2=0x7f6e4c0cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7f6e547c9640 1 -- 172.21.15.98:0/1140303381 shutdown_connections 2023-11-28T17:41:25.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7f3728a0e640 1 -- 172.21.15.98:0/1766953126 >> v2:172.21.15.113:3300/0 conn(0x7f3724071940 msgr2=0x7f372406e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:25.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7f3728a0e640 1 --2- 172.21.15.98:0/1766953126 >> v2:172.21.15.113:3300/0 conn(0x7f3724071940 0x7f372406e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7f3728a0e640 1 -- 172.21.15.98:0/1766953126 >> v1:172.21.15.113:6789/0 conn(0x7f372406ddf0 legacy=0x7f3724159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.893+0000 7fa178b9e640 1 -- 172.21.15.98:0/1817189691 wait complete. 2023-11-28T17:41:25.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.893+0000 7ff6bf7de640 1 -- 172.21.15.98:0/680139394 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b000a8e0 con 0x7ff6b00550f0 2023-11-28T17:41:25.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.893+0000 7fac07fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf40047c0 0x7fac0815cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.893+0000 7f6e547c9640 1 -- 172.21.15.98:0/1140303381 wait complete. 2023-11-28T17:41:25.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.892+0000 7f3728a0e640 1 -- 172.21.15.98:0/1766953126 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3724071660 con 0x7f372414fdd0 2023-11-28T17:41:25.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.894+0000 7fac07fff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac080c6780 0x7fac080b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.894+0000 7fac07fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac080c6be0 0x7fac081f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.894+0000 7fac07fff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac081f4c50 con 0x7fac080c6be0 2023-11-28T17:41:25.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.894+0000 7fac07fff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac080c9430 con 0x7fabf40047c0 2023-11-28T17:41:25.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.894+0000 7fac07fff640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac080b4a30 con 0x7fac080c6780 2023-11-28T17:41:25.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.894+0000 7f3728a0e640 1 --2- 172.21.15.98:0/1766953126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f372414fdd0 0x7f3724071120 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f371800bb10 tx=0x7f371802e220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=734508c0a570c7a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.894+0000 7f9523fff640 1 -- 172.21.15.98:0/394266859 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f953000cc70 con 0x7f953c0c69b0 2023-11-28T17:41:25.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.894+0000 7fac06ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf40047c0 0x7fac0815cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.894+0000 7fac077fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac080c6be0 0x7fac081f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.894+0000 7fac077fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac080c6be0 0x7fac081f71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49406/0 (socket says 172.21.15.98:49406) 2023-11-28T17:41:25.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.894+0000 7fac077fe640 1 -- 172.21.15.98:0/1180359512 learned_addr learned my addr 172.21.15.98:0/1180359512 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:25.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.895+0000 7fabfe7fc640 1 --2- 172.21.15.98:0/1180359512 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac080c6780 0x7fac080b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.895+0000 7fac077fe640 1 -- 172.21.15.98:0/1180359512 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac080c6780 msgr2=0x7fac080b4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.895+0000 7ff6bf7de640 1 -- 172.21.15.98:0/680139394 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff6b0006c70 con 0x7ff6b00550f0 2023-11-28T17:41:25.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.895+0000 7fac077fe640 1 --2- 172.21.15.98:0/1180359512 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fac080c6780 0x7fac080b4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.895+0000 7fac077fe640 1 -- 172.21.15.98:0/1180359512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf40047c0 msgr2=0x7fac0815cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.895+0000 7fac077fe640 1 --2- 172.21.15.98:0/1180359512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf40047c0 0x7fac0815cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.895+0000 7fac077fe640 1 -- 172.21.15.98:0/1180359512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac080c96d0 con 0x7fac080c6be0 2023-11-28T17:41:25.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.895+0000 7f6e547c9640 1 Processor -- start 2023-11-28T17:41:25.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.896+0000 7ff6adffb640 1 -- 172.21.15.98:0/680139394 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff6a8003c80 con 0x7ff6b00550f0 2023-11-28T17:41:25.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.896+0000 7f6e547c9640 1 -- start start 2023-11-28T17:41:25.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.896+0000 7f099cec9640 1 Processor -- start 2023-11-28T17:41:25.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.896+0000 7fac077fe640 1 --2- 172.21.15.98:0/1180359512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac080c6be0 0x7fac081f71d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fac080c50c0 tx=0x7fac0000fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.896+0000 7f37237fe640 1 -- 172.21.15.98:0/1766953126 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718036440 con 0x7f372414fdd0 2023-11-28T17:41:25.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.896+0000 7f37237fe640 1 -- 172.21.15.98:0/1766953126 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37180365a0 con 0x7f372414fdd0 2023-11-28T17:41:25.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.896+0000 7f37237fe640 1 -- 172.21.15.98:0/1766953126 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718036970 con 0x7f372414fdd0 2023-11-28T17:41:25.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.896+0000 7f099cec9640 1 -- start start 2023-11-28T17:41:25.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.896+0000 7fac04ff9640 1 -- 172.21.15.98:0/1180359512 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac000050d0 con 0x7fac080c6be0 2023-11-28T17:41:25.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.896+0000 7fac04ff9640 1 -- 172.21.15.98:0/1180359512 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac00005250 con 0x7fac080c6be0 2023-11-28T17:41:25.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.896+0000 7fac04ff9640 1 -- 172.21.15.98:0/1180359512 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0000d680 con 0x7fac080c6be0 2023-11-28T17:41:25.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.897+0000 7f3414875640 1 Processor -- start 2023-11-28T17:41:25.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.897+0000 7f3414875640 1 -- start start 2023-11-28T17:41:25.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.897+0000 7f9523fff640 1 --2- 172.21.15.98:0/394266859 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f952806b300 0x7f952806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.897+0000 7ff6adffb640 1 --2- 172.21.15.98:0/680139394 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff69c06b2f0 0x7ff69c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.897+0000 7f372ac72640 1 -- 172.21.15.98:0/1766953126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f372414fdd0 msgr2=0x7f3724071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.897+0000 7f372ac72640 1 --2- 172.21.15.98:0/1766953126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f372414fdd0 0x7f3724071120 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f371800bb10 tx=0x7f371802e220 comp rx=0 tx=0).stop 2023-11-28T17:41:25.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7f372ac72640 1 -- 172.21.15.98:0/1766953126 shutdown_connections 2023-11-28T17:41:25.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7f372ac72640 1 --2- 172.21.15.98:0/1766953126 >> v2:172.21.15.113:3300/0 conn(0x7f3724071940 0x7f372406e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7f372ac72640 1 --2- 172.21.15.98:0/1766953126 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f372414fdd0 0x7f3724071120 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7f372ac72640 1 -- 172.21.15.98:0/1766953126 >> 172.21.15.98:0/1766953126 conn(0x7f372406c0f0 msgr2=0x7f372406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7f372ac72640 1 -- 172.21.15.98:0/1766953126 shutdown_connections 2023-11-28T17:41:25.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7f372ac72640 1 -- 172.21.15.98:0/1766953126 wait complete. 2023-11-28T17:41:25.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7f6e547c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e4c0ca280 0x7f6e4c1f4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7f6e547c9640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e4c0cb070 0x7f6e4c1eada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7fa178b9e640 1 Processor -- start 2023-11-28T17:41:25.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.897+0000 7fac07fff640 1 -- 172.21.15.98:0/1180359512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac080c8650 con 0x7fac080c6be0 2023-11-28T17:41:25.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.897+0000 7fac07fff640 1 -- 172.21.15.98:0/1180359512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac080c8c10 con 0x7fac080c6be0 2023-11-28T17:41:25.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7f6e547c9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e4c1540d0 0x7f6e4c1f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7ff6adffb640 1 -- 172.21.15.98:0/680139394 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6a8089220 con 0x7ff6b00550f0 2023-11-28T17:41:25.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7ff6adffb640 1 --2- 172.21.15.98:0/680139394 >> [v2:172.21.15.98:6809/2691532400,v1:172.21.15.98:6812/2691532400] conn(0x7ff69c073490 0x7ff69c0758b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7ff6adffb640 1 -- 172.21.15.98:0/680139394 --> [v2:172.21.15.98:6809/2691532400,v1:172.21.15.98:6812/2691532400] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff69c075f60 con 0x7ff69c073490 2023-11-28T17:41:25.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7ff6adffb640 1 -- 172.21.15.98:0/680139394 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6a808b800 con 0x7ff6b00550f0 2023-11-28T17:41:25.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7f6e547c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c1f5190 con 0x7f6e4c0ca280 2023-11-28T17:41:25.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7f6e547c9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c0c4310 con 0x7f6e4c1540d0 2023-11-28T17:41:25.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.898+0000 7f6e547c9640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c0c45f0 con 0x7f6e4c0cb070 2023-11-28T17:41:25.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.899+0000 7ff6bd57a640 1 --2- 172.21.15.98:0/680139394 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff69c06b2f0 0x7ff69c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.899+0000 7f6e52d66640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e4c1540d0 0x7f6e4c1f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.899+0000 7f6e52d66640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e4c1540d0 0x7f6e4c1f59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:42696/0 (socket says 172.21.15.98:42696) 2023-11-28T17:41:25.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.899+0000 7f6e52d66640 1 -- 172.21.15.98:0/3817627943 learned_addr learned my addr 172.21.15.98:0/3817627943 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:25.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.899+0000 7f6e51d64640 1 --2- 172.21.15.98:0/3817627943 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e4c0cb070 0x7f6e4c1eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.899+0000 7ff6bcd79640 1 --2- 172.21.15.98:0/680139394 >> [v2:172.21.15.98:6809/2691532400,v1:172.21.15.98:6812/2691532400] conn(0x7ff69c073490 0x7ff69c0758b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.899+0000 7f9523fff640 1 -- 172.21.15.98:0/394266859 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f953008a2b0 con 0x7f953c0c69b0 2023-11-28T17:41:25.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.899+0000 7f953affd640 1 --2- 172.21.15.98:0/394266859 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f952806b300 0x7f952806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.899+0000 7f3414875640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f34100cb070 0x7f341006b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.899+0000 7f3414875640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f34100ca280 0x7f34100ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.899+0000 7f3414875640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341006a660 con 0x7f34101540d0 2023-11-28T17:41:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.899+0000 7f3414875640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34100cbdd0 con 0x7f34100cb070 2023-11-28T17:41:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.899+0000 7f3414875640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34101544b0 con 0x7f34100ca280 2023-11-28T17:41:25.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.899+0000 7fac07fff640 1 -- 172.21.15.98:0/1180359512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fac0815a710 con 0x7fac080c6be0 2023-11-28T17:41:25.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.900+0000 7f6e52d66640 1 -- 172.21.15.98:0/3817627943 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e4c0cb070 msgr2=0x7f6e4c1eada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.900+0000 7f6e52d66640 1 --2- 172.21.15.98:0/3817627943 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6e4c0cb070 0x7f6e4c1eada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.900+0000 7f6e52d66640 1 -- 172.21.15.98:0/3817627943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e4c0ca280 msgr2=0x7f6e4c1f4c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:41:25.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.900+0000 7f6e52d66640 1 --2- 172.21.15.98:0/3817627943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e4c0ca280 0x7f6e4c1f4c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.900+0000 7f340dd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f34100cb070 0x7f341006b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.900+0000 7f6e52d66640 1 -- 172.21.15.98:0/3817627943 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e44009a70 con 0x7f6e4c1540d0 2023-11-28T17:41:25.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.900+0000 7ff6bd57a640 1 --2- 172.21.15.98:0/680139394 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff69c06b2f0 0x7ff69c06d7b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff6b80699e0 tx=0x7ff6b806e040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.900+0000 7f372ac72640 1 Processor -- start 2023-11-28T17:41:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.900+0000 7f340dd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f34100cb070 0x7f341006b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49410/0 (socket says 172.21.15.98:49410) 2023-11-28T17:41:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.900+0000 7f340dd9b640 1 -- 172.21.15.98:0/4129095329 learned_addr learned my addr 172.21.15.98:0/4129095329 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.900+0000 7f9523fff640 1 --2- 172.21.15.98:0/394266859 >> [v2:172.21.15.98:6800/3918226860,v1:172.21.15.98:6801/3918226860] conn(0x7f9528073520 0x7f9528075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.900+0000 7f9523fff640 1 -- 172.21.15.98:0/394266859 --> [v2:172.21.15.98:6800/3918226860,v1:172.21.15.98:6801/3918226860] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9528075ff0 con 0x7f9528073520 2023-11-28T17:41:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.900+0000 7f9523fff640 1 -- 172.21.15.98:0/394266859 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f953008a660 con 0x7f953c0c69b0 2023-11-28T17:41:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.901+0000 7f953bfff640 1 --2- 172.21.15.98:0/394266859 >> [v2:172.21.15.98:6800/3918226860,v1:172.21.15.98:6801/3918226860] conn(0x7f9528073520 0x7f9528075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.901+0000 7f953bfff640 1 --2- 172.21.15.98:0/394266859 >> [v2:172.21.15.98:6800/3918226860,v1:172.21.15.98:6801/3918226860] conn(0x7f9528073520 0x7f9528075940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.900+0000 7ff6bcd79640 1 --2- 172.21.15.98:0/680139394 >> [v2:172.21.15.98:6809/2691532400,v1:172.21.15.98:6812/2691532400] conn(0x7ff69c073490 0x7ff69c0758b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.900+0000 7fa178b9e640 1 -- start start 2023-11-28T17:41:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.901+0000 7fa178b9e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1740ca290 0x7fa1740ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.901+0000 7fa178b9e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa174151560 0x7fa174152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.901+0000 7fa178b9e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa1740cb1b0 0x7fa1740cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.901+0000 7fa178b9e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1740c19d0 con 0x7fa1740ca290 2023-11-28T17:41:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.901+0000 7fa178b9e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa174151340 con 0x7fa174151560 2023-11-28T17:41:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.901+0000 7fa178b9e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa174151e20 con 0x7fa1740cb1b0 2023-11-28T17:41:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.901+0000 7f372ac72640 1 -- start start 2023-11-28T17:41:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.901+0000 7f340ed9d640 1 --2- 172.21.15.98:0/4129095329 >> v2:172.21.15.113:3300/0 conn(0x7f34100ca280 0x7f34100ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.901+0000 7f340dd9b640 1 -- 172.21.15.98:0/4129095329 >> v2:172.21.15.113:3300/0 conn(0x7f34100ca280 msgr2=0x7f34100ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.901+0000 7f340dd9b640 1 --2- 172.21.15.98:0/4129095329 >> v2:172.21.15.113:3300/0 conn(0x7f34100ca280 0x7f34100ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.901+0000 7f340dd9b640 1 -- 172.21.15.98:0/4129095329 >> v1:172.21.15.113:6789/0 conn(0x7f34101540d0 legacy=0x7f341015dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.901+0000 7f340dd9b640 1 -- 172.21.15.98:0/4129095329 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341006a8b0 con 0x7f34100cb070 2023-11-28T17:41:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.902+0000 7fa17259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1740ca290 0x7fa1740ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.902+0000 7fa172d9d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa1740cb1b0 0x7fa1740cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.902+0000 7fa17259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1740ca290 0x7fa1740ca670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49424/0 (socket says 172.21.15.98:49424) 2023-11-28T17:41:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.902+0000 7fa17259c640 1 -- 172.21.15.98:0/3630819129 learned_addr learned my addr 172.21.15.98:0/3630819129 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.902+0000 7f953affd640 1 --2- 172.21.15.98:0/394266859 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f952806b300 0x7f952806d7c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f953c067ad0 tx=0x7f9534009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.902+0000 7fa16bfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa174151560 0x7fa174152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.902+0000 7f372ac72640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f372406ddf0 0x7f37241efab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.902+0000 7f099cec9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f09981540d0 0x7f099815dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.902+0000 7f099cec9640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09980cb070 0x7f099806b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7fa172d9d640 1 -- 172.21.15.98:0/3630819129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa174151560 msgr2=0x7fa174152c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7fa172d9d640 1 --2- 172.21.15.98:0/3630819129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa174151560 0x7fa174152c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7fa172d9d640 1 -- 172.21.15.98:0/3630819129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1740ca290 msgr2=0x7fa1740ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7f372ac72640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3724071940 0x7f37240c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.902+0000 7f099cec9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099806a660 con 0x7f09980ca280 2023-11-28T17:41:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.902+0000 7f099cec9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09980cbdd0 con 0x7f09981540d0 2023-11-28T17:41:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.902+0000 7f099cec9640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09981544b0 con 0x7f09980cb070 2023-11-28T17:41:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7f6e52d66640 1 --2- 172.21.15.98:0/3817627943 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e4c1540d0 0x7f6e4c1f59e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c00c3c0 tx=0x7f6e3c00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7f372ac72640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f372414fdd0 0x7f372414b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7fa172d9d640 1 --2- 172.21.15.98:0/3630819129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1740ca290 0x7fa1740ca670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7fa172d9d640 1 -- 172.21.15.98:0/3630819129 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1741548a0 con 0x7fa1740cb1b0 2023-11-28T17:41:25.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7fa17259c640 1 --2- 172.21.15.98:0/3630819129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1740ca290 0x7fa1740ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:41:25.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7f372ac72640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3724147e20 con 0x7f372414fdd0 2023-11-28T17:41:25.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7f0996d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f09980ca280 0x7f0998159c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37512/0 (socket says 172.21.15.98:37512) 2023-11-28T17:41:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7f6e42ffd640 1 -- 172.21.15.98:0/3817627943 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c005da0 con 0x7f6e4c1540d0 2023-11-28T17:41:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7f3728a0e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f372406ddf0 0x7f37241efab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7f3723fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3724071940 0x7f37240c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7f3723fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3724071940 0x7f37240c4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:42718/0 (socket says 172.21.15.98:42718) 2023-11-28T17:41:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7f3723fff640 1 -- 172.21.15.98:0/1210194007 learned_addr learned my addr 172.21.15.98:0/1210194007 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7f372ac72640 1 -- 172.21.15.98:0/1210194007 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372414a5a0 con 0x7f3724071940 2023-11-28T17:41:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.904+0000 7f372ac72640 1 -- 172.21.15.98:0/1210194007 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37241efff0 con 0x7f372406ddf0 2023-11-28T17:41:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.904+0000 7fa172d9d640 1 --2- 172.21.15.98:0/3630819129 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa1740cb1b0 0x7fa1740cb610 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa1740c7f60 tx=0x7fa16c00a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7f6e547c9640 1 -- 172.21.15.98:0/3817627943 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4c0c48d0 con 0x7f6e4c1540d0 2023-11-28T17:41:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.904+0000 7f340dd9b640 1 --2- 172.21.15.98:0/4129095329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34100cb070 0x7f341006b950 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f341015aca0 tx=0x7f33f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae0e0a5d7f952fd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.904+0000 7ff6adffb640 1 -- 172.21.15.98:0/680139394 <== osd.2 v2:172.21.15.98:6809/2691532400 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff69c075f60 con 0x7ff69c073490 2023-11-28T17:41:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.904+0000 7f372920f640 1 --2- 172.21.15.98:0/1210194007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f372414fdd0 0x7f372414b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.904+0000 7f9523fff640 1 -- 172.21.15.98:0/394266859 <== osd.1 v2:172.21.15.98:6800/3918226860 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9528075ff0 con 0x7f9528073520 2023-11-28T17:41:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.904+0000 7f6e547c9640 1 -- 172.21.15.98:0/3817627943 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4c0c4e90 con 0x7f6e4c1540d0 2023-11-28T17:41:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.904+0000 7f6e42ffd640 1 -- 172.21.15.98:0/3817627943 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3c0070d0 con 0x7f6e4c1540d0 2023-11-28T17:41:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.904+0000 7f6e42ffd640 1 -- 172.21.15.98:0/3817627943 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c0074c0 con 0x7f6e4c1540d0 2023-11-28T17:41:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7fac04ff9640 1 -- 172.21.15.98:0/1180359512 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fac0000d7e0 con 0x7fac080c6be0 2023-11-28T17:41:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.905+0000 7fa16affd640 1 -- 172.21.15.98:0/3630819129 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c0133f0 con 0x7fa1740cb1b0 2023-11-28T17:41:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.905+0000 7fa16affd640 1 -- 172.21.15.98:0/3630819129 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16c013550 con 0x7fa1740cb1b0 2023-11-28T17:41:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.905+0000 7fa16affd640 1 -- 172.21.15.98:0/3630819129 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c013900 con 0x7fa1740cb1b0 2023-11-28T17:41:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.905+0000 7f6e547c9640 1 -- 172.21.15.98:0/3817627943 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6e34000fc0 con 0x7f6e4c1540d0 2023-11-28T17:41:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.903+0000 7f0996d9d640 1 -- 172.21.15.98:0/1938292262 learned_addr learned my addr 172.21.15.98:0/1938292262 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.904+0000 7f099659c640 1 --2- 172.21.15.98:0/1938292262 >> v2:172.21.15.98:3300/0 conn(0x7f09981540d0 0x7f099815dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.904+0000 7f099659c640 1 -- 172.21.15.98:0/1938292262 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09980cb070 msgr2=0x7f099806b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.904+0000 7f099659c640 1 --2- 172.21.15.98:0/1938292262 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09980cb070 0x7f099806b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.904+0000 7f099659c640 1 -- 172.21.15.98:0/1938292262 >> v1:172.21.15.98:6789/0 conn(0x7f09980ca280 legacy=0x7f0998159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.904+0000 7f099659c640 1 -- 172.21.15.98:0/1938292262 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099815e1b0 con 0x7f09981540d0 2023-11-28T17:41:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.905+0000 7f3728a0e640 1 -- 172.21.15.98:0/1210194007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3724071940 msgr2=0x7f37240c4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.905+0000 7f3728a0e640 1 --2- 172.21.15.98:0/1210194007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3724071940 0x7f37240c4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.905+0000 7fac04ff9640 1 --2- 172.21.15.98:0/1180359512 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fabe806b2f0 0x7fabe806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.905+0000 7fac04ff9640 1 -- 172.21.15.98:0/1180359512 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac0008a1d0 con 0x7fac080c6be0 2023-11-28T17:41:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.905+0000 7fac04ff9640 1 --2- 172.21.15.98:0/1180359512 >> [v2:172.21.15.98:6808/2188704385,v1:172.21.15.98:6810/2188704385] conn(0x7fabe8073510 0x7fabe8075930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.905+0000 7fac04ff9640 1 -- 172.21.15.98:0/1180359512 --> [v2:172.21.15.98:6808/2188704385,v1:172.21.15.98:6810/2188704385] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabe8075fe0 con 0x7fabe8073510 2023-11-28T17:41:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.905+0000 7fac04ff9640 1 -- 172.21.15.98:0/1180359512 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac0008a540 con 0x7fac080c6be0 2023-11-28T17:41:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.905+0000 7f3728a0e640 1 -- 172.21.15.98:0/1210194007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f372414fdd0 msgr2=0x7f372414b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.905+0000 7f3728a0e640 1 --2- 172.21.15.98:0/1210194007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f372414fdd0 0x7f372414b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.906+0000 7f3728a0e640 1 -- 172.21.15.98:0/1210194007 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371800b770 con 0x7f372406ddf0 2023-11-28T17:41:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.906+0000 7fac06ffd640 1 --2- 172.21.15.98:0/1180359512 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fabe806b2f0 0x7fabe806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.906+0000 7fabfe7fc640 1 --2- 172.21.15.98:0/1180359512 >> [v2:172.21.15.98:6808/2188704385,v1:172.21.15.98:6810/2188704385] conn(0x7fabe8073510 0x7fabe8075930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.906+0000 7f372920f640 1 --2- 172.21.15.98:0/1210194007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f372414fdd0 0x7f372414b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:41:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.906+0000 7f3723fff640 1 --2- 172.21.15.98:0/1210194007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3724071940 0x7f37240c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:41:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.906+0000 7fa178b9e640 1 -- 172.21.15.98:0/3630819129 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1740c1c20 con 0x7fa1740cb1b0 2023-11-28T17:41:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.906+0000 7f099659c640 1 --2- 172.21.15.98:0/1938292262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09981540d0 0x7f099815dc70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0990009d90 tx=0x7f099002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec4c9e70833d228a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.906+0000 7f099559a640 1 -- 172.21.15.98:0/1938292262 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099002cc50 con 0x7f09981540d0 2023-11-28T17:41:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.907+0000 7f3728a0e640 1 --2- 172.21.15.98:0/1210194007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f372406ddf0 0x7f37241efab0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f37240c7760 tx=0x7f37180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.907+0000 7f099559a640 1 -- 172.21.15.98:0/1938292262 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099002cdb0 con 0x7f09981540d0 2023-11-28T17:41:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.907+0000 7f099559a640 1 -- 172.21.15.98:0/1938292262 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0990034660 con 0x7f09981540d0 2023-11-28T17:41:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.907+0000 7fabfe7fc640 1 --2- 172.21.15.98:0/1180359512 >> [v2:172.21.15.98:6808/2188704385,v1:172.21.15.98:6810/2188704385] conn(0x7fabe8073510 0x7fabe8075930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.907+0000 7f340d59a640 1 -- 172.21.15.98:0/4129095329 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f8031870 con 0x7f34100cb070 2023-11-28T17:41:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.906+0000 7fa178b9e640 1 -- 172.21.15.98:0/3630819129 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1741f49a0 con 0x7fa1740cb1b0 2023-11-28T17:41:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.907+0000 7f340d59a640 1 -- 172.21.15.98:0/4129095329 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f80319d0 con 0x7f34100cb070 2023-11-28T17:41:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.907+0000 7f340d59a640 1 -- 172.21.15.98:0/4129095329 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f8031d60 con 0x7f34100cb070 2023-11-28T17:41:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.907+0000 7fa178b9e640 1 -- 172.21.15.98:0/3630819129 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa1740711a0 con 0x7fa1740cb1b0 2023-11-28T17:41:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.908+0000 7f099cec9640 1 -- 172.21.15.98:0/1938292262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09981540d0 msgr2=0x7f099815dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.908+0000 7f099cec9640 1 --2- 172.21.15.98:0/1938292262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09981540d0 0x7f099815dc70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0990009d90 tx=0x7f099002bea0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.908+0000 7fac06ffd640 1 --2- 172.21.15.98:0/1180359512 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fabe806b2f0 0x7fabe806d7b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fabf0009a70 tx=0x7fabf0009590 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.908+0000 7f3414875640 1 -- 172.21.15.98:0/4129095329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34100cb070 msgr2=0x7f341006b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.908+0000 7f3414875640 1 --2- 172.21.15.98:0/4129095329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34100cb070 0x7f341006b950 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f341015aca0 tx=0x7f33f802b910 comp rx=0 tx=0).stop 2023-11-28T17:41:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.908+0000 7f37217fa640 1 -- 172.21.15.98:0/1210194007 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718036440 con 0x7f372406ddf0 2023-11-28T17:41:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.908+0000 7f3414875640 1 -- 172.21.15.98:0/4129095329 shutdown_connections 2023-11-28T17:41:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.908+0000 7f37217fa640 1 -- 172.21.15.98:0/1210194007 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3718034ba0 con 0x7f372406ddf0 2023-11-28T17:41:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.908+0000 7f37217fa640 1 -- 172.21.15.98:0/1210194007 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718041460 con 0x7f372406ddf0 2023-11-28T17:41:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.908+0000 7f372ac72640 1 -- 172.21.15.98:0/1210194007 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372414ae00 con 0x7f372406ddf0 2023-11-28T17:41:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.909+0000 7f099cec9640 1 -- 172.21.15.98:0/1938292262 shutdown_connections 2023-11-28T17:41:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.909+0000 7f099cec9640 1 --2- 172.21.15.98:0/1938292262 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09980cb070 0x7f099806b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.909+0000 7f099cec9640 1 --2- 172.21.15.98:0/1938292262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09981540d0 0x7f099815dc70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.909+0000 7f099cec9640 1 -- 172.21.15.98:0/1938292262 >> 172.21.15.98:0/1938292262 conn(0x7f09980cf490 msgr2=0x7f09980cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.908+0000 7f3414875640 1 --2- 172.21.15.98:0/4129095329 >> v2:172.21.15.113:3300/0 conn(0x7f34100ca280 0x7f34100ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.908+0000 7f3414875640 1 --2- 172.21.15.98:0/4129095329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34100cb070 0x7f341006b950 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.908+0000 7f3414875640 1 -- 172.21.15.98:0/4129095329 >> 172.21.15.98:0/4129095329 conn(0x7f34100cf490 msgr2=0x7f34100cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.908+0000 7f3414875640 1 -- 172.21.15.98:0/4129095329 shutdown_connections 2023-11-28T17:41:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.909+0000 7f3414875640 1 -- 172.21.15.98:0/4129095329 wait complete. 2023-11-28T17:41:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.910+0000 7f372ac72640 1 -- 172.21.15.98:0/1210194007 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372414d0c0 con 0x7f372406ddf0 2023-11-28T17:41:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.910+0000 7f099cec9640 1 -- 172.21.15.98:0/1938292262 shutdown_connections 2023-11-28T17:41:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.910+0000 7fac04ff9640 1 -- 172.21.15.98:0/1180359512 <== osd.0 v2:172.21.15.98:6808/2188704385 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fabe8075fe0 con 0x7fabe8073510 2023-11-28T17:41:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.910+0000 7f099cec9640 1 -- 172.21.15.98:0/1938292262 wait complete. 2023-11-28T17:41:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.911+0000 7f372ac72640 1 -- 172.21.15.98:0/1210194007 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f37240c2730 con 0x7f372406ddf0 2023-11-28T17:41:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.911+0000 7f3414875640 1 Processor -- start 2023-11-28T17:41:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.910+0000 7f6e42ffd640 1 -- 172.21.15.98:0/3817627943 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6e3c007620 con 0x7f6e4c1540d0 2023-11-28T17:41:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.911+0000 7f6e42ffd640 1 --2- 172.21.15.98:0/3817627943 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6e3006b320 0x7f6e3006d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.911+0000 7f6e42ffd640 1 -- 172.21.15.98:0/3817627943 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e3c088f40 con 0x7f6e4c1540d0 2023-11-28T17:41:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.911+0000 7f6e42ffd640 1 --2- 172.21.15.98:0/3817627943 >> [v2:172.21.15.113:6816/2685000355,v1:172.21.15.113:6817/2685000355] conn(0x7f6e30073540 0x7f6e30075960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.911+0000 7f6e42ffd640 1 -- 172.21.15.98:0/3817627943 --> [v2:172.21.15.113:6816/2685000355,v1:172.21.15.113:6817/2685000355] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e30076010 con 0x7f6e30073540 2023-11-28T17:41:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.911+0000 7f6e42ffd640 1 -- 172.21.15.98:0/3817627943 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e3c08b520 con 0x7f6e4c1540d0 2023-11-28T17:41:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.912+0000 7f3414875640 1 -- start start 2023-11-28T17:41:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.912+0000 7f6e51d64640 1 --2- 172.21.15.98:0/3817627943 >> [v2:172.21.15.113:6816/2685000355,v1:172.21.15.113:6817/2685000355] conn(0x7f6e30073540 0x7f6e30075960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.912+0000 7f6e52565640 1 --2- 172.21.15.98:0/3817627943 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6e3006b320 0x7f6e3006d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.913+0000 7f3414875640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34100ca280 0x7f34101f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.913+0000 7f3414875640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34101540d0 0x7f34101e90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.913+0000 7f3414875640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f341015c4d0 0x7f34100b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.913+0000 7f3414875640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341014fe10 con 0x7f34100ca280 2023-11-28T17:41:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.913+0000 7f3414875640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34101f4fb0 con 0x7f341015c4d0 2023-11-28T17:41:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.913+0000 7f3414875640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34101e8c20 con 0x7f34101540d0 2023-11-28T17:41:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.913+0000 7f6e51d64640 1 --2- 172.21.15.98:0/3817627943 >> [v2:172.21.15.113:6816/2685000355,v1:172.21.15.113:6817/2685000355] conn(0x7f6e30073540 0x7f6e30075960 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.913+0000 7f6e52565640 1 --2- 172.21.15.98:0/3817627943 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6e3006b320 0x7f6e3006d7e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6e480095d0 tx=0x7f6e48006b30 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.914+0000 7fa16affd640 1 -- 172.21.15.98:0/3630819129 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa16c013a60 con 0x7fa1740cb1b0 2023-11-28T17:41:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.914+0000 7f340dd9b640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34101540d0 0x7f34101e90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.914+0000 7f340dd9b640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34101540d0 0x7f34101e90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:55614/0 (socket says 172.21.15.98:55614) 2023-11-28T17:41:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.914+0000 7f340ed9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f341015c4d0 0x7f34100b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.914+0000 7f340dd9b640 1 -- 172.21.15.98:0/3325038154 learned_addr learned my addr 172.21.15.98:0/3325038154 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.914+0000 7f099cec9640 1 Processor -- start 2023-11-28T17:41:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.914+0000 7f340dd9b640 1 -- 172.21.15.98:0/3325038154 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f341015c4d0 msgr2=0x7f34100b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.914+0000 7f340dd9b640 1 --2- 172.21.15.98:0/3325038154 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f341015c4d0 0x7f34100b4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.914+0000 7f340dd9b640 1 -- 172.21.15.98:0/3325038154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34100ca280 msgr2=0x7f34101f4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.915+0000 7f340e59c640 1 --2- 172.21.15.98:0/3325038154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34100ca280 0x7f34101f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.915+0000 7f340dd9b640 1 --2- 172.21.15.98:0/3325038154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34100ca280 0x7f34101f4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.915+0000 7f340dd9b640 1 -- 172.21.15.98:0/3325038154 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f80094d0 con 0x7f34101540d0 2023-11-28T17:41:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.915+0000 7f340e59c640 1 --2- 172.21.15.98:0/3325038154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34100ca280 0x7f34101f4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.915+0000 7fa16affd640 1 --2- 172.21.15.98:0/3630819129 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa15406b3a0 0x7fa15406d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.915+0000 7fa16affd640 1 -- 172.21.15.98:0/3630819129 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa16c08ac80 con 0x7fa1740cb1b0 2023-11-28T17:41:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.915+0000 7fa16affd640 1 --2- 172.21.15.98:0/3630819129 >> [v2:172.21.15.98:6824/1905487008,v1:172.21.15.98:6825/1905487008] conn(0x7fa1540735c0 0x7fa1540759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.915+0000 7fa16affd640 1 -- 172.21.15.98:0/3630819129 --> [v2:172.21.15.98:6824/1905487008,v1:172.21.15.98:6825/1905487008] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa154076090 con 0x7fa1540735c0 2023-11-28T17:41:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.915+0000 7fa16affd640 1 -- 172.21.15.98:0/3630819129 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa16c08b030 con 0x7fa1740cb1b0 2023-11-28T17:41:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.914+0000 7f099cec9640 1 -- start start 2023-11-28T17:41:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.916+0000 7f099cec9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09980ca280 0x7f09980c2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.916+0000 7f099cec9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09980cb070 0x7f099815bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.916+0000 7f099cec9640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09981540d0 0x7f099814f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.916+0000 7f099cec9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09980c2ca0 con 0x7f09980ca280 2023-11-28T17:41:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.916+0000 7f099cec9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09980b3800 con 0x7f09980cb070 2023-11-28T17:41:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.916+0000 7f099cec9640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09980b3ae0 con 0x7f09981540d0 2023-11-28T17:41:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.916+0000 7f099659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09980ca280 0x7f09980c2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.916+0000 7f099659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09980ca280 0x7f09980c2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49456/0 (socket says 172.21.15.98:49456) 2023-11-28T17:41:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.916+0000 7f099659c640 1 -- 172.21.15.98:0/1723505801 learned_addr learned my addr 172.21.15.98:0/1723505801 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.916+0000 7f340dd9b640 1 --2- 172.21.15.98:0/3325038154 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34101540d0 0x7f34101e90b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f33f8000c00 tx=0x7f33f80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.916+0000 7f0996d9d640 1 --2- 172.21.15.98:0/1723505801 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09981540d0 0x7f099814f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.916+0000 7fa16bfff640 1 --2- 172.21.15.98:0/3630819129 >> [v2:172.21.15.98:6824/1905487008,v1:172.21.15.98:6825/1905487008] conn(0x7fa1540735c0 0x7fa1540759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.917+0000 7f099659c640 1 -- 172.21.15.98:0/1723505801 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09981540d0 msgr2=0x7f099814f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.917+0000 7f099659c640 1 --2- 172.21.15.98:0/1723505801 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f09981540d0 0x7f099814f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.917+0000 7f099659c640 1 -- 172.21.15.98:0/1723505801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09980cb070 msgr2=0x7f099815bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:41:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.917+0000 7f099659c640 1 --2- 172.21.15.98:0/1723505801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09980cb070 0x7f099815bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.917+0000 7f099659c640 1 -- 172.21.15.98:0/1723505801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0990009a70 con 0x7f09980ca280 2023-11-28T17:41:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.917+0000 7f3406ffd640 1 -- 172.21.15.98:0/3325038154 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f8004230 con 0x7f34101540d0 2023-11-28T17:41:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.917+0000 7f3406ffd640 1 -- 172.21.15.98:0/3325038154 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f8004390 con 0x7f34101540d0 2023-11-28T17:41:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.917+0000 7f3406ffd640 1 -- 172.21.15.98:0/3325038154 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f803aa70 con 0x7f34101540d0 2023-11-28T17:41:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.917+0000 7f3414875640 1 -- 172.21.15.98:0/3325038154 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34101ea510 con 0x7f34101540d0 2023-11-28T17:41:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.917+0000 7f099659c640 1 --2- 172.21.15.98:0/1723505801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09980ca280 0x7f09980c2760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0990009d90 tx=0x7f0990004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.917+0000 7f3414875640 1 -- 172.21.15.98:0/3325038154 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34101ea230 con 0x7f34101540d0 2023-11-28T17:41:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.917+0000 7f0986ffd640 1 -- 172.21.15.98:0/1723505801 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099003e610 con 0x7f09980ca280 2023-11-28T17:41:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.917+0000 7f0986ffd640 1 -- 172.21.15.98:0/1723505801 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099003e770 con 0x7f09980ca280 2023-11-28T17:41:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.918+0000 7f099cec9640 1 -- 172.21.15.98:0/1723505801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09980b3dc0 con 0x7f09980ca280 2023-11-28T17:41:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.918+0000 7f0986ffd640 1 -- 172.21.15.98:0/1723505801 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099003eb00 con 0x7f09980ca280 2023-11-28T17:41:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.918+0000 7fa16bfff640 1 --2- 172.21.15.98:0/3630819129 >> [v2:172.21.15.98:6824/1905487008,v1:172.21.15.98:6825/1905487008] conn(0x7fa1540735c0 0x7fa1540759e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.916+0000 7f37217fa640 1 -- 172.21.15.98:0/1210194007 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f371804a3b0 con 0x7f372406ddf0 2023-11-28T17:41:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.919+0000 7f6e42ffd640 1 -- 172.21.15.98:0/3817627943 <== osd.5 v2:172.21.15.113:6816/2685000355 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f6e30076010 con 0x7f6e30073540 2023-11-28T17:41:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.919+0000 7f099cec9640 1 -- 172.21.15.98:0/1723505801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09980b4380 con 0x7f09980ca280 2023-11-28T17:41:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.919+0000 7fa17259c640 1 --2- 172.21.15.98:0/3630819129 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa15406b3a0 0x7fa15406d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.920+0000 7f37217fa640 1 --2- 172.21.15.98:0/1210194007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f370806b320 0x7f370806d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.920+0000 7fa17259c640 1 --2- 172.21.15.98:0/3630819129 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa15406b3a0 0x7fa15406d860 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa16402c560 tx=0x7fa164036040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.920+0000 7f099cec9640 1 -- 172.21.15.98:0/1723505801 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f09980c9ae0 con 0x7f09980ca280 2023-11-28T17:41:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.920+0000 7f37217fa640 1 -- 172.21.15.98:0/1210194007 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37180ac090 con 0x7f372406ddf0 2023-11-28T17:41:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.920+0000 7fa16affd640 1 -- 172.21.15.98:0/3630819129 <== osd.3 v2:172.21.15.98:6824/1905487008 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fa154076090 con 0x7fa1540735c0 2023-11-28T17:41:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.921+0000 7f3414875640 1 -- 172.21.15.98:0/3325038154 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f33f4000fc0 con 0x7f34101540d0 2023-11-28T17:41:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.922+0000 7f0986ffd640 1 -- 172.21.15.98:0/1723505801 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f099003b050 con 0x7f09980ca280 2023-11-28T17:41:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.923+0000 7f3406ffd640 1 -- 172.21.15.98:0/3325038154 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f33f8042070 con 0x7f34101540d0 2023-11-28T17:41:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.924+0000 7f3723fff640 1 --2- 172.21.15.98:0/1210194007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f370806b320 0x7f370806d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.924+0000 7f0986ffd640 1 --2- 172.21.15.98:0/1723505801 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f097806b2a0 0x7f097806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.924+0000 7f0986ffd640 1 -- 172.21.15.98:0/1723505801 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09900adcf0 con 0x7f09980ca280 2023-11-28T17:41:25.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.924+0000 7f37217fa640 1 --2- 172.21.15.98:0/1210194007 >> [v2:172.21.15.113:6800/2974677215,v1:172.21.15.113:6801/2974677215] conn(0x7f3708073540 0x7f3708075960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.924+0000 7f37217fa640 1 -- 172.21.15.98:0/1210194007 --> [v2:172.21.15.113:6800/2974677215,v1:172.21.15.113:6801/2974677215] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3708076010 con 0x7f3708073540 2023-11-28T17:41:25.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.924+0000 7f37217fa640 1 -- 172.21.15.98:0/1210194007 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37180ae720 con 0x7f372406ddf0 2023-11-28T17:41:25.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.925+0000 7f0986ffd640 1 --2- 172.21.15.98:0/1723505801 >> [v2:172.21.15.113:6808/2764300831,v1:172.21.15.113:6809/2764300831] conn(0x7f0978073440 0x7f0978075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.925+0000 7f0986ffd640 1 -- 172.21.15.98:0/1723505801 --> [v2:172.21.15.113:6808/2764300831,v1:172.21.15.113:6809/2764300831] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0978075f10 con 0x7f0978073440 2023-11-28T17:41:25.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.925+0000 7f0986ffd640 1 -- 172.21.15.98:0/1723505801 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09900b53a0 con 0x7f09980ca280 2023-11-28T17:41:25.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.925+0000 7f3406ffd640 1 --2- 172.21.15.98:0/3325038154 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f33f006b2a0 0x7f33f006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.925+0000 7f3406ffd640 1 -- 172.21.15.98:0/3325038154 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33f802d040 con 0x7f34101540d0 2023-11-28T17:41:25.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.925+0000 7f3723fff640 1 --2- 172.21.15.98:0/1210194007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f370806b320 0x7f370806d7e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f372414dc40 tx=0x7f3710008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.925+0000 7f372920f640 1 --2- 172.21.15.98:0/1210194007 >> [v2:172.21.15.113:6800/2974677215,v1:172.21.15.113:6801/2974677215] conn(0x7f3708073540 0x7f3708075960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.925+0000 7f0996d9d640 1 --2- 172.21.15.98:0/1723505801 >> [v2:172.21.15.113:6808/2764300831,v1:172.21.15.113:6809/2764300831] conn(0x7f0978073440 0x7f0978075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.925+0000 7f3406ffd640 1 --2- 172.21.15.98:0/3325038154 >> [v2:172.21.15.113:6819/1656248400,v1:172.21.15.113:6820/1656248400] conn(0x7f33f0073440 0x7f33f0075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:25.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.926+0000 7f3406ffd640 1 -- 172.21.15.98:0/3325038154 --> [v2:172.21.15.113:6819/1656248400,v1:172.21.15.113:6820/1656248400] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33f0075f10 con 0x7f33f0073440 2023-11-28T17:41:25.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.926+0000 7f3406ffd640 1 -- 172.21.15.98:0/3325038154 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33f80b02a0 con 0x7f34101540d0 2023-11-28T17:41:25.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.926+0000 7f340e59c640 1 --2- 172.21.15.98:0/3325038154 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f33f006b2a0 0x7f33f006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.926+0000 7f372920f640 1 --2- 172.21.15.98:0/1210194007 >> [v2:172.21.15.113:6800/2974677215,v1:172.21.15.113:6801/2974677215] conn(0x7f3708073540 0x7f3708075960 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.926+0000 7f0995d9b640 1 --2- 172.21.15.98:0/1723505801 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f097806b2a0 0x7f097806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.926+0000 7f0996d9d640 1 --2- 172.21.15.98:0/1723505801 >> [v2:172.21.15.113:6808/2764300831,v1:172.21.15.113:6809/2764300831] conn(0x7f0978073440 0x7f0978075860 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.927+0000 7f0995d9b640 1 --2- 172.21.15.98:0/1723505801 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f097806b2a0 0x7f097806d760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0998157450 tx=0x7f098c006c20 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.927+0000 7f340ed9d640 1 --2- 172.21.15.98:0/3325038154 >> [v2:172.21.15.113:6819/1656248400,v1:172.21.15.113:6820/1656248400] conn(0x7f33f0073440 0x7f33f0075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:25.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.927+0000 7f340e59c640 1 --2- 172.21.15.98:0/3325038154 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f33f006b2a0 0x7f33f006d760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3400007eb0 tx=0x7f34000073c0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.927+0000 7f340ed9d640 1 --2- 172.21.15.98:0/3325038154 >> [v2:172.21.15.113:6819/1656248400,v1:172.21.15.113:6820/1656248400] conn(0x7f33f0073440 0x7f33f0075860 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:25.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.929+0000 7f37217fa640 1 -- 172.21.15.98:0/1210194007 <== osd.4 v2:172.21.15.113:6800/2974677215 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f3708076010 con 0x7f3708073540 2023-11-28T17:41:25.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.929+0000 7f0986ffd640 1 -- 172.21.15.98:0/1723505801 <== osd.6 v2:172.21.15.113:6808/2764300831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f0978075f10 con 0x7f0978073440 2023-11-28T17:41:25.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.929+0000 7ff6bf7de640 1 -- 172.21.15.98:0/680139394 --> [v2:172.21.15.98:6809/2691532400,v1:172.21.15.98:6812/2691532400] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff6b0006990 con 0x7ff69c073490 2023-11-28T17:41:25.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.930+0000 7f3406ffd640 1 -- 172.21.15.98:0/3325038154 <== osd.7 v2:172.21.15.113:6819/1656248400 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f33f0075f10 con 0x7f33f0073440 2023-11-28T17:41:25.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.932+0000 7ff6adffb640 1 -- 172.21.15.98:0/680139394 <== osd.2 v2:172.21.15.98:6809/2691532400 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff6b4039450 con 0x7ff69c073490 2023-11-28T17:41:25.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.933+0000 7ff6bf7de640 1 -- 172.21.15.98:0/680139394 >> [v2:172.21.15.98:6809/2691532400,v1:172.21.15.98:6812/2691532400] conn(0x7ff69c073490 msgr2=0x7ff69c0758b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.933+0000 7ff6bf7de640 1 --2- 172.21.15.98:0/680139394 >> [v2:172.21.15.98:6809/2691532400,v1:172.21.15.98:6812/2691532400] conn(0x7ff69c073490 0x7ff69c0758b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.933+0000 7ff6bf7de640 1 -- 172.21.15.98:0/680139394 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff69c06b2f0 msgr2=0x7ff69c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.933+0000 7ff6bf7de640 1 --2- 172.21.15.98:0/680139394 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff69c06b2f0 0x7ff69c06d7b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff6b80699e0 tx=0x7ff6b806e040 comp rx=0 tx=0).stop 2023-11-28T17:41:25.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.934+0000 7ff6bf7de640 1 -- 172.21.15.98:0/680139394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6b00550f0 msgr2=0x7ff6b0048ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.934+0000 7ff6bf7de640 1 --2- 172.21.15.98:0/680139394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6b00550f0 0x7ff6b0048ff0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff6a800c430 tx=0x7ff6a800c8f0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.936+0000 7ff6bdd7b640 1 -- 172.21.15.98:0/680139394 reap_dead start 2023-11-28T17:41:25.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.936+0000 7ff6bf7de640 1 -- 172.21.15.98:0/680139394 shutdown_connections 2023-11-28T17:41:25.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.936+0000 7ff6bf7de640 1 -- 172.21.15.98:0/680139394 >> 172.21.15.98:0/680139394 conn(0x7ff6b00545d0 msgr2=0x7ff6b0007190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.936+0000 7ff6bf7de640 1 -- 172.21.15.98:0/680139394 shutdown_connections 2023-11-28T17:41:25.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.936+0000 7ff6bf7de640 1 -- 172.21.15.98:0/680139394 wait complete. 2023-11-28T17:41:25.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.943+0000 7f9540ad1640 1 -- 172.21.15.98:0/394266859 --> [v2:172.21.15.98:6800/3918226860,v1:172.21.15.98:6801/3918226860] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f951c002cb0 con 0x7f9528073520 2023-11-28T17:41:25.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.944+0000 7f9523fff640 1 -- 172.21.15.98:0/394266859 <== osd.1 v2:172.21.15.98:6800/3918226860 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f951c002cb0 con 0x7f9528073520 2023-11-28T17:41:25.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.945+0000 7f9540ad1640 1 -- 172.21.15.98:0/394266859 >> [v2:172.21.15.98:6800/3918226860,v1:172.21.15.98:6801/3918226860] conn(0x7f9528073520 msgr2=0x7f9528075940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.945+0000 7f9540ad1640 1 --2- 172.21.15.98:0/394266859 >> [v2:172.21.15.98:6800/3918226860,v1:172.21.15.98:6801/3918226860] conn(0x7f9528073520 0x7f9528075940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.945+0000 7f9540ad1640 1 -- 172.21.15.98:0/394266859 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f952806b300 msgr2=0x7f952806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.945+0000 7f9540ad1640 1 --2- 172.21.15.98:0/394266859 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f952806b300 0x7f952806d7c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f953c067ad0 tx=0x7f9534009040 comp rx=0 tx=0).stop 2023-11-28T17:41:25.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.945+0000 7f9540ad1640 1 -- 172.21.15.98:0/394266859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c0c69b0 msgr2=0x7f953c1f7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.945+0000 7f9540ad1640 1 --2- 172.21.15.98:0/394266859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f953c0c69b0 0x7f953c1f7560 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f953000e9b0 tx=0x7f953000ee70 comp rx=0 tx=0).stop 2023-11-28T17:41:25.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.945+0000 7f953bfff640 1 -- 172.21.15.98:0/394266859 reap_dead start 2023-11-28T17:41:25.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.946+0000 7fac07fff640 1 -- 172.21.15.98:0/1180359512 --> [v2:172.21.15.98:6808/2188704385,v1:172.21.15.98:6810/2188704385] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fac080b4c50 con 0x7fabe8073510 2023-11-28T17:41:25.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.947+0000 7fac04ff9640 1 -- 172.21.15.98:0/1180359512 <== osd.0 v2:172.21.15.98:6808/2188704385 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fac080b4c50 con 0x7fabe8073510 2023-11-28T17:41:25.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.947+0000 7f9540ad1640 1 -- 172.21.15.98:0/394266859 shutdown_connections 2023-11-28T17:41:25.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.947+0000 7f9540ad1640 1 -- 172.21.15.98:0/394266859 >> 172.21.15.98:0/394266859 conn(0x7f953c0cf300 msgr2=0x7f953c0cf6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.947+0000 7f9540ad1640 1 -- 172.21.15.98:0/394266859 shutdown_connections 2023-11-28T17:41:25.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.948+0000 7fac07fff640 1 -- 172.21.15.98:0/1180359512 >> [v2:172.21.15.98:6808/2188704385,v1:172.21.15.98:6810/2188704385] conn(0x7fabe8073510 msgr2=0x7fabe8075930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.948+0000 7fac07fff640 1 --2- 172.21.15.98:0/1180359512 >> [v2:172.21.15.98:6808/2188704385,v1:172.21.15.98:6810/2188704385] conn(0x7fabe8073510 0x7fabe8075930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.948+0000 7f9540ad1640 1 -- 172.21.15.98:0/394266859 wait complete. 2023-11-28T17:41:25.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.948+0000 7fac07fff640 1 -- 172.21.15.98:0/1180359512 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fabe806b2f0 msgr2=0x7fabe806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.948+0000 7fac07fff640 1 --2- 172.21.15.98:0/1180359512 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fabe806b2f0 0x7fabe806d7b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fabf0009a70 tx=0x7fabf0009590 comp rx=0 tx=0).stop 2023-11-28T17:41:25.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.948+0000 7fac07fff640 1 -- 172.21.15.98:0/1180359512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac080c6be0 msgr2=0x7fac081f71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.948+0000 7fac07fff640 1 --2- 172.21.15.98:0/1180359512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac080c6be0 0x7fac081f71d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fac080c50c0 tx=0x7fac0000fc00 comp rx=0 tx=0).stop 2023-11-28T17:41:25.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.949+0000 7fac077fe640 1 -- 172.21.15.98:0/1180359512 reap_dead start 2023-11-28T17:41:25.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.949+0000 7fac07fff640 1 -- 172.21.15.98:0/1180359512 shutdown_connections 2023-11-28T17:41:25.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.949+0000 7fac07fff640 1 -- 172.21.15.98:0/1180359512 >> 172.21.15.98:0/1180359512 conn(0x7fac0806ade0 msgr2=0x7fac08073880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.949+0000 7fa178b9e640 1 -- 172.21.15.98:0/3630819129 --> [v2:172.21.15.98:6824/1905487008,v1:172.21.15.98:6825/1905487008] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa1741f4520 con 0x7fa1540735c0 2023-11-28T17:41:25.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.949+0000 7fac07fff640 1 -- 172.21.15.98:0/1180359512 shutdown_connections 2023-11-28T17:41:25.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.950+0000 7fac07fff640 1 -- 172.21.15.98:0/1180359512 wait complete. 2023-11-28T17:41:25.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.950+0000 7fa16affd640 1 -- 172.21.15.98:0/3630819129 <== osd.3 v2:172.21.15.98:6824/1905487008 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa1741f4520 con 0x7fa1540735c0 2023-11-28T17:41:25.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.951+0000 7fa178b9e640 1 -- 172.21.15.98:0/3630819129 >> [v2:172.21.15.98:6824/1905487008,v1:172.21.15.98:6825/1905487008] conn(0x7fa1540735c0 msgr2=0x7fa1540759e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.951+0000 7fa178b9e640 1 --2- 172.21.15.98:0/3630819129 >> [v2:172.21.15.98:6824/1905487008,v1:172.21.15.98:6825/1905487008] conn(0x7fa1540735c0 0x7fa1540759e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.951+0000 7fa178b9e640 1 -- 172.21.15.98:0/3630819129 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa15406b3a0 msgr2=0x7fa15406d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.951+0000 7fa178b9e640 1 --2- 172.21.15.98:0/3630819129 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa15406b3a0 0x7fa15406d860 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa16402c560 tx=0x7fa164036040 comp rx=0 tx=0).stop 2023-11-28T17:41:25.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.951+0000 7fa178b9e640 1 -- 172.21.15.98:0/3630819129 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa1740cb1b0 msgr2=0x7fa1740cb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.951+0000 7fa178b9e640 1 --2- 172.21.15.98:0/3630819129 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa1740cb1b0 0x7fa1740cb610 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa1740c7f60 tx=0x7fa16c00a760 comp rx=0 tx=0).stop 2023-11-28T17:41:25.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.953+0000 7fa172d9d640 1 -- 172.21.15.98:0/3630819129 reap_dead start 2023-11-28T17:41:25.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.953+0000 7fa178b9e640 1 -- 172.21.15.98:0/3630819129 shutdown_connections 2023-11-28T17:41:25.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.953+0000 7fa178b9e640 1 -- 172.21.15.98:0/3630819129 >> 172.21.15.98:0/3630819129 conn(0x7fa17406c0f0 msgr2=0x7fa17415d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.953+0000 7fa178b9e640 1 -- 172.21.15.98:0/3630819129 shutdown_connections 2023-11-28T17:41:25.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.953+0000 7fa178b9e640 1 -- 172.21.15.98:0/3630819129 wait complete. 2023-11-28T17:41:25.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.953+0000 7f6e547c9640 1 -- 172.21.15.98:0/3817627943 --> [v2:172.21.15.113:6816/2685000355,v1:172.21.15.113:6817/2685000355] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6e34002cb0 con 0x7f6e30073540 2023-11-28T17:41:25.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.954+0000 7f6e42ffd640 1 -- 172.21.15.98:0/3817627943 <== osd.5 v2:172.21.15.113:6816/2685000355 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6e34002cb0 con 0x7f6e30073540 2023-11-28T17:41:25.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.955+0000 7f6e547c9640 1 -- 172.21.15.98:0/3817627943 >> [v2:172.21.15.113:6816/2685000355,v1:172.21.15.113:6817/2685000355] conn(0x7f6e30073540 msgr2=0x7f6e30075960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.955+0000 7f6e547c9640 1 --2- 172.21.15.98:0/3817627943 >> [v2:172.21.15.113:6816/2685000355,v1:172.21.15.113:6817/2685000355] conn(0x7f6e30073540 0x7f6e30075960 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.955+0000 7f6e547c9640 1 -- 172.21.15.98:0/3817627943 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6e3006b320 msgr2=0x7f6e3006d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.955+0000 7f6e547c9640 1 --2- 172.21.15.98:0/3817627943 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6e3006b320 0x7f6e3006d7e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6e480095d0 tx=0x7f6e48006b30 comp rx=0 tx=0).stop 2023-11-28T17:41:25.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.955+0000 7f6e547c9640 1 -- 172.21.15.98:0/3817627943 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e4c1540d0 msgr2=0x7f6e4c1f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.955+0000 7f6e547c9640 1 --2- 172.21.15.98:0/3817627943 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6e4c1540d0 0x7f6e4c1f59e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c00c3c0 tx=0x7f6e3c00c880 comp rx=0 tx=0).stop 2023-11-28T17:41:25.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.956+0000 7f6e52d66640 1 -- 172.21.15.98:0/3817627943 reap_dead start 2023-11-28T17:41:25.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.956+0000 7f6e547c9640 1 -- 172.21.15.98:0/3817627943 shutdown_connections 2023-11-28T17:41:25.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.956+0000 7f6e547c9640 1 -- 172.21.15.98:0/3817627943 >> 172.21.15.98:0/3817627943 conn(0x7f6e4c0cf490 msgr2=0x7f6e4c0cd270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.956+0000 7f6e547c9640 1 -- 172.21.15.98:0/3817627943 shutdown_connections 2023-11-28T17:41:25.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.956+0000 7f6e547c9640 1 -- 172.21.15.98:0/3817627943 wait complete. 2023-11-28T17:41:25.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.958+0000 7f3414875640 1 -- 172.21.15.98:0/3325038154 --> [v2:172.21.15.113:6819/1656248400,v1:172.21.15.113:6820/1656248400] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f33f4002cb0 con 0x7f33f0073440 2023-11-28T17:41:25.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.959+0000 7f099cec9640 1 -- 172.21.15.98:0/1723505801 --> [v2:172.21.15.113:6808/2764300831,v1:172.21.15.113:6809/2764300831] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f099806ae70 con 0x7f0978073440 2023-11-28T17:41:25.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.959+0000 7f3406ffd640 1 -- 172.21.15.98:0/3325038154 <== osd.7 v2:172.21.15.113:6819/1656248400 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f33f4002cb0 con 0x7f33f0073440 2023-11-28T17:41:25.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.960+0000 7f3414875640 1 -- 172.21.15.98:0/3325038154 >> [v2:172.21.15.113:6819/1656248400,v1:172.21.15.113:6820/1656248400] conn(0x7f33f0073440 msgr2=0x7f33f0075860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.960+0000 7f3414875640 1 --2- 172.21.15.98:0/3325038154 >> [v2:172.21.15.113:6819/1656248400,v1:172.21.15.113:6820/1656248400] conn(0x7f33f0073440 0x7f33f0075860 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.961+0000 7f3414875640 1 -- 172.21.15.98:0/3325038154 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f33f006b2a0 msgr2=0x7f33f006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.961+0000 7f3414875640 1 --2- 172.21.15.98:0/3325038154 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f33f006b2a0 0x7f33f006d760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3400007eb0 tx=0x7f34000073c0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.961+0000 7f3414875640 1 -- 172.21.15.98:0/3325038154 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34101540d0 msgr2=0x7f34101e90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.961+0000 7f3414875640 1 --2- 172.21.15.98:0/3325038154 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34101540d0 0x7f34101e90b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f33f8000c00 tx=0x7f33f80046c0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.961+0000 7f340ed9d640 1 -- 172.21.15.98:0/3325038154 reap_dead start 2023-11-28T17:41:25.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.961+0000 7f3414875640 1 -- 172.21.15.98:0/3325038154 shutdown_connections 2023-11-28T17:41:25.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.961+0000 7f3414875640 1 -- 172.21.15.98:0/3325038154 >> 172.21.15.98:0/3325038154 conn(0x7f34100cf490 msgr2=0x7f34100c7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.961+0000 7f3414875640 1 -- 172.21.15.98:0/3325038154 shutdown_connections 2023-11-28T17:41:25.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.961+0000 7f3414875640 1 -- 172.21.15.98:0/3325038154 wait complete. 2023-11-28T17:41:25.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.961+0000 7f0986ffd640 1 -- 172.21.15.98:0/1723505801 <== osd.6 v2:172.21.15.113:6808/2764300831 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0988001340 con 0x7f0978073440 2023-11-28T17:41:25.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.962+0000 7f372ac72640 1 -- 172.21.15.98:0/1210194007 --> [v2:172.21.15.113:6800/2974677215,v1:172.21.15.113:6801/2974677215] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f37240744b0 con 0x7f3708073540 2023-11-28T17:41:25.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.963+0000 7f099cec9640 1 -- 172.21.15.98:0/1723505801 >> [v2:172.21.15.113:6808/2764300831,v1:172.21.15.113:6809/2764300831] conn(0x7f0978073440 msgr2=0x7f0978075860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.963+0000 7f099cec9640 1 --2- 172.21.15.98:0/1723505801 >> [v2:172.21.15.113:6808/2764300831,v1:172.21.15.113:6809/2764300831] conn(0x7f0978073440 0x7f0978075860 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.995 INFO:teuthology.orchestra.run.smithi098.stdout:55834574853 2023-11-28T17:41:25.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-28T17:41:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.963+0000 7f099cec9640 1 -- 172.21.15.98:0/1723505801 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f097806b2a0 msgr2=0x7f097806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.963+0000 7f099cec9640 1 --2- 172.21.15.98:0/1723505801 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f097806b2a0 0x7f097806d760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0998157450 tx=0x7f098c006c20 comp rx=0 tx=0).stop 2023-11-28T17:41:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.963+0000 7f099cec9640 1 -- 172.21.15.98:0/1723505801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09980ca280 msgr2=0x7f09980c2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.963+0000 7f099cec9640 1 --2- 172.21.15.98:0/1723505801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09980ca280 0x7f09980c2760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0990009d90 tx=0x7f0990004a40 comp rx=0 tx=0).stop 2023-11-28T17:41:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.963+0000 7f0996d9d640 1 -- 172.21.15.98:0/1723505801 reap_dead start 2023-11-28T17:41:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.964+0000 7f099cec9640 1 -- 172.21.15.98:0/1723505801 shutdown_connections 2023-11-28T17:41:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.964+0000 7f099cec9640 1 -- 172.21.15.98:0/1723505801 >> 172.21.15.98:0/1723505801 conn(0x7f09980cf490 msgr2=0x7f0998150500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.964+0000 7f37217fa640 1 -- 172.21.15.98:0/1210194007 <== osd.4 v2:172.21.15.113:6800/2974677215 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f37240744b0 con 0x7f3708073540 2023-11-28T17:41:25.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.964+0000 7f099cec9640 1 -- 172.21.15.98:0/1723505801 shutdown_connections 2023-11-28T17:41:25.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.964+0000 7f099cec9640 1 -- 172.21.15.98:0/1723505801 wait complete. 2023-11-28T17:41:25.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.965+0000 7f372ac72640 1 -- 172.21.15.98:0/1210194007 >> [v2:172.21.15.113:6800/2974677215,v1:172.21.15.113:6801/2974677215] conn(0x7f3708073540 msgr2=0x7f3708075960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.965+0000 7f372ac72640 1 --2- 172.21.15.98:0/1210194007 >> [v2:172.21.15.113:6800/2974677215,v1:172.21.15.113:6801/2974677215] conn(0x7f3708073540 0x7f3708075960 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.965+0000 7f372ac72640 1 -- 172.21.15.98:0/1210194007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f370806b320 msgr2=0x7f370806d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.965+0000 7f372ac72640 1 --2- 172.21.15.98:0/1210194007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f370806b320 0x7f370806d7e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f372414dc40 tx=0x7f3710008040 comp rx=0 tx=0).stop 2023-11-28T17:41:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.965+0000 7f372ac72640 1 -- 172.21.15.98:0/1210194007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f372406ddf0 msgr2=0x7f37241efab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.965+0000 7f372ac72640 1 --2- 172.21.15.98:0/1210194007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f372406ddf0 0x7f37241efab0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f37240c7760 tx=0x7f37180046c0 comp rx=0 tx=0).stop 2023-11-28T17:41:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.965+0000 7f372920f640 1 -- 172.21.15.98:0/1210194007 reap_dead start 2023-11-28T17:41:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.965+0000 7f372ac72640 1 -- 172.21.15.98:0/1210194007 shutdown_connections 2023-11-28T17:41:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.965+0000 7f372ac72640 1 -- 172.21.15.98:0/1210194007 >> 172.21.15.98:0/1210194007 conn(0x7f372406c0f0 msgr2=0x7f372414df90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.966+0000 7f372ac72640 1 -- 172.21.15.98:0/1210194007 shutdown_connections 2023-11-28T17:41:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:25.966+0000 7f372ac72640 1 -- 172.21.15.98:0/1210194007 wait complete. 2023-11-28T17:41:26.000 INFO:teuthology.orchestra.run.smithi098.stdout:64424509446 2023-11-28T17:41:26.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-28T17:41:26.001 INFO:teuthology.orchestra.run.smithi098.stdout:64424509447 2023-11-28T17:41:26.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-28T17:41:26.002 INFO:teuthology.orchestra.run.smithi098.stdout:64424509445 2023-11-28T17:41:26.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-28T17:41:26.003 INFO:teuthology.orchestra.run.smithi098.stdout:64424509446 2023-11-28T17:41:26.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-28T17:41:26.004 INFO:teuthology.orchestra.run.smithi098.stdout:64424509446 2023-11-28T17:41:26.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-28T17:41:26.005 INFO:teuthology.orchestra.run.smithi098.stdout:64424509447 2023-11-28T17:41:26.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-28T17:41:26.006 INFO:teuthology.orchestra.run.smithi098.stdout:64424509445 2023-11-28T17:41:26.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-28T17:41:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.328+0000 7fcdfe38a640 1 Processor -- start 2023-11-28T17:41:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.329+0000 7fcdfe38a640 1 -- start start 2023-11-28T17:41:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.332+0000 7fcdfe38a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcdf003a010 0x7fcdf0007190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.333+0000 7fcdfe38a640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcdf00550f0 0x7fcdf00076d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.333+0000 7fcdfe38a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf0006180 con 0x7fcdf004f590 2023-11-28T17:41:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.333+0000 7fcdfe38a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf003a3f0 con 0x7fcdf003a010 2023-11-28T17:41:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.333+0000 7fcdfe38a640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf00e3220 con 0x7fcdf00550f0 2023-11-28T17:41:26.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.335+0000 7fcdfc927640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcdf004f590 0x7fcdf0050140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37516/0 (socket says 172.21.15.98:37516) 2023-11-28T17:41:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.335+0000 7fcdfc927640 1 -- 172.21.15.98:0/1785359527 learned_addr learned my addr 172.21.15.98:0/1785359527 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.335+0000 7fcdf77fe640 1 --2- 172.21.15.98:0/1785359527 >> v2:172.21.15.113:3300/0 conn(0x7fcdf00550f0 0x7fcdf00076d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.335+0000 7fcdf7fff640 1 --2- 172.21.15.98:0/1785359527 >> v2:172.21.15.98:3300/0 conn(0x7fcdf003a010 0x7fcdf0007190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.337+0000 7fcdf7fff640 1 -- 172.21.15.98:0/1785359527 >> v2:172.21.15.113:3300/0 conn(0x7fcdf00550f0 msgr2=0x7fcdf00076d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.337+0000 7fcdf7fff640 1 --2- 172.21.15.98:0/1785359527 >> v2:172.21.15.113:3300/0 conn(0x7fcdf00550f0 0x7fcdf00076d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.337+0000 7fcdf7fff640 1 -- 172.21.15.98:0/1785359527 >> v1:172.21.15.98:6789/0 conn(0x7fcdf004f590 legacy=0x7fcdf0050140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.338+0000 7fcdf7fff640 1 -- 172.21.15.98:0/1785359527 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf00063d0 con 0x7fcdf003a010 2023-11-28T17:41:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.340+0000 7f918ffff640 1 Processor -- start 2023-11-28T17:41:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.340+0000 7f918ffff640 1 -- start start 2023-11-28T17:41:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.341+0000 7fcdf7fff640 1 --2- 172.21.15.98:0/1785359527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf003a010 0x7fcdf0007190 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcdf004eb40 tx=0x7fcde402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b43dc4c6358f3af4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.342+0000 7fcdf6ffd640 1 -- 172.21.15.98:0/1785359527 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde4030620 con 0x7fcdf003a010 2023-11-28T17:41:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.342+0000 7fcdf6ffd640 1 -- 172.21.15.98:0/1785359527 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde4030780 con 0x7fcdf003a010 2023-11-28T17:41:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.342+0000 7fcdf6ffd640 1 -- 172.21.15.98:0/1785359527 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde4030b10 con 0x7fcdf003a010 2023-11-28T17:41:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.342+0000 7f918ffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91880e24e0 0x7f9188046840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.342+0000 7f918ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f91880e2de0 0x7f9188005e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.342+0000 7f918ffff640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9188045830 con 0x7f91880e28c0 2023-11-28T17:41:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.342+0000 7f918ffff640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f918804b3d0 con 0x7f91880e2de0 2023-11-28T17:41:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.342+0000 7f918ffff640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9188045a80 con 0x7f91880e24e0 2023-11-28T17:41:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.343+0000 7fcdfe38a640 1 -- 172.21.15.98:0/1785359527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf003a010 msgr2=0x7fcdf0007190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.343+0000 7fcdfe38a640 1 --2- 172.21.15.98:0/1785359527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf003a010 0x7fcdf0007190 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcdf004eb40 tx=0x7fcde402b9f0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.344+0000 7fcdfe38a640 1 -- 172.21.15.98:0/1785359527 shutdown_connections 2023-11-28T17:41:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.344+0000 7fcdfe38a640 1 --2- 172.21.15.98:0/1785359527 >> v2:172.21.15.113:3300/0 conn(0x7fcdf00550f0 0x7fcdf00076d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.344+0000 7fcdfe38a640 1 --2- 172.21.15.98:0/1785359527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf003a010 0x7fcdf0007190 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.344+0000 7fcdfe38a640 1 -- 172.21.15.98:0/1785359527 >> 172.21.15.98:0/1785359527 conn(0x7fcdf00545d0 msgr2=0x7fcdf00549e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.344+0000 7f918e7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f91880e2de0 0x7f9188005e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.344+0000 7fcdfe38a640 1 -- 172.21.15.98:0/1785359527 shutdown_connections 2023-11-28T17:41:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.344+0000 7f918effd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91880e24e0 0x7f9188046840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.344+0000 7f918effd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91880e24e0 0x7f9188046840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:55632/0 (socket says 172.21.15.98:55632) 2023-11-28T17:41:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.344+0000 7f918effd640 1 -- 172.21.15.98:0/1403087185 learned_addr learned my addr 172.21.15.98:0/1403087185 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.346+0000 7f918e7fc640 1 -- 172.21.15.98:0/1403087185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91880e24e0 msgr2=0x7f9188046840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.346+0000 7f918e7fc640 1 --2- 172.21.15.98:0/1403087185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91880e24e0 0x7f9188046840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.346+0000 7f918dffb640 1 -- 172.21.15.98:0/1403087185 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607792562 0 0) 0x7f9188045830 con 0x7f91880e28c0 2023-11-28T17:41:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.344+0000 7fcdfe38a640 1 -- 172.21.15.98:0/1785359527 wait complete. 2023-11-28T17:41:26.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.347+0000 7fcdfe38a640 1 Processor -- start 2023-11-28T17:41:26.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.347+0000 7fcdfe38a640 1 -- start start 2023-11-28T17:41:26.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.346+0000 7f918e7fc640 1 -- 172.21.15.98:0/1403087185 >> v1:172.21.15.113:6789/0 conn(0x7f91880e28c0 legacy=0x7f918804c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.346+0000 7f918e7fc640 1 -- 172.21.15.98:0/1403087185 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9188045d60 con 0x7f91880e2de0 2023-11-28T17:41:26.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.347+0000 7f918effd640 1 --2- 172.21.15.98:0/1403087185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91880e24e0 0x7f9188046840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:41:26.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.348+0000 7f918e7fc640 1 --2- 172.21.15.98:0/1403087185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880e2de0 0x7f9188005e60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9184009a30 tx=0x7f918402bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a044d4a18cc87e9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.348+0000 7f918dffb640 1 -- 172.21.15.98:0/1403087185 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918402d700 con 0x7f91880e2de0 2023-11-28T17:41:26.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.349+0000 7f918dffb640 1 -- 172.21.15.98:0/1403087185 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918402d860 con 0x7f91880e2de0 2023-11-28T17:41:26.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.349+0000 7f918dffb640 1 -- 172.21.15.98:0/1403087185 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918402d700 con 0x7f91880e2de0 2023-11-28T17:41:26.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.350+0000 7fcdfe38a640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdf003a010 0x7fcdf0048070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.350+0000 7f918ffff640 1 -- 172.21.15.98:0/1403087185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880e2de0 msgr2=0x7f9188005e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.350+0000 7f918ffff640 1 --2- 172.21.15.98:0/1403087185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880e2de0 0x7f9188005e60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9184009a30 tx=0x7f918402bb50 comp rx=0 tx=0).stop 2023-11-28T17:41:26.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.350+0000 7fcdfe38a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf004f590 0x7fcdf0184cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.351+0000 7f918ffff640 1 -- 172.21.15.98:0/1403087185 shutdown_connections 2023-11-28T17:41:26.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.351+0000 7f918ffff640 1 --2- 172.21.15.98:0/1403087185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880e2de0 0x7f9188005e60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.351+0000 7f918ffff640 1 --2- 172.21.15.98:0/1403087185 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91880e24e0 0x7f9188046840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.351+0000 7f918ffff640 1 -- 172.21.15.98:0/1403087185 >> 172.21.15.98:0/1403087185 conn(0x7f9188050150 msgr2=0x7f9188050560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.351+0000 7f918ffff640 1 -- 172.21.15.98:0/1403087185 shutdown_connections 2023-11-28T17:41:26.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.351+0000 7f918ffff640 1 -- 172.21.15.98:0/1403087185 wait complete. 2023-11-28T17:41:26.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.351+0000 7fcdfe38a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdf00550f0 0x7fcdf0047060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.351+0000 7fcdf7fff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdf003a010 0x7fcdf0048070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.352+0000 7fcdfe38a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf00e2680 con 0x7fcdf004f590 2023-11-28T17:41:26.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.352+0000 7fcdfe38a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf00497f0 con 0x7fcdf00550f0 2023-11-28T17:41:26.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.352+0000 7fcdfe38a640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf004a3d0 con 0x7fcdf003a010 2023-11-28T17:41:26.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.352+0000 7fcdfc927640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdf00550f0 0x7fcdf0047060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.352+0000 7fcdfc927640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdf00550f0 0x7fcdf0047060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:42736/0 (socket says 172.21.15.98:42736) 2023-11-28T17:41:26.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.352+0000 7fcdfc927640 1 -- 172.21.15.98:0/2360460754 learned_addr learned my addr 172.21.15.98:0/2360460754 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:26.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.352+0000 7f49150d3640 1 Processor -- start 2023-11-28T17:41:26.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.353+0000 7f49150d3640 1 -- start start 2023-11-28T17:41:26.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.353+0000 7fcdf77fe640 1 --2- 172.21.15.98:0/2360460754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf004f590 0x7fcdf0184cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.354+0000 7fcdfc927640 1 -- 172.21.15.98:0/2360460754 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdf003a010 msgr2=0x7fcdf0048070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.354+0000 7fcdfc927640 1 --2- 172.21.15.98:0/2360460754 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdf003a010 0x7fcdf0048070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.354+0000 7fcdfc927640 1 -- 172.21.15.98:0/2360460754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf004f590 msgr2=0x7fcdf0184cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.354+0000 7fcdfc927640 1 --2- 172.21.15.98:0/2360460754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf004f590 0x7fcdf0184cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.354+0000 7fcdfc927640 1 -- 172.21.15.98:0/2360460754 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde4009400 con 0x7fcdf00550f0 2023-11-28T17:41:26.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.354+0000 7fcdf77fe640 1 --2- 172.21.15.98:0/2360460754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf004f590 0x7fcdf0184cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:41:26.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.354+0000 7fcdfc927640 1 --2- 172.21.15.98:0/2360460754 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdf00550f0 0x7fcdf0047060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8068450 tx=0x7fcdf80681b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.355+0000 7fcdf4ff9640 1 -- 172.21.15.98:0/2360460754 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf806cc40 con 0x7fcdf00550f0 2023-11-28T17:41:26.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.355+0000 7fcdfe38a640 1 -- 172.21.15.98:0/2360460754 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf0038980 con 0x7fcdf00550f0 2023-11-28T17:41:26.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.355+0000 7f918ffff640 1 Processor -- start 2023-11-28T17:41:26.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.355+0000 7f49150d3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f490803a010 0x7f4908007190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.356+0000 7f49150d3640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49080550f0 0x7f49080076d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.356+0000 7f490ed9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f490803a010 0x7f4908007190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.356+0000 7f918ffff640 1 -- start start 2023-11-28T17:41:26.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.356+0000 7fcdfe38a640 1 -- 172.21.15.98:0/2360460754 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf0185620 con 0x7fcdf00550f0 2023-11-28T17:41:26.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.356+0000 7f490ed9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f490803a010 0x7f4908007190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49496/0 (socket says 172.21.15.98:49496) 2023-11-28T17:41:26.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.356+0000 7f490ed9d640 1 -- 172.21.15.98:0/3679549525 learned_addr learned my addr 172.21.15.98:0/3679549525 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:26.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.356+0000 7f49150d3640 1 -- 172.21.15.98:0/3679549525 --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4908006180 con 0x7f490804f590 2023-11-28T17:41:26.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.356+0000 7f49150d3640 1 -- 172.21.15.98:0/3679549525 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f490803a3f0 con 0x7f490803a010 2023-11-28T17:41:26.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.356+0000 7f49150d3640 1 -- 172.21.15.98:0/3679549525 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49080e3220 con 0x7f49080550f0 2023-11-28T17:41:26.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.357+0000 7fcdf4ff9640 1 -- 172.21.15.98:0/2360460754 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf806cda0 con 0x7fcdf00550f0 2023-11-28T17:41:26.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.357+0000 7f490e59c640 1 --2- 172.21.15.98:0/3679549525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49080550f0 0x7f49080076d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.357+0000 7f490e59c640 1 -- 172.21.15.98:0/3679549525 >> v2:172.21.15.98:3300/0 conn(0x7f490803a010 msgr2=0x7f4908007190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:26.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.357+0000 7f490e59c640 1 --2- 172.21.15.98:0/3679549525 >> v2:172.21.15.98:3300/0 conn(0x7f490803a010 0x7f4908007190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.357+0000 7f490e59c640 1 -- 172.21.15.98:0/3679549525 >> v1:172.21.15.113:6789/0 conn(0x7f490804f590 legacy=0x7f4908050140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.358+0000 7f490e59c640 1 -- 172.21.15.98:0/3679549525 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49080063d0 con 0x7f49080550f0 2023-11-28T17:41:26.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.358+0000 7f490ed9d640 1 --2- 172.21.15.98:0/3679549525 >> v2:172.21.15.98:3300/0 conn(0x7f490803a010 0x7f4908007190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:41:26.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.359+0000 7f918ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880e24e0 0x7f91881760f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.359+0000 7f918ffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91880e28c0 0x7f91880d9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.359+0000 7f918ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91880e2de0 0x7f91880db470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.359+0000 7f918ffff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91880db1b0 con 0x7f91880e24e0 2023-11-28T17:41:26.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.359+0000 7f918ffff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9188032d10 con 0x7f91880e2de0 2023-11-28T17:41:26.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.359+0000 7f918ffff640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9188176630 con 0x7f91880e28c0 2023-11-28T17:41:26.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.359+0000 7fcdf4ff9640 1 -- 172.21.15.98:0/2360460754 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf806cc40 con 0x7fcdf00550f0 2023-11-28T17:41:26.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.359+0000 7fcdfe38a640 1 -- 172.21.15.98:0/2360460754 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd8005250 con 0x7fcdf00550f0 2023-11-28T17:41:26.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.360+0000 7f918effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880e24e0 0x7f91881760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.360+0000 7f918effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880e24e0 0x7f91881760f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49504/0 (socket says 172.21.15.98:49504) 2023-11-28T17:41:26.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.360+0000 7f918effd640 1 -- 172.21.15.98:0/157184694 learned_addr learned my addr 172.21.15.98:0/157184694 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:26.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.360+0000 7f918e7fc640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91880e28c0 0x7f91880d9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.360+0000 7f918f7fe640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91880e2de0 0x7f91880db470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.360+0000 7f490e59c640 1 --2- 172.21.15.98:0/3679549525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49080550f0 0x7f49080076d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f49040097a0 tx=0x7f490402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aba61a0c1d5a68ba server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.360+0000 7f918effd640 1 -- 172.21.15.98:0/157184694 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91880e28c0 msgr2=0x7f91880d9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.360+0000 7f918effd640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91880e28c0 0x7f91880d9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.360+0000 7f918effd640 1 -- 172.21.15.98:0/157184694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91880e2de0 msgr2=0x7f91880db470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.360+0000 7f918effd640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91880e2de0 0x7f91880db470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.360+0000 7f918effd640 1 -- 172.21.15.98:0/157184694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91800094d0 con 0x7f91880e24e0 2023-11-28T17:41:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.360+0000 7f918e7fc640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91880e28c0 0x7f91880d9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:41:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.360+0000 7f918f7fe640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91880e2de0 0x7f91880db470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:41:26.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.361+0000 7f490dd9b640 1 -- 172.21.15.98:0/3679549525 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490402d040 con 0x7f49080550f0 2023-11-28T17:41:26.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.361+0000 7f918effd640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880e24e0 0x7f91881760f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f918000e9b0 tx=0x7f918000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.361+0000 7f9177fff640 1 -- 172.21.15.98:0/157184694 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918000c9a0 con 0x7f91880e24e0 2023-11-28T17:41:26.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.361+0000 7f9177fff640 1 -- 172.21.15.98:0/157184694 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918000cb00 con 0x7f91880e24e0 2023-11-28T17:41:26.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.361+0000 7f9177fff640 1 -- 172.21.15.98:0/157184694 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918000ce70 con 0x7f91880e24e0 2023-11-28T17:41:26.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.361+0000 7f918ffff640 1 -- 172.21.15.98:0/157184694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9184009400 con 0x7f91880e24e0 2023-11-28T17:41:26.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.362+0000 7f918ffff640 1 -- 172.21.15.98:0/157184694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9188031960 con 0x7f91880e24e0 2023-11-28T17:41:26.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.362+0000 7f490dd9b640 1 -- 172.21.15.98:0/3679549525 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4904031810 con 0x7f49080550f0 2023-11-28T17:41:26.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.362+0000 7f490dd9b640 1 -- 172.21.15.98:0/3679549525 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904031ba0 con 0x7f49080550f0 2023-11-28T17:41:26.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.363+0000 7f49150d3640 1 -- 172.21.15.98:0/3679549525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49080550f0 msgr2=0x7f49080076d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.363+0000 7f49150d3640 1 --2- 172.21.15.98:0/3679549525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49080550f0 0x7f49080076d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f49040097a0 tx=0x7f490402b8b0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.364+0000 7f49150d3640 1 -- 172.21.15.98:0/3679549525 shutdown_connections 2023-11-28T17:41:26.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.364+0000 7f49150d3640 1 --2- 172.21.15.98:0/3679549525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49080550f0 0x7f49080076d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.364+0000 7f49150d3640 1 --2- 172.21.15.98:0/3679549525 >> v2:172.21.15.98:3300/0 conn(0x7f490803a010 0x7f4908007190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.364+0000 7f49150d3640 1 -- 172.21.15.98:0/3679549525 >> 172.21.15.98:0/3679549525 conn(0x7f49080545d0 msgr2=0x7f49080549e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.364+0000 7fcdf4ff9640 1 -- 172.21.15.98:0/2360460754 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fcdf8091020 con 0x7fcdf00550f0 2023-11-28T17:41:26.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.364+0000 7f49150d3640 1 -- 172.21.15.98:0/3679549525 shutdown_connections 2023-11-28T17:41:26.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.364+0000 7f49150d3640 1 -- 172.21.15.98:0/3679549525 wait complete. 2023-11-28T17:41:26.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.365+0000 7f918ffff640 1 -- 172.21.15.98:0/157184694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9188005360 con 0x7f91880e24e0 2023-11-28T17:41:26.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.364+0000 7f1b4c63d640 1 Processor -- start 2023-11-28T17:41:26.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.365+0000 7f9177fff640 1 -- 172.21.15.98:0/157184694 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f9180013470 con 0x7f91880e24e0 2023-11-28T17:41:26.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.366+0000 7f41dd067640 1 Processor -- start 2023-11-28T17:41:26.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.366+0000 7fcdf4ff9640 1 --2- 172.21.15.98:0/2360460754 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcddc06b360 0x7fcddc06d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.366+0000 7f41dd067640 1 -- start start 2023-11-28T17:41:26.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.367+0000 7fcdf7fff640 1 --2- 172.21.15.98:0/2360460754 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcddc06b360 0x7fcddc06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.367+0000 7f1b4c63d640 1 -- start start 2023-11-28T17:41:26.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.367+0000 7f9177fff640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f917c06b2a0 0x7f917c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.367+0000 7f9177fff640 1 -- 172.21.15.98:0/157184694 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f918008a140 con 0x7f91880e24e0 2023-11-28T17:41:26.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.368+0000 7f41dd067640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d80c6720 0x7f41d80c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.367+0000 7fcdf4ff9640 1 -- 172.21.15.98:0/2360460754 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcdf8104410 con 0x7fcdf00550f0 2023-11-28T17:41:26.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.368+0000 7fcdf7fff640 1 --2- 172.21.15.98:0/2360460754 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcddc06b360 0x7fcddc06d820 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcde4036000 tx=0x7fcde403b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.368+0000 7f41dd067640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f41d806c940 0x7f41d806cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.368+0000 7f918e7fc640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f917c06b2a0 0x7f917c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.369+0000 7f41dd067640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f41d80c58e0 0x7f41d80c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.369+0000 7f41d7fff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d80c6720 0x7f41d80c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.369+0000 7f41d7fff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d80c6720 0x7f41d80c6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:55664/0 (socket says 172.21.15.98:55664) 2023-11-28T17:41:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.369+0000 7f918e7fc640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f917c06b2a0 0x7f917c06d760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9184036000 tx=0x7f9184008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.369+0000 7f1b4c63d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1b440c3050 0x7f1b440c3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.369+0000 7f1b4c63d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1b4415bbd0 0x7f1b44154d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.369+0000 7f1b4c63d640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4415c110 con 0x7f1b440af9f0 2023-11-28T17:41:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.369+0000 7f41dd067640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d80c7040 con 0x7f41d80c58e0 2023-11-28T17:41:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.369+0000 7f41dd067640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d80c7570 con 0x7f41d806c940 2023-11-28T17:41:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.369+0000 7f41dd067640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d815a100 con 0x7f41d80c6720 2023-11-28T17:41:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.369+0000 7f41d77fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f41d806c940 0x7f41d806cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.369+0000 7f1b4c63d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b440c2a80 con 0x7f1b440c3050 2023-11-28T17:41:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.369+0000 7f1b4c63d640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b440c2500 con 0x7f1b4415bbd0 2023-11-28T17:41:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.370+0000 7f49150d3640 1 Processor -- start 2023-11-28T17:41:26.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.370+0000 7f1b49bd8640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1b4415bbd0 0x7f1b44154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.370+0000 7f1b49bd8640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1b4415bbd0 0x7f1b44154d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:42772/0 (socket says 172.21.15.98:42772) 2023-11-28T17:41:26.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.370+0000 7f1b49bd8640 1 -- 172.21.15.98:0/2052778275 learned_addr learned my addr 172.21.15.98:0/2052778275 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:26.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.370+0000 7f49150d3640 1 -- start start 2023-11-28T17:41:26.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.369+0000 7f41d77fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f41d806c940 0x7f41d806cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:42760/0 (socket says 172.21.15.98:42760) 2023-11-28T17:41:26.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.370+0000 7f41d77fe640 1 -- 172.21.15.98:0/1429023868 learned_addr learned my addr 172.21.15.98:0/1429023868 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:26.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.370+0000 7f41d77fe640 1 -- 172.21.15.98:0/1429023868 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d80c6720 msgr2=0x7f41d80c6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:26.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.370+0000 7f41d77fe640 1 --2- 172.21.15.98:0/1429023868 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d80c6720 0x7f41d80c6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.370+0000 7f41d77fe640 1 -- 172.21.15.98:0/1429023868 >> v2:172.21.15.98:3300/0 conn(0x7f41d80c58e0 msgr2=0x7f41d80c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:26.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.370+0000 7f41d77fe640 1 --2- 172.21.15.98:0/1429023868 >> v2:172.21.15.98:3300/0 conn(0x7f41d80c58e0 0x7f41d80c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.370+0000 7f41d77fe640 1 -- 172.21.15.98:0/1429023868 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d80c5510 con 0x7f41d806c940 2023-11-28T17:41:26.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.370+0000 7f41d7fff640 1 --2- 172.21.15.98:0/1429023868 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d80c6720 0x7f41d80c6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:41:26.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.371+0000 7f1b49bd8640 1 -- 172.21.15.98:0/2052778275 >> v2:172.21.15.98:3300/0 conn(0x7f1b440c3050 msgr2=0x7f1b440c3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:26.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.371+0000 7f1b49bd8640 1 --2- 172.21.15.98:0/2052778275 >> v2:172.21.15.98:3300/0 conn(0x7f1b440c3050 0x7f1b440c3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.371+0000 7f1b49bd8640 1 -- 172.21.15.98:0/2052778275 >> v1:172.21.15.113:6789/0 conn(0x7f1b440af9f0 legacy=0x7f1b44157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:26.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.371+0000 7f1b49bd8640 1 -- 172.21.15.98:0/2052778275 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4406d5f0 con 0x7f1b4415bbd0 2023-11-28T17:41:26.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.372+0000 7f49150d3640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f490803a010 0x7f49081771d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.372+0000 7fa6782da640 1 Processor -- start 2023-11-28T17:41:26.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.372+0000 7f49150d3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f490804f590 0x7f4908181f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.372+0000 7f49150d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49080550f0 0x7f4908047160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.372+0000 7f49150d3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490804a360 con 0x7f49080550f0 2023-11-28T17:41:26.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.372+0000 7f49150d3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4908181ae0 con 0x7f490804f590 2023-11-28T17:41:26.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.372+0000 7f49150d3640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49081757d0 con 0x7f490803a010 2023-11-28T17:41:26.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.372+0000 7f490ed9d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f490803a010 0x7f49081771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.372+0000 7f490ed9d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f490803a010 0x7f49081771d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:55670/0 (socket says 172.21.15.98:55670) 2023-11-28T17:41:26.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.372+0000 7f490ed9d640 1 -- 172.21.15.98:0/2273214372 learned_addr learned my addr 172.21.15.98:0/2273214372 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:26.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.373+0000 7f1b49bd8640 1 --2- 172.21.15.98:0/2052778275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b4415bbd0 0x7f1b44154d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1b44070d90 tx=0x7f1b3802bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=457689bb3cbfcee7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.373+0000 7f41d77fe640 1 --2- 172.21.15.98:0/1429023868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d806c940 0x7f41d806cd80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f41d80cbfa0 tx=0x7f41c802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3396c12d0e6e9639 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.372+0000 7fa6782da640 1 -- start start 2023-11-28T17:41:26.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.373+0000 7f490ed9d640 1 -- 172.21.15.98:0/2273214372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f490804f590 msgr2=0x7f4908181f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.373+0000 7f490ed9d640 1 --2- 172.21.15.98:0/2273214372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f490804f590 0x7f4908181f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.373+0000 7f490ed9d640 1 -- 172.21.15.98:0/2273214372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49080550f0 msgr2=0x7f4908047160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:41:26.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.373+0000 7f490f59e640 1 --2- 172.21.15.98:0/2273214372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49080550f0 0x7f4908047160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.373+0000 7f490ed9d640 1 --2- 172.21.15.98:0/2273214372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49080550f0 0x7f4908047160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.373+0000 7f490ed9d640 1 -- 172.21.15.98:0/2273214372 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49000094d0 con 0x7f490803a010 2023-11-28T17:41:26.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.373+0000 7f490f59e640 1 --2- 172.21.15.98:0/2273214372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49080550f0 0x7f4908047160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:26.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.374+0000 7f35db275640 1 Processor -- start 2023-11-28T17:41:26.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.374+0000 7fa6782da640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa670161680 0x7fa670161b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.374+0000 7fa6782da640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6700cab90 con 0x7fa6701582e0 2023-11-28T17:41:26.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.374+0000 7fa6782da640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6700c66b0 con 0x7fa6700c5f10 2023-11-28T17:41:26.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.374+0000 7fa6782da640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6700c7470 con 0x7fa670161680 2023-11-28T17:41:26.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.374+0000 7f35db275640 1 -- start start 2023-11-28T17:41:26.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.374+0000 7f9177fff640 1 -- 172.21.15.98:0/157184694 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9180054fb0 con 0x7f91880e24e0 2023-11-28T17:41:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.374+0000 7fa676076640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa6701582e0 0x7fa6700c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37528/0 (socket says 172.21.15.98:37528) 2023-11-28T17:41:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.374+0000 7fa676076640 1 -- 172.21.15.98:0/3861606406 learned_addr learned my addr 172.21.15.98:0/3861606406 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.375+0000 7fa676877640 1 --2- 172.21.15.98:0/3861606406 >> v2:172.21.15.98:3300/0 conn(0x7fa670161680 0x7fa670161b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.375+0000 7f490ed9d640 1 --2- 172.21.15.98:0/2273214372 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f490803a010 0x7f49081771d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f490000e9d0 tx=0x7f490000ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.375+0000 7fa675074640 1 -- 172.21.15.98:0/3861606406 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629033960 0 0) 0x7fa6700cab90 con 0x7fa6701582e0 2023-11-28T17:41:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.375+0000 7fa675074640 1 -- 172.21.15.98:0/3861606406 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa660003520 con 0x7fa6701582e0 2023-11-28T17:41:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.375+0000 7f48f77fe640 1 -- 172.21.15.98:0/2273214372 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490000ca60 con 0x7f490803a010 2023-11-28T17:41:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.375+0000 7f48f77fe640 1 -- 172.21.15.98:0/2273214372 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490000cbc0 con 0x7f490803a010 2023-11-28T17:41:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.375+0000 7f48f77fe640 1 -- 172.21.15.98:0/2273214372 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4900013470 con 0x7f490803a010 2023-11-28T17:41:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.375+0000 7fa675074640 1 -- 172.21.15.98:0/3861606406 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2494315730 0 0) 0x7fa660003520 con 0x7fa6701582e0 2023-11-28T17:41:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.375+0000 7fa675074640 1 -- 172.21.15.98:0/3861606406 >> v2:172.21.15.98:3300/0 conn(0x7fa670161680 msgr2=0x7fa670161b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.375+0000 7fa675074640 1 --2- 172.21.15.98:0/3861606406 >> v2:172.21.15.98:3300/0 conn(0x7fa670161680 0x7fa670161b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.375+0000 7fa675074640 1 -- 172.21.15.98:0/3861606406 >> v1:172.21.15.113:6789/0 conn(0x7fa6700c5f10 legacy=0x7fa6700c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.375+0000 7fa675074640 1 -- 172.21.15.98:0/3861606406 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6700c6990 con 0x7fa6701582e0 2023-11-28T17:41:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.376+0000 7f49150d3640 1 -- 172.21.15.98:0/2273214372 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4904009400 con 0x7f490803a010 2023-11-28T17:41:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.376+0000 7fa675074640 1 -- 172.21.15.98:0/3861606406 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7fa6640019d0 con 0x7fa6701582e0 2023-11-28T17:41:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.376+0000 7fa675074640 1 -- 172.21.15.98:0/3861606406 >> v1:172.21.15.98:6789/0 conn(0x7fa6701582e0 legacy=0x7fa6700c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.376+0000 7fa675074640 1 --2- 172.21.15.98:0/3861606406 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa660003a10 0x7fa660003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.376+0000 7fa675074640 1 --2- 172.21.15.98:0/3861606406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6600047c0 0x7fa660004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.376+0000 7fa675074640 1 --2- 172.21.15.98:0/3861606406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa660005180 0x7fa66000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.376+0000 7fa675074640 1 -- 172.21.15.98:0/3861606406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6700c7470 con 0x7fa660005180 2023-11-28T17:41:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.376+0000 7fa675074640 1 -- 172.21.15.98:0/3861606406 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66000c1b0 con 0x7fa6600047c0 2023-11-28T17:41:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.376+0000 7fa675074640 1 -- 172.21.15.98:0/3861606406 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66000c370 con 0x7fa660003a10 2023-11-28T17:41:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.376+0000 7f49150d3640 1 -- 172.21.15.98:0/2273214372 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4908048940 con 0x7f490803a010 2023-11-28T17:41:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.376+0000 7f35db275640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35d40c6d70 0x7f35d40ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.376+0000 7f35db275640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d40c72c0 con 0x7f35d41541f0 2023-11-28T17:41:26.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.376+0000 7f35db275640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d41545d0 con 0x7f35d40ca920 2023-11-28T17:41:26.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.376+0000 7f35db275640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d40c9410 con 0x7f35d40c6d70 2023-11-28T17:41:26.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.376+0000 7fa675875640 1 --2- 172.21.15.98:0/3861606406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6600047c0 0x7fa660004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.377+0000 7f35d9011640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f35d41541f0 0x7f35d4159e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37536/0 (socket says 172.21.15.98:37536) 2023-11-28T17:41:26.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.377+0000 7f35d9011640 1 -- 172.21.15.98:0/904164915 learned_addr learned my addr 172.21.15.98:0/904164915 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:26.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.377+0000 7fa676877640 1 --2- 172.21.15.98:0/3861606406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa660005180 0x7fa66000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.377+0000 7f35d8810640 1 --2- 172.21.15.98:0/904164915 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35d40c6d70 0x7f35d40ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.377+0000 7fa676076640 1 --2- 172.21.15.98:0/3861606406 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa660003a10 0x7fa660003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.377+0000 7fa676877640 1 -- 172.21.15.98:0/3861606406 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa660003a10 msgr2=0x7fa660003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:26.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.377+0000 7fa676877640 1 --2- 172.21.15.98:0/3861606406 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa660003a10 0x7fa660003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.377+0000 7f35cbfff640 1 -- 172.21.15.98:0/904164915 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268054639 0 0) 0x7f35d40c72c0 con 0x7f35d41541f0 2023-11-28T17:41:26.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.377+0000 7f35cbfff640 1 -- 172.21.15.98:0/904164915 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c0003520 con 0x7f35d41541f0 2023-11-28T17:41:26.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.377+0000 7f49150d3640 1 -- 172.21.15.98:0/2273214372 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4908005350 con 0x7f490803a010 2023-11-28T17:41:26.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.377+0000 7fa676877640 1 -- 172.21.15.98:0/3861606406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6600047c0 msgr2=0x7fa660004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:26.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.377+0000 7fa676877640 1 --2- 172.21.15.98:0/3861606406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa6600047c0 0x7fa660004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.377+0000 7fa676877640 1 -- 172.21.15.98:0/3861606406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa660003520 con 0x7fa660005180 2023-11-28T17:41:26.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.377+0000 7fcb10d43640 1 Processor -- start 2023-11-28T17:41:26.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.377+0000 7fcb10d43640 1 -- start start 2023-11-28T17:41:26.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7f35cbfff640 1 -- 172.21.15.98:0/904164915 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3771733833 0 0) 0x7f35c0003520 con 0x7f35d41541f0 2023-11-28T17:41:26.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7f35cbfff640 1 -- 172.21.15.98:0/904164915 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35d40c6d70 msgr2=0x7f35d40ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:26.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7f35cbfff640 1 --2- 172.21.15.98:0/904164915 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35d40c6d70 0x7f35d40ca3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7f35cbfff640 1 -- 172.21.15.98:0/904164915 >> v1:172.21.15.113:6789/0 conn(0x7f35d40ca920 legacy=0x7f35d40c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7f35cbfff640 1 -- 172.21.15.98:0/904164915 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d415a660 con 0x7f35d41541f0 2023-11-28T17:41:26.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7f35cbfff640 1 -- 172.21.15.98:0/904164915 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f35d0001670 con 0x7f35d41541f0 2023-11-28T17:41:26.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7f35cbfff640 1 -- 172.21.15.98:0/904164915 >> v1:172.21.15.98:6789/0 conn(0x7f35d41541f0 legacy=0x7f35d4159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7f35cbfff640 1 --2- 172.21.15.98:0/904164915 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35c00039c0 0x7f35c0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7f35cbfff640 1 --2- 172.21.15.98:0/904164915 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35c00047c0 0x7f35c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7f35cbfff640 1 --2- 172.21.15.98:0/904164915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c0005180 0x7f35c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7f35cbfff640 1 -- 172.21.15.98:0/904164915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c000bfa0 con 0x7f35c0005180 2023-11-28T17:41:26.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7f35cbfff640 1 -- 172.21.15.98:0/904164915 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c0014920 con 0x7f35c00047c0 2023-11-28T17:41:26.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7f35cbfff640 1 -- 172.21.15.98:0/904164915 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c0014ae0 con 0x7f35c00039c0 2023-11-28T17:41:26.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7fcb10d43640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb0c0cf6d0 0x7fcb0c15e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7fcb10d43640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fcb0c0b33d0 0x7fcb0c0ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7fcb10d43640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0c161d60 con 0x7fcb0c0cfab0 2023-11-28T17:41:26.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7fcb10d43640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb0c15f2b0 con 0x7fcb0c0b33d0 2023-11-28T17:41:26.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.378+0000 7fcb10d43640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c15e7a0 con 0x7fcb0c0cf6d0 2023-11-28T17:41:26.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.379+0000 7f35d9812640 1 --2- 172.21.15.98:0/904164915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c0005180 0x7f35c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.379+0000 7fcb09d9b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcb0c0cfab0 0x7fcb0c1621f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37544/0 (socket says 172.21.15.98:37544) 2023-11-28T17:41:26.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.379+0000 7fcb09d9b640 1 -- 172.21.15.98:0/3113501820 learned_addr learned my addr 172.21.15.98:0/3113501820 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:26.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.379+0000 7fa676877640 1 -- 172.21.15.98:0/3861606406 reap_dead start 2023-11-28T17:41:26.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.379+0000 7fcb0ad9d640 1 --2- 172.21.15.98:0/3113501820 >> v2:172.21.15.113:3300/0 conn(0x7fcb0c0b33d0 0x7fcb0c0ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.379+0000 7fcb0a59c640 1 --2- 172.21.15.98:0/3113501820 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb0c0cf6d0 0x7fcb0c15e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.379+0000 7f35d9011640 1 --2- 172.21.15.98:0/904164915 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35c00039c0 0x7f35c0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.379+0000 7f41d6ffd640 1 -- 172.21.15.98:0/1429023868 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c802ca80 con 0x7f41d806c940 2023-11-28T17:41:26.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.379+0000 7f35d8810640 1 --2- 172.21.15.98:0/904164915 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35c00047c0 0x7f35c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.379+0000 7f1b493d7640 1 -- 172.21.15.98:0/2052778275 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3802cb30 con 0x7f1b4415bbd0 2023-11-28T17:41:26.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.380+0000 7f1b493d7640 1 -- 172.21.15.98:0/2052778275 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3802cc90 con 0x7f1b4415bbd0 2023-11-28T17:41:26.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.380+0000 7f1b493d7640 1 -- 172.21.15.98:0/2052778275 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b38034610 con 0x7f1b4415bbd0 2023-11-28T17:41:26.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.380+0000 7f41d6ffd640 1 -- 172.21.15.98:0/1429023868 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c802cbe0 con 0x7f41d806c940 2023-11-28T17:41:26.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.380+0000 7f41d6ffd640 1 -- 172.21.15.98:0/1429023868 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c80343f0 con 0x7f41d806c940 2023-11-28T17:41:26.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.379+0000 7f35d9812640 1 -- 172.21.15.98:0/904164915 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35c00039c0 msgr2=0x7f35c0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:26.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.379+0000 7f35d9812640 1 --2- 172.21.15.98:0/904164915 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35c00039c0 0x7f35c0003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.379+0000 7f35d9812640 1 -- 172.21.15.98:0/904164915 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35c00047c0 msgr2=0x7f35c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:26.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.379+0000 7f35d9812640 1 --2- 172.21.15.98:0/904164915 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35c00047c0 0x7f35c0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.379+0000 7f35d9812640 1 -- 172.21.15.98:0/904164915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d40c72c0 con 0x7f35c0005180 2023-11-28T17:41:26.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.381+0000 7fcdf4ff9640 1 -- 172.21.15.98:0/2360460754 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcdf80cf330 con 0x7fcdf00550f0 2023-11-28T17:41:26.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.381+0000 7f41dd067640 1 -- 172.21.15.98:0/1429023868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d806c940 msgr2=0x7f41d806cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.381+0000 7f41dd067640 1 --2- 172.21.15.98:0/1429023868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d806c940 0x7f41d806cd80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f41d80cbfa0 tx=0x7f41c802bee0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.381+0000 7f35d9812640 1 --2- 172.21.15.98:0/904164915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c0005180 0x7f35c000b980 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f35cc001e50 tx=0x7f35c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24c5c0c8d2d0fc79 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.382+0000 7f35cbfff640 1 -- 172.21.15.98:0/904164915 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc003b70 con 0x7f35c0005180 2023-11-28T17:41:26.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.382+0000 7f35cbfff640 1 -- 172.21.15.98:0/904164915 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35cc003cd0 con 0x7f35c0005180 2023-11-28T17:41:26.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.382+0000 7f35cbfff640 1 -- 172.21.15.98:0/904164915 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc005310 con 0x7f35c0005180 2023-11-28T17:41:26.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.380+0000 7fa676877640 1 --2- 172.21.15.98:0/3861606406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa660005180 0x7fa66000b980 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa668009ac0 tx=0x7fa6600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de69abf65d8e4d62 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.380+0000 7fcb0ad9d640 1 -- 172.21.15.98:0/3113501820 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb0c0cf6d0 msgr2=0x7fcb0c15e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:26.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.380+0000 7fcb0ad9d640 1 --2- 172.21.15.98:0/3113501820 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb0c0cf6d0 0x7fcb0c15e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.380+0000 7fcb0ad9d640 1 -- 172.21.15.98:0/3113501820 >> v1:172.21.15.98:6789/0 conn(0x7fcb0c0cfab0 legacy=0x7fcb0c1621f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.380+0000 7fcb0ad9d640 1 -- 172.21.15.98:0/3113501820 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c0caf90 con 0x7fcb0c0b33d0 2023-11-28T17:41:26.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.380+0000 7fcb0959a640 1 -- 172.21.15.98:0/3113501820 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667821824 0 0) 0x7fcb0c161d60 con 0x7fcb0c0cfab0 2023-11-28T17:41:26.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.380+0000 7fcb0a59c640 1 --2- 172.21.15.98:0/3113501820 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb0c0cf6d0 0x7fcb0c15e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:41:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.382+0000 7fa675074640 1 -- 172.21.15.98:0/3861606406 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa668004020 con 0x7fa660005180 2023-11-28T17:41:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.382+0000 7fa675074640 1 -- 172.21.15.98:0/3861606406 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa668004180 con 0x7fa660005180 2023-11-28T17:41:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.382+0000 7fcb0ad9d640 1 --2- 172.21.15.98:0/3113501820 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb0c0b33d0 0x7fcb0c0ca420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcafc0097a0 tx=0x7fcafc02b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=690aef2bf58e5ab6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.382+0000 7fa675074640 1 -- 172.21.15.98:0/3861606406 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6680044f0 con 0x7fa660005180 2023-11-28T17:41:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.383+0000 7f48f77fe640 1 -- 172.21.15.98:0/2273214372 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f49000135d0 con 0x7f490803a010 2023-11-28T17:41:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.383+0000 7f41dd067640 1 -- 172.21.15.98:0/1429023868 shutdown_connections 2023-11-28T17:41:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.383+0000 7f41dd067640 1 --2- 172.21.15.98:0/1429023868 >> v2:172.21.15.98:3300/0 conn(0x7f41d80c58e0 0x7f41d80c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.383+0000 7f41dd067640 1 --2- 172.21.15.98:0/1429023868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d806c940 0x7f41d806cd80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.383+0000 7f41dd067640 1 --2- 172.21.15.98:0/1429023868 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d80c6720 0x7f41d80c6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.383+0000 7f41dd067640 1 -- 172.21.15.98:0/1429023868 >> 172.21.15.98:0/1429023868 conn(0x7f41d80cafe0 msgr2=0x7f41d80cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.383+0000 7f41dd067640 1 -- 172.21.15.98:0/1429023868 shutdown_connections 2023-11-28T17:41:26.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.383+0000 7f41dd067640 1 -- 172.21.15.98:0/1429023868 wait complete. 2023-11-28T17:41:26.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.382+0000 7f35db275640 1 -- 172.21.15.98:0/904164915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c0005180 msgr2=0x7f35c000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.382+0000 7f35db275640 1 --2- 172.21.15.98:0/904164915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c0005180 0x7f35c000b980 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f35cc001e50 tx=0x7f35c00057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.382+0000 7f35d9812640 1 -- 172.21.15.98:0/904164915 reap_dead start 2023-11-28T17:41:26.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.383+0000 7f48f77fe640 1 --2- 172.21.15.98:0/2273214372 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f48fc06b2a0 0x7f48fc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.384+0000 7fcb0959a640 1 -- 172.21.15.98:0/3113501820 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc030800 con 0x7fcb0c0b33d0 2023-11-28T17:41:26.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.384+0000 7f35db275640 1 -- 172.21.15.98:0/904164915 shutdown_connections 2023-11-28T17:41:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.384+0000 7f35db275640 1 -- 172.21.15.98:0/904164915 >> 172.21.15.98:0/904164915 conn(0x7f35d406af30 msgr2=0x7f35d406b340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.384+0000 7f48f77fe640 1 -- 172.21.15.98:0/2273214372 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f490008a710 con 0x7f490803a010 2023-11-28T17:41:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.383+0000 7f1b4c63d640 1 -- 172.21.15.98:0/2052778275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b4415bbd0 msgr2=0x7f1b44154d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.383+0000 7f1b4c63d640 1 --2- 172.21.15.98:0/2052778275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b4415bbd0 0x7f1b44154d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1b44070d90 tx=0x7f1b3802bf90 comp rx=0 tx=0).stop 2023-11-28T17:41:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.384+0000 7fcb0959a640 1 -- 172.21.15.98:0/3113501820 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcafc030960 con 0x7fcb0c0b33d0 2023-11-28T17:41:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.384+0000 7fcb0959a640 1 -- 172.21.15.98:0/3113501820 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc030cf0 con 0x7fcb0c0b33d0 2023-11-28T17:41:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.384+0000 7fa6782da640 1 -- 172.21.15.98:0/3861606406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa660005180 msgr2=0x7fa66000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.384+0000 7fa6782da640 1 --2- 172.21.15.98:0/3861606406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa660005180 0x7fa66000b980 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa668009ac0 tx=0x7fa6600057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.384+0000 7f490e59c640 1 --2- 172.21.15.98:0/2273214372 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f48fc06b2a0 0x7f48fc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.384+0000 7f1b4c63d640 1 -- 172.21.15.98:0/2052778275 shutdown_connections 2023-11-28T17:41:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.384+0000 7f1b4c63d640 1 --2- 172.21.15.98:0/2052778275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b4415bbd0 0x7f1b44154d80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.384+0000 7f1b4c63d640 1 --2- 172.21.15.98:0/2052778275 >> v2:172.21.15.98:3300/0 conn(0x7f1b440c3050 0x7f1b440c3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.384+0000 7f1b4c63d640 1 -- 172.21.15.98:0/2052778275 >> 172.21.15.98:0/2052778275 conn(0x7f1b44066860 msgr2=0x7f1b440700f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.385+0000 7f35db275640 1 -- 172.21.15.98:0/904164915 shutdown_connections 2023-11-28T17:41:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.385+0000 7f35db275640 1 -- 172.21.15.98:0/904164915 wait complete. 2023-11-28T17:41:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.385+0000 7fa6782da640 1 -- 172.21.15.98:0/3861606406 shutdown_connections 2023-11-28T17:41:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.385+0000 7fa6782da640 1 --2- 172.21.15.98:0/3861606406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa660005180 0x7fa66000b980 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.385+0000 7fa6782da640 1 -- 172.21.15.98:0/3861606406 >> 172.21.15.98:0/3861606406 conn(0x7fa67006a4d0 msgr2=0x7fa67006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.385+0000 7fa6782da640 1 -- 172.21.15.98:0/3861606406 shutdown_connections 2023-11-28T17:41:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.385+0000 7fcb10d43640 1 -- 172.21.15.98:0/3113501820 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb0c0b33d0 msgr2=0x7fcb0c0ca420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.385+0000 7fcb10d43640 1 --2- 172.21.15.98:0/3113501820 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb0c0b33d0 0x7fcb0c0ca420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcafc0097a0 tx=0x7fcafc02b940 comp rx=0 tx=0).stop 2023-11-28T17:41:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.385+0000 7f490e59c640 1 --2- 172.21.15.98:0/2273214372 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f48fc06b2a0 0x7f48fc06d760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4904036000 tx=0x7f4904030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.385+0000 7f1b4c63d640 1 -- 172.21.15.98:0/2052778275 shutdown_connections 2023-11-28T17:41:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.385+0000 7f1b4c63d640 1 -- 172.21.15.98:0/2052778275 wait complete. 2023-11-28T17:41:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.385+0000 7fa6782da640 1 -- 172.21.15.98:0/3861606406 wait complete. 2023-11-28T17:41:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.385+0000 7fcb10d43640 1 -- 172.21.15.98:0/3113501820 shutdown_connections 2023-11-28T17:41:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.385+0000 7fcb10d43640 1 --2- 172.21.15.98:0/3113501820 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb0c0b33d0 0x7fcb0c0ca420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.385+0000 7fcb10d43640 1 --2- 172.21.15.98:0/3113501820 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb0c0cf6d0 0x7fcb0c15e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.386+0000 7fcb10d43640 1 -- 172.21.15.98:0/3113501820 >> 172.21.15.98:0/3113501820 conn(0x7fcb0c06af10 msgr2=0x7fcb0c06b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.386+0000 7f41dd067640 1 Processor -- start 2023-11-28T17:41:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.387+0000 7f35db275640 1 Processor -- start 2023-11-28T17:41:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.387+0000 7f35db275640 1 -- start start 2023-11-28T17:41:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.386+0000 7fcb10d43640 1 -- 172.21.15.98:0/3113501820 shutdown_connections 2023-11-28T17:41:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.386+0000 7fcb10d43640 1 -- 172.21.15.98:0/3113501820 wait complete. 2023-11-28T17:41:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.386+0000 7f41dd067640 1 -- start start 2023-11-28T17:41:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.387+0000 7f41dd067640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d806c940 0x7f41d81567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.387+0000 7f41dd067640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d80c58e0 0x7f41d8155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.387+0000 7f41dd067640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d80c6720 0x7f41d80c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.388+0000 7f41dd067640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d8156530 con 0x7f41d80c58e0 2023-11-28T17:41:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.388+0000 7f41dd067640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d8146bb0 con 0x7f41d806c940 2023-11-28T17:41:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.388+0000 7f41dd067640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d8156120 con 0x7f41d80c6720 2023-11-28T17:41:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.388+0000 7f1b4c63d640 1 Processor -- start 2023-11-28T17:41:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.388+0000 7f1b4c63d640 1 -- start start 2023-11-28T17:41:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.388+0000 7fcb10d43640 1 Processor -- start 2023-11-28T17:41:26.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.389+0000 7f41d7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d806c940 0x7f41d81567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.389+0000 7f41dc866640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d80c6720 0x7f41d80c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.389+0000 7f41dc866640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d80c6720 0x7f41d80c3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:55724/0 (socket says 172.21.15.98:55724) 2023-11-28T17:41:26.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.389+0000 7f41dc866640 1 -- 172.21.15.98:0/3636381290 learned_addr learned my addr 172.21.15.98:0/3636381290 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:26.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.389+0000 7fa6782da640 1 Processor -- start 2023-11-28T17:41:26.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.389+0000 7fa6782da640 1 -- start start 2023-11-28T17:41:26.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.388+0000 7f35db275640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c0005180 0x7f35d41ec290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.388+0000 7f35db275640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35d41483e0 0x7f35d41590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.388+0000 7f35db275640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35d41ee390 0x7f35d41ee810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.388+0000 7f35db275640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d4147fc0 con 0x7f35c0005180 2023-11-28T17:41:26.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.388+0000 7f35db275640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d414a380 con 0x7f35d41483e0 2023-11-28T17:41:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.388+0000 7f35db275640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d40c2d70 con 0x7f35d41ee390 2023-11-28T17:41:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.389+0000 7f35d9011640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c0005180 0x7f35d41ec290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.389+0000 7f35d9011640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c0005180 0x7f35d41ec290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49574/0 (socket says 172.21.15.98:49574) 2023-11-28T17:41:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.389+0000 7f35d9011640 1 -- 172.21.15.98:0/1249706644 learned_addr learned my addr 172.21.15.98:0/1249706644 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.389+0000 7f35d9812640 1 --2- 172.21.15.98:0/1249706644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35d41ee390 0x7f35d41ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.389+0000 7f35d8810640 1 --2- 172.21.15.98:0/1249706644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35d41483e0 0x7f35d41590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.389+0000 7fcb10d43640 1 -- start start 2023-11-28T17:41:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7f1b4c63d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1b440af9f0 0x7f1b4406e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7f1b4c63d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b440c3050 0x7f1b4416a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7f1b4c63d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b4415bbd0 0x7f1b4416af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7f1b4c63d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b44145e20 con 0x7f1b4415bbd0 2023-11-28T17:41:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7f1b4c63d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4415ad50 con 0x7f1b440c3050 2023-11-28T17:41:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7f1b4c63d640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b44146030 con 0x7f1b440af9f0 2023-11-28T17:41:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7f41d77fe640 1 --2- 172.21.15.98:0/3636381290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d80c58e0 0x7f41d8155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7f35d9011640 1 -- 172.21.15.98:0/1249706644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35d41ee390 msgr2=0x7f35d41ee810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7fcb10d43640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb0c0b33d0 0x7fcb0c0cee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7fcb10d43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb0c0cf6d0 0x7fcb0c0cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7fcb10d43640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb0c0cfab0 0x7fcb0c1608b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7fcb10d43640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c1f4970 con 0x7fcb0c0cf6d0 2023-11-28T17:41:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7fcb10d43640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c161cc0 con 0x7fcb0c0b33d0 2023-11-28T17:41:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7fcb10d43640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c161570 con 0x7fcb0c0cfab0 2023-11-28T17:41:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7f1b4abda640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b4415bbd0 0x7f1b4416af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7fcb0ad9d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb0c0cfab0 0x7fcb0c1608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7f1b4abda640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b4415bbd0 0x7f1b4416af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49578/0 (socket says 172.21.15.98:49578) 2023-11-28T17:41:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7f1b4abda640 1 -- 172.21.15.98:0/3408276170 learned_addr learned my addr 172.21.15.98:0/3408276170 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7f41dc866640 1 -- 172.21.15.98:0/3636381290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d806c940 msgr2=0x7f41d81567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7f41dc866640 1 --2- 172.21.15.98:0/3636381290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d806c940 0x7f41d81567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7f35d9011640 1 --2- 172.21.15.98:0/1249706644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35d41ee390 0x7f35d41ee810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7f35d9011640 1 -- 172.21.15.98:0/1249706644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35d41483e0 msgr2=0x7f35d41590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7f35d9011640 1 --2- 172.21.15.98:0/1249706644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35d41483e0 0x7f35d41590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7f35d9011640 1 -- 172.21.15.98:0/1249706644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d41512e0 con 0x7f35c0005180 2023-11-28T17:41:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.390+0000 7f35d9011640 1 --2- 172.21.15.98:0/1249706644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c0005180 0x7f35d41ec290 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f35d000d8b0 tx=0x7f35d000dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7f35c9ffb640 1 -- 172.21.15.98:0/1249706644 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d00074e0 con 0x7f35c0005180 2023-11-28T17:41:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7f35c9ffb640 1 -- 172.21.15.98:0/1249706644 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d0007640 con 0x7f35c0005180 2023-11-28T17:41:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7f35db275640 1 -- 172.21.15.98:0/1249706644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d4149dc0 con 0x7f35c0005180 2023-11-28T17:41:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7fcb0a59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb0c0b33d0 0x7fcb0c0cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7fcb09d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb0c0cf6d0 0x7fcb0c0cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7fcb09d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb0c0cf6d0 0x7fcb0c0cd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49592/0 (socket says 172.21.15.98:49592) 2023-11-28T17:41:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7fcb09d9b640 1 -- 172.21.15.98:0/2692142389 learned_addr learned my addr 172.21.15.98:0/2692142389 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7f41dc866640 1 -- 172.21.15.98:0/3636381290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d80c58e0 msgr2=0x7f41d8155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7f41dc866640 1 --2- 172.21.15.98:0/3636381290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d80c58e0 0x7f41d8155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7f1b4a3d9640 1 --2- 172.21.15.98:0/3408276170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1b440af9f0 0x7f1b4406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.392+0000 7f1b4abda640 1 -- 172.21.15.98:0/3408276170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1b440af9f0 msgr2=0x7f1b4406e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.392+0000 7f1b4abda640 1 --2- 172.21.15.98:0/3408276170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1b440af9f0 0x7f1b4406e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.392+0000 7f1b4abda640 1 -- 172.21.15.98:0/3408276170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b440c3050 msgr2=0x7f1b4416a9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T17:41:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.392+0000 7f1b4abda640 1 --2- 172.21.15.98:0/3408276170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b440c3050 0x7f1b4416a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.392+0000 7f1b4abda640 1 -- 172.21.15.98:0/3408276170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b38009b30 con 0x7f1b4415bbd0 2023-11-28T17:41:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.392+0000 7f41d7fff640 1 --2- 172.21.15.98:0/3636381290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d806c940 0x7f41d81567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.392+0000 7f41dc866640 1 -- 172.21.15.98:0/3636381290 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c8009b30 con 0x7f41d80c6720 2023-11-28T17:41:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.392+0000 7f41d77fe640 1 --2- 172.21.15.98:0/3636381290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d80c58e0 0x7f41d8155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:41:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.392+0000 7fcb09d9b640 1 -- 172.21.15.98:0/2692142389 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb0c0cfab0 msgr2=0x7fcb0c1608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.392+0000 7fcb09d9b640 1 --2- 172.21.15.98:0/2692142389 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb0c0cfab0 0x7fcb0c1608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.392+0000 7fcb09d9b640 1 -- 172.21.15.98:0/2692142389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb0c0b33d0 msgr2=0x7fcb0c0cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.392+0000 7fcb09d9b640 1 --2- 172.21.15.98:0/2692142389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb0c0b33d0 0x7fcb0c0cee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.392+0000 7fcb09d9b640 1 -- 172.21.15.98:0/2692142389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb04009a70 con 0x7fcb0c0cf6d0 2023-11-28T17:41:26.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.393+0000 7f1b4abda640 1 --2- 172.21.15.98:0/3408276170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b4415bbd0 0x7f1b4416af10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00c410 tx=0x7f1b3c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.393+0000 7fcb09d9b640 1 --2- 172.21.15.98:0/2692142389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb0c0cf6d0 0x7fcb0c0cd380 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000c430 tx=0x7fcb0000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.393+0000 7f41dc866640 1 --2- 172.21.15.98:0/3636381290 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d80c6720 0x7f41d80c3fa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009fd0 tx=0x7f41c000b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.394+0000 7f1b42ffd640 1 -- 172.21.15.98:0/3408276170 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c011790 con 0x7f1b4415bbd0 2023-11-28T17:41:26.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.394+0000 7fcafaffd640 1 -- 172.21.15.98:0/2692142389 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb000074e0 con 0x7fcb0c0cf6d0 2023-11-28T17:41:26.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.394+0000 7fa6782da640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa660005180 0x7fa670160df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.394+0000 7fa6782da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67015f8d0 0x7fa67015d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.394+0000 7fa6782da640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa67015f200 0x7fa6701eab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.394+0000 7fa6782da640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6700c9710 con 0x7fa67015f8d0 2023-11-28T17:41:26.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.394+0000 7fa6782da640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6701e7ba0 con 0x7fa67015f200 2023-11-28T17:41:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.394+0000 7fa6782da640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa670161330 con 0x7fa660005180 2023-11-28T17:41:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.394+0000 7f1b4c63d640 1 -- 172.21.15.98:0/3408276170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b44146310 con 0x7f1b4415bbd0 2023-11-28T17:41:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.394+0000 7f1b4c63d640 1 -- 172.21.15.98:0/3408276170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b440b3140 con 0x7f1b4415bbd0 2023-11-28T17:41:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.395+0000 7fa675875640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67015f8d0 0x7fa67015d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.395+0000 7fa675875640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67015f8d0 0x7fa67015d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49594/0 (socket says 172.21.15.98:49594) 2023-11-28T17:41:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.395+0000 7fa675875640 1 -- 172.21.15.98:0/3331158847 learned_addr learned my addr 172.21.15.98:0/3331158847 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.395+0000 7fa676076640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa660005180 0x7fa670160df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.395+0000 7fa676877640 1 --2- 172.21.15.98:0/3331158847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa67015f200 0x7fa6701eab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7f35c9ffb640 1 -- 172.21.15.98:0/1249706644 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d00049e0 con 0x7f35c0005180 2023-11-28T17:41:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.391+0000 7f35db275640 1 -- 172.21.15.98:0/1249706644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d41ec7d0 con 0x7f35c0005180 2023-11-28T17:41:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.392+0000 7f35db275640 1 -- 172.21.15.98:0/1249706644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d4148100 con 0x7f35c0005180 2023-11-28T17:41:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.395+0000 7f35c9ffb640 1 -- 172.21.15.98:0/1249706644 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f35d0004b40 con 0x7f35c0005180 2023-11-28T17:41:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.395+0000 7f35c9ffb640 1 --2- 172.21.15.98:0/1249706644 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f35b806b2d0 0x7f35b806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.395+0000 7f35c9ffb640 1 -- 172.21.15.98:0/1249706644 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35d0089ac0 con 0x7f35c0005180 2023-11-28T17:41:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.396+0000 7f35d8810640 1 --2- 172.21.15.98:0/1249706644 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f35b806b2d0 0x7f35b806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.396+0000 7fa676076640 1 -- 172.21.15.98:0/3331158847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa67015f200 msgr2=0x7fa6701eab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.396+0000 7fa676076640 1 --2- 172.21.15.98:0/3331158847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa67015f200 0x7fa6701eab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.396+0000 7f41d4ff9640 1 -- 172.21.15.98:0/3636381290 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c0011830 con 0x7f41d80c6720 2023-11-28T17:41:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.396+0000 7fa676076640 1 -- 172.21.15.98:0/3331158847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67015f8d0 msgr2=0x7fa67015d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.396+0000 7f41dd067640 1 -- 172.21.15.98:0/3636381290 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d8145aa0 con 0x7f41d80c6720 2023-11-28T17:41:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.396+0000 7f41d4ff9640 1 -- 172.21.15.98:0/3636381290 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c0011990 con 0x7f41d80c6720 2023-11-28T17:41:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.396+0000 7fa676076640 1 --2- 172.21.15.98:0/3331158847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67015f8d0 0x7fa67015d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.396+0000 7fcb10d43640 1 -- 172.21.15.98:0/2692142389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcafc009400 con 0x7fcb0c0cf6d0 2023-11-28T17:41:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.396+0000 7fcb10d43640 1 -- 172.21.15.98:0/2692142389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb0c0cde00 con 0x7fcb0c0cf6d0 2023-11-28T17:41:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.397+0000 7f41d4ff9640 1 -- 172.21.15.98:0/3636381290 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c0011d20 con 0x7f41d80c6720 2023-11-28T17:41:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.396+0000 7fa676076640 1 -- 172.21.15.98:0/3331158847 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6701e6c40 con 0x7fa660005180 2023-11-28T17:41:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.397+0000 7fa675875640 1 --2- 172.21.15.98:0/3331158847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67015f8d0 0x7fa67015d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:41:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.397+0000 7f1b4c63d640 1 -- 172.21.15.98:0/3408276170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b4416a310 con 0x7f1b4415bbd0 2023-11-28T17:41:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.397+0000 7fa676076640 1 --2- 172.21.15.98:0/3331158847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa660005180 0x7fa670160df0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa6640076f0 tx=0x7fa664007bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.398+0000 7f41dd067640 1 -- 172.21.15.98:0/3636381290 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d81470e0 con 0x7f41d80c6720 2023-11-28T17:41:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.398+0000 7fcb10d43640 1 -- 172.21.15.98:0/2692142389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb0c074f60 con 0x7fcb0c0cf6d0 2023-11-28T17:41:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.398+0000 7fa66effd640 1 -- 172.21.15.98:0/3331158847 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66400e8c0 con 0x7fa660005180 2023-11-28T17:41:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.398+0000 7fa66effd640 1 -- 172.21.15.98:0/3331158847 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa66400ea20 con 0x7fa660005180 2023-11-28T17:41:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.398+0000 7fa66effd640 1 -- 172.21.15.98:0/3331158847 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66400edd0 con 0x7fa660005180 2023-11-28T17:41:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.398+0000 7fa6782da640 1 -- 172.21.15.98:0/3331158847 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6701e6f20 con 0x7fa660005180 2023-11-28T17:41:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.399+0000 7fa6782da640 1 -- 172.21.15.98:0/3331158847 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6701e74e0 con 0x7fa660005180 2023-11-28T17:41:26.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.399+0000 7f35d8810640 1 --2- 172.21.15.98:0/1249706644 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f35b806b2d0 0x7f35b806d790 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f35d40ccaa0 tx=0x7f35c4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.399+0000 7f48f77fe640 1 -- 172.21.15.98:0/2273214372 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4900055580 con 0x7f490803a010 2023-11-28T17:41:26.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.400+0000 7f41ce7fc640 1 -- 172.21.15.98:0/3636381290 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d8146040 con 0x7f41d80c6720 2023-11-28T17:41:26.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.400+0000 7f41d4ff9640 1 -- 172.21.15.98:0/3636381290 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f41c001b460 con 0x7f41d80c6720 2023-11-28T17:41:26.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.401+0000 7fa6782da640 1 -- 172.21.15.98:0/3331158847 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa670067bf0 con 0x7fa660005180 2023-11-28T17:41:26.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.401+0000 7f41d4ff9640 1 --2- 172.21.15.98:0/3636381290 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f41b806b2d0 0x7f41b806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.401+0000 7f1b42ffd640 1 -- 172.21.15.98:0/3408276170 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c003800 con 0x7f1b4415bbd0 2023-11-28T17:41:26.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.401+0000 7f1b42ffd640 1 -- 172.21.15.98:0/3408276170 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c011b30 con 0x7f1b4415bbd0 2023-11-28T17:41:26.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.402+0000 7f41d4ff9640 1 -- 172.21.15.98:0/3636381290 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41c0089b90 con 0x7f41d80c6720 2023-11-28T17:41:26.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.402+0000 7f41d7fff640 1 --2- 172.21.15.98:0/3636381290 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f41b806b2d0 0x7f41b806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.403+0000 7f1b42ffd640 1 -- 172.21.15.98:0/3408276170 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f1b3c011c90 con 0x7f1b4415bbd0 2023-11-28T17:41:26.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.403+0000 7fcafaffd640 1 -- 172.21.15.98:0/2692142389 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb00007640 con 0x7fcb0c0cf6d0 2023-11-28T17:41:26.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.403+0000 7fcafaffd640 1 -- 172.21.15.98:0/2692142389 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb000049e0 con 0x7fcb0c0cf6d0 2023-11-28T17:41:26.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.404+0000 7fa66effd640 1 -- 172.21.15.98:0/3331158847 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa66401d460 con 0x7fa660005180 2023-11-28T17:41:26.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.404+0000 7f35c9ffb640 1 -- 172.21.15.98:0/1249706644 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35d0054a50 con 0x7f35c0005180 2023-11-28T17:41:26.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.405+0000 7fa66effd640 1 --2- 172.21.15.98:0/3331158847 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa65406b320 0x7fa65406d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.405+0000 7fa66effd640 1 -- 172.21.15.98:0/3331158847 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa66408a2d0 con 0x7fa660005180 2023-11-28T17:41:26.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.405+0000 7f1b42ffd640 1 --2- 172.21.15.98:0/3408276170 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1b2806b2d0 0x7f1b2806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.405+0000 7f1b42ffd640 1 -- 172.21.15.98:0/3408276170 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b3c08a330 con 0x7f1b4415bbd0 2023-11-28T17:41:26.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.406+0000 7f41d7fff640 1 --2- 172.21.15.98:0/3636381290 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f41b806b2d0 0x7f41b806d790 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009ad0 tx=0x7f41d0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.406+0000 7f1b4a3d9640 1 --2- 172.21.15.98:0/3408276170 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1b2806b2d0 0x7f1b2806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.406+0000 7fa675875640 1 --2- 172.21.15.98:0/3331158847 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa65406b320 0x7fa65406d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.407+0000 7fa675875640 1 --2- 172.21.15.98:0/3331158847 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa65406b320 0x7fa65406d7e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa6701ea700 tx=0x7fa65c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.407+0000 7f1b4a3d9640 1 --2- 172.21.15.98:0/3408276170 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1b2806b2d0 0x7f1b2806d790 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1b30009890 tx=0x7f1b30008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.408+0000 7fcafaffd640 1 -- 172.21.15.98:0/2692142389 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fcb00004c00 con 0x7fcb0c0cf6d0 2023-11-28T17:41:26.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.409+0000 7fcafaffd640 1 --2- 172.21.15.98:0/2692142389 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcaec06b2a0 0x7fcaec06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:26.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.410+0000 7fcafaffd640 1 -- 172.21.15.98:0/2692142389 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb00089ed0 con 0x7fcb0c0cf6d0 2023-11-28T17:41:26.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.410+0000 7fcb0a59c640 1 --2- 172.21.15.98:0/2692142389 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcaec06b2a0 0x7fcaec06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:26.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.410+0000 7f1b42ffd640 1 -- 172.21.15.98:0/3408276170 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b3c055280 con 0x7f1b4415bbd0 2023-11-28T17:41:26.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.411+0000 7fcb0a59c640 1 --2- 172.21.15.98:0/2692142389 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcaec06b2a0 0x7fcaec06d760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcb04009fd0 tx=0x7fcb0400b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:26.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.416+0000 7f41d4ff9640 1 -- 172.21.15.98:0/3636381290 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f41c0054850 con 0x7f41d80c6720 2023-11-28T17:41:26.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.419+0000 7fcafaffd640 1 -- 172.21.15.98:0/2692142389 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb00054e20 con 0x7fcb0c0cf6d0 2023-11-28T17:41:26.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.429+0000 7fa66effd640 1 -- 172.21.15.98:0/3331158847 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6640552d0 con 0x7fa660005180 2023-11-28T17:41:26.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.763+0000 7f918ffff640 1 -- 172.21.15.98:0/157184694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f918804ac40 con 0x7f91880e24e0 2023-11-28T17:41:26.768 INFO:teuthology.orchestra.run.smithi098.stdout:64424509446 2023-11-28T17:41:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.764+0000 7f9177fff640 1 -- 172.21.15.98:0/157184694 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f91800585d0 con 0x7f91880e24e0 2023-11-28T17:41:26.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.768+0000 7f918ffff640 1 -- 172.21.15.98:0/157184694 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f917c06b2a0 msgr2=0x7f917c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.768+0000 7f918ffff640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f917c06b2a0 0x7f917c06d760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9184036000 tx=0x7f9184008030 comp rx=0 tx=0).stop 2023-11-28T17:41:26.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.768+0000 7f918ffff640 1 -- 172.21.15.98:0/157184694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880e24e0 msgr2=0x7f91881760f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.769+0000 7f918ffff640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880e24e0 0x7f91881760f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f918000e9b0 tx=0x7f918000ee70 comp rx=0 tx=0).stop 2023-11-28T17:41:26.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.769+0000 7f918ffff640 1 -- 172.21.15.98:0/157184694 shutdown_connections 2023-11-28T17:41:26.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.769+0000 7f918ffff640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f917c06b2a0 0x7f917c06d760 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.769+0000 7f918ffff640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91880e2de0 0x7f91880db470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.769+0000 7f918ffff640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91880e28c0 0x7f91880d9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.769+0000 7f918ffff640 1 --2- 172.21.15.98:0/157184694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91880e24e0 0x7f91881760f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.769+0000 7f918ffff640 1 -- 172.21.15.98:0/157184694 >> 172.21.15.98:0/157184694 conn(0x7f9188050150 msgr2=0x7f91880509e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.770+0000 7f918ffff640 1 -- 172.21.15.98:0/157184694 shutdown_connections 2023-11-28T17:41:26.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.770+0000 7f918ffff640 1 -- 172.21.15.98:0/157184694 wait complete. 2023-11-28T17:41:26.799 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.1 2023-11-28T17:41:26.799 DEBUG:teuthology.parallel:result is None 2023-11-28T17:41:26.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.799+0000 7fcdfe38a640 1 -- 172.21.15.98:0/2360460754 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fcdd80057d0 con 0x7fcdf00550f0 2023-11-28T17:41:26.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.800+0000 7fcdf4ff9640 1 -- 172.21.15.98:0/2360460754 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcdf8099070 con 0x7fcdf00550f0 2023-11-28T17:41:26.802 INFO:teuthology.orchestra.run.smithi098.stdout:55834574853 2023-11-28T17:41:26.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.803+0000 7fcdfe38a640 1 -- 172.21.15.98:0/2360460754 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcddc06b360 msgr2=0x7fcddc06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.803+0000 7fcdfe38a640 1 --2- 172.21.15.98:0/2360460754 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcddc06b360 0x7fcddc06d820 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcde4036000 tx=0x7fcde403b040 comp rx=0 tx=0).stop 2023-11-28T17:41:26.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.804+0000 7fcdfe38a640 1 -- 172.21.15.98:0/2360460754 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdf00550f0 msgr2=0x7fcdf0047060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.804+0000 7fcdfe38a640 1 --2- 172.21.15.98:0/2360460754 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdf00550f0 0x7fcdf0047060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8068450 tx=0x7fcdf80681b0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.804+0000 7fcdfe38a640 1 -- 172.21.15.98:0/2360460754 shutdown_connections 2023-11-28T17:41:26.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.804+0000 7fcdfe38a640 1 --2- 172.21.15.98:0/2360460754 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcddc06b360 0x7fcddc06d820 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.804+0000 7fcdfe38a640 1 --2- 172.21.15.98:0/2360460754 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdf00550f0 0x7fcdf0047060 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.804+0000 7fcdfe38a640 1 --2- 172.21.15.98:0/2360460754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf004f590 0x7fcdf0184cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.804+0000 7fcdfe38a640 1 --2- 172.21.15.98:0/2360460754 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcdf003a010 0x7fcdf0048070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.804+0000 7fcdfe38a640 1 -- 172.21.15.98:0/2360460754 >> 172.21.15.98:0/2360460754 conn(0x7fcdf00545d0 msgr2=0x7fcdf004b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.804+0000 7fcdfe38a640 1 -- 172.21.15.98:0/2360460754 shutdown_connections 2023-11-28T17:41:26.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.804+0000 7fcdfe38a640 1 -- 172.21.15.98:0/2360460754 wait complete. 2023-11-28T17:41:26.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.812+0000 7fcb10d43640 1 -- 172.21.15.98:0/2692142389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fcb0c15e130 con 0x7fcb0c0cf6d0 2023-11-28T17:41:26.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.812+0000 7f49150d3640 1 -- 172.21.15.98:0/2273214372 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4908039290 con 0x7f490803a010 2023-11-28T17:41:26.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.813+0000 7fcafaffd640 1 -- 172.21.15.98:0/2692142389 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcb00058440 con 0x7fcb0c0cf6d0 2023-11-28T17:41:26.815 INFO:teuthology.orchestra.run.smithi098.stdout:64424509445 2023-11-28T17:41:26.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.813+0000 7f48f77fe640 1 -- 172.21.15.98:0/2273214372 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4900021020 con 0x7f490803a010 2023-11-28T17:41:26.815 INFO:teuthology.orchestra.run.smithi098.stdout:64424509447 2023-11-28T17:41:26.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.815+0000 7fcb10d43640 1 -- 172.21.15.98:0/2692142389 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcaec06b2a0 msgr2=0x7fcaec06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.815+0000 7fcb10d43640 1 --2- 172.21.15.98:0/2692142389 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcaec06b2a0 0x7fcaec06d760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcb04009fd0 tx=0x7fcb0400b040 comp rx=0 tx=0).stop 2023-11-28T17:41:26.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.815+0000 7fcb10d43640 1 -- 172.21.15.98:0/2692142389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb0c0cf6d0 msgr2=0x7fcb0c0cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.815+0000 7fcb10d43640 1 --2- 172.21.15.98:0/2692142389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb0c0cf6d0 0x7fcb0c0cd380 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000c430 tx=0x7fcb0000c8f0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.816+0000 7fcb10d43640 1 -- 172.21.15.98:0/2692142389 shutdown_connections 2023-11-28T17:41:26.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.816+0000 7fcb10d43640 1 --2- 172.21.15.98:0/2692142389 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb0c0cfab0 0x7fcb0c1608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.816+0000 7fcb10d43640 1 --2- 172.21.15.98:0/2692142389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb0c0cf6d0 0x7fcb0c0cd380 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.816+0000 7fcb10d43640 1 --2- 172.21.15.98:0/2692142389 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcaec06b2a0 0x7fcaec06d760 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.816+0000 7fcb10d43640 1 --2- 172.21.15.98:0/2692142389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb0c0b33d0 0x7fcb0c0cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.816+0000 7fcb10d43640 1 -- 172.21.15.98:0/2692142389 >> 172.21.15.98:0/2692142389 conn(0x7fcb0c06af10 msgr2=0x7fcb0c0cb150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.816+0000 7fcb10d43640 1 -- 172.21.15.98:0/2692142389 shutdown_connections 2023-11-28T17:41:26.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.816+0000 7f49150d3640 1 -- 172.21.15.98:0/2273214372 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f48fc06b2a0 msgr2=0x7f48fc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.816+0000 7fcb10d43640 1 -- 172.21.15.98:0/2692142389 wait complete. 2023-11-28T17:41:26.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.816+0000 7f49150d3640 1 --2- 172.21.15.98:0/2273214372 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f48fc06b2a0 0x7f48fc06d760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4904036000 tx=0x7f4904030000 comp rx=0 tx=0).stop 2023-11-28T17:41:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.816+0000 7f49150d3640 1 -- 172.21.15.98:0/2273214372 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f490803a010 msgr2=0x7f49081771d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.816+0000 7f49150d3640 1 --2- 172.21.15.98:0/2273214372 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f490803a010 0x7f49081771d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f490000e9d0 tx=0x7f490000ee90 comp rx=0 tx=0).stop 2023-11-28T17:41:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.817+0000 7f49150d3640 1 -- 172.21.15.98:0/2273214372 shutdown_connections 2023-11-28T17:41:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.817+0000 7f49150d3640 1 --2- 172.21.15.98:0/2273214372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49080550f0 0x7f4908047160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.817+0000 7f49150d3640 1 --2- 172.21.15.98:0/2273214372 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f48fc06b2a0 0x7f48fc06d760 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.817+0000 7f49150d3640 1 --2- 172.21.15.98:0/2273214372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f490804f590 0x7f4908181f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.817+0000 7f49150d3640 1 --2- 172.21.15.98:0/2273214372 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f490803a010 0x7f49081771d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.817+0000 7f49150d3640 1 -- 172.21.15.98:0/2273214372 >> 172.21.15.98:0/2273214372 conn(0x7f49080545d0 msgr2=0x7f490804c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.817+0000 7f49150d3640 1 -- 172.21.15.98:0/2273214372 shutdown_connections 2023-11-28T17:41:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.817+0000 7f49150d3640 1 -- 172.21.15.98:0/2273214372 wait complete. 2023-11-28T17:41:26.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.821+0000 7f35db275640 1 -- 172.21.15.98:0/1249706644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f35d4148100 con 0x7f35c0005180 2023-11-28T17:41:26.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.822+0000 7f1b4c63d640 1 -- 172.21.15.98:0/3408276170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1b4416a6f0 con 0x7f1b4415bbd0 2023-11-28T17:41:26.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.822+0000 7f35c9ffb640 1 -- 172.21.15.98:0/1249706644 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f35d4148100 con 0x7f35c0005180 2023-11-28T17:41:26.825 INFO:teuthology.orchestra.run.smithi098.stdout:64424509447 2023-11-28T17:41:26.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.823+0000 7f1b42ffd640 1 -- 172.21.15.98:0/3408276170 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1b4416a6f0 con 0x7f1b4415bbd0 2023-11-28T17:41:26.825 INFO:teuthology.orchestra.run.smithi098.stdout:64424509445 2023-11-28T17:41:26.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.826+0000 7f35db275640 1 -- 172.21.15.98:0/1249706644 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f35b806b2d0 msgr2=0x7f35b806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.826+0000 7f35db275640 1 --2- 172.21.15.98:0/1249706644 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f35b806b2d0 0x7f35b806d790 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f35d40ccaa0 tx=0x7f35c4008040 comp rx=0 tx=0).stop 2023-11-28T17:41:26.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.826+0000 7f35db275640 1 -- 172.21.15.98:0/1249706644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c0005180 msgr2=0x7f35d41ec290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.826+0000 7f35db275640 1 --2- 172.21.15.98:0/1249706644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c0005180 0x7f35d41ec290 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f35d000d8b0 tx=0x7f35d000dd70 comp rx=0 tx=0).stop 2023-11-28T17:41:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.827+0000 7f41ce7fc640 1 -- 172.21.15.98:0/3636381290 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f41d80c4a70 con 0x7f41d80c6720 2023-11-28T17:41:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.827+0000 7f35db275640 1 -- 172.21.15.98:0/1249706644 shutdown_connections 2023-11-28T17:41:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.827+0000 7f35db275640 1 --2- 172.21.15.98:0/1249706644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35d41ee390 0x7f35d41ee810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.827+0000 7f35db275640 1 --2- 172.21.15.98:0/1249706644 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f35b806b2d0 0x7f35b806d790 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.827+0000 7f35db275640 1 --2- 172.21.15.98:0/1249706644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35d41483e0 0x7f35d41590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.827+0000 7f35db275640 1 --2- 172.21.15.98:0/1249706644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35c0005180 0x7f35d41ec290 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.827+0000 7f35db275640 1 -- 172.21.15.98:0/1249706644 >> 172.21.15.98:0/1249706644 conn(0x7f35d406af30 msgr2=0x7f35d40c7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.827+0000 7f1b4c63d640 1 -- 172.21.15.98:0/3408276170 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1b2806b2d0 msgr2=0x7f1b2806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.827+0000 7f1b4c63d640 1 --2- 172.21.15.98:0/3408276170 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1b2806b2d0 0x7f1b2806d790 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1b30009890 tx=0x7f1b30008040 comp rx=0 tx=0).stop 2023-11-28T17:41:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.827+0000 7f1b4c63d640 1 -- 172.21.15.98:0/3408276170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b4415bbd0 msgr2=0x7f1b4416af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.827+0000 7f1b4c63d640 1 --2- 172.21.15.98:0/3408276170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b4415bbd0 0x7f1b4416af10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00c410 tx=0x7f1b3c00c8d0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.828+0000 7f35db275640 1 -- 172.21.15.98:0/1249706644 shutdown_connections 2023-11-28T17:41:26.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.828+0000 7f41d4ff9640 1 -- 172.21.15.98:0/3636381290 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f41c0057e70 con 0x7f41d80c6720 2023-11-28T17:41:26.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.828+0000 7f1b4c63d640 1 -- 172.21.15.98:0/3408276170 shutdown_connections 2023-11-28T17:41:26.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.828+0000 7f1b4c63d640 1 --2- 172.21.15.98:0/3408276170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b4415bbd0 0x7f1b4416af10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.828+0000 7f1b4c63d640 1 --2- 172.21.15.98:0/3408276170 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1b2806b2d0 0x7f1b2806d790 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.828+0000 7f1b4c63d640 1 --2- 172.21.15.98:0/3408276170 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b440c3050 0x7f1b4416a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.828+0000 7f1b4c63d640 1 --2- 172.21.15.98:0/3408276170 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1b440af9f0 0x7f1b4406e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.828+0000 7f1b4c63d640 1 -- 172.21.15.98:0/3408276170 >> 172.21.15.98:0/3408276170 conn(0x7f1b44066860 msgr2=0x7f1b44157ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.832 INFO:teuthology.orchestra.run.smithi098.stdout:64424509446 2023-11-28T17:41:26.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.830+0000 7f1b4c63d640 1 -- 172.21.15.98:0/3408276170 shutdown_connections 2023-11-28T17:41:26.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.830+0000 7f1b4c63d640 1 -- 172.21.15.98:0/3408276170 wait complete. 2023-11-28T17:41:26.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.830+0000 7f35db275640 1 -- 172.21.15.98:0/1249706644 wait complete. 2023-11-28T17:41:26.835 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.2 2023-11-28T17:41:26.835 DEBUG:teuthology.parallel:result is None 2023-11-28T17:41:26.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.831+0000 7f41ce7fc640 1 -- 172.21.15.98:0/3636381290 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f41b806b2d0 msgr2=0x7f41b806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.831+0000 7f41ce7fc640 1 --2- 172.21.15.98:0/3636381290 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f41b806b2d0 0x7f41b806d790 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009ad0 tx=0x7f41d0008040 comp rx=0 tx=0).stop 2023-11-28T17:41:26.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.831+0000 7f41ce7fc640 1 -- 172.21.15.98:0/3636381290 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d80c6720 msgr2=0x7f41d80c3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.831+0000 7f41ce7fc640 1 --2- 172.21.15.98:0/3636381290 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d80c6720 0x7f41d80c3fa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009fd0 tx=0x7f41c000b8e0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.832+0000 7f41ce7fc640 1 -- 172.21.15.98:0/3636381290 shutdown_connections 2023-11-28T17:41:26.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.832+0000 7f41ce7fc640 1 --2- 172.21.15.98:0/3636381290 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41d80c6720 0x7f41d80c3fa0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.832+0000 7f41ce7fc640 1 --2- 172.21.15.98:0/3636381290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d80c58e0 0x7f41d8155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.832+0000 7f41ce7fc640 1 --2- 172.21.15.98:0/3636381290 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f41b806b2d0 0x7f41b806d790 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.832+0000 7f41ce7fc640 1 --2- 172.21.15.98:0/3636381290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41d806c940 0x7f41d81567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.832+0000 7f41ce7fc640 1 -- 172.21.15.98:0/3636381290 >> 172.21.15.98:0/3636381290 conn(0x7f41d80cafe0 msgr2=0x7f41d80c8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.834+0000 7f41ce7fc640 1 -- 172.21.15.98:0/3636381290 shutdown_connections 2023-11-28T17:41:26.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.834+0000 7f41ce7fc640 1 -- 172.21.15.98:0/3636381290 wait complete. 2023-11-28T17:41:26.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.838+0000 7fa6782da640 1 -- 172.21.15.98:0/3331158847 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa670067bf0 con 0x7fa660005180 2023-11-28T17:41:26.840 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.6 2023-11-28T17:41:26.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.839+0000 7fa66effd640 1 -- 172.21.15.98:0/3331158847 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa6640588f0 con 0x7fa660005180 2023-11-28T17:41:26.841 DEBUG:teuthology.parallel:result is None 2023-11-28T17:41:26.841 INFO:teuthology.orchestra.run.smithi098.stdout:64424509446 2023-11-28T17:41:26.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.842+0000 7fa6782da640 1 -- 172.21.15.98:0/3331158847 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa65406b320 msgr2=0x7fa65406d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.842+0000 7fa6782da640 1 --2- 172.21.15.98:0/3331158847 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa65406b320 0x7fa65406d7e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa6701ea700 tx=0x7fa65c009040 comp rx=0 tx=0).stop 2023-11-28T17:41:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.842+0000 7fa6782da640 1 -- 172.21.15.98:0/3331158847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa660005180 msgr2=0x7fa670160df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.842+0000 7fa6782da640 1 --2- 172.21.15.98:0/3331158847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa660005180 0x7fa670160df0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa6640076f0 tx=0x7fa664007bb0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.843+0000 7fa6782da640 1 -- 172.21.15.98:0/3331158847 shutdown_connections 2023-11-28T17:41:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.843+0000 7fa6782da640 1 --2- 172.21.15.98:0/3331158847 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa65406b320 0x7fa65406d7e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.843+0000 7fa6782da640 1 --2- 172.21.15.98:0/3331158847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa67015f200 0x7fa6701eab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.843+0000 7fa6782da640 1 --2- 172.21.15.98:0/3331158847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa67015f8d0 0x7fa67015d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.843+0000 7fa6782da640 1 --2- 172.21.15.98:0/3331158847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa660005180 0x7fa670160df0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.843+0000 7fa6782da640 1 -- 172.21.15.98:0/3331158847 >> 172.21.15.98:0/3331158847 conn(0x7fa67006a4d0 msgr2=0x7fa67015c570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.843+0000 7fa6782da640 1 -- 172.21.15.98:0/3331158847 shutdown_connections 2023-11-28T17:41:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:26.843+0000 7fa6782da640 1 -- 172.21.15.98:0/3331158847 wait complete. 2023-11-28T17:41:26.848 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.0 2023-11-28T17:41:26.848 DEBUG:teuthology.parallel:result is None 2023-11-28T17:41:26.854 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.3 2023-11-28T17:41:26.854 DEBUG:teuthology.parallel:result is None 2023-11-28T17:41:26.857 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.5 2023-11-28T17:41:26.858 DEBUG:teuthology.parallel:result is None 2023-11-28T17:41:26.858 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.7 2023-11-28T17:41:26.858 DEBUG:teuthology.parallel:result is None 2023-11-28T17:41:26.863 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.4 2023-11-28T17:41:26.863 DEBUG:teuthology.parallel:result is None 2023-11-28T17:41:26.863 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-28T17:41:26.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:41:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.104+0000 7effa7031640 1 Processor -- start 2023-11-28T17:41:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.105+0000 7effa7031640 1 -- start start 2023-11-28T17:41:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.106+0000 7effa7031640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effa00c5c00 0x7effa00c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.106+0000 7effa7031640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7effa00c4a20 0x7effa00c6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.106+0000 7effa7031640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa0157850 con 0x7effa0150c00 2023-11-28T17:41:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.106+0000 7effa7031640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa00c2e10 con 0x7effa00c4a20 2023-11-28T17:41:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.106+0000 7effa7031640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa0156980 con 0x7effa00c5c00 2023-11-28T17:41:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.107+0000 7effa55ce640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7effa00c4a20 0x7effa00c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.107+0000 7eff9ffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effa00c5c00 0x7effa00c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.107+0000 7effa55ce640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7effa00c4a20 0x7effa00c6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:42874/0 (socket says 172.21.15.98:42874) 2023-11-28T17:41:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.107+0000 7eff9ffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effa00c5c00 0x7effa00c44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:55756/0 (socket says 172.21.15.98:55756) 2023-11-28T17:41:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.107+0000 7effa55ce640 1 -- 172.21.15.98:0/2915031968 learned_addr learned my addr 172.21.15.98:0/2915031968 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.108+0000 7eff9ffff640 1 -- 172.21.15.98:0/2915031968 >> v2:172.21.15.113:3300/0 conn(0x7effa00c4a20 msgr2=0x7effa00c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.108+0000 7eff9ffff640 1 --2- 172.21.15.98:0/2915031968 >> v2:172.21.15.113:3300/0 conn(0x7effa00c4a20 0x7effa00c6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.108+0000 7eff9ffff640 1 -- 172.21.15.98:0/2915031968 >> v1:172.21.15.113:6789/0 conn(0x7effa0150c00 legacy=0x7effa00c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.108+0000 7eff9ffff640 1 -- 172.21.15.98:0/2915031968 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa0156c40 con 0x7effa00c5c00 2023-11-28T17:41:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.108+0000 7effa55ce640 1 --2- 172.21.15.98:0/2915031968 >> v2:172.21.15.113:3300/0 conn(0x7effa00c4a20 0x7effa00c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:41:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.109+0000 7eff9ffff640 1 --2- 172.21.15.98:0/2915031968 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effa00c5c00 0x7effa00c44e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0097a0 tx=0x7eff8c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84f503ad4fb6eaee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.109+0000 7eff9f7fe640 1 -- 172.21.15.98:0/2915031968 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c02d040 con 0x7effa00c5c00 2023-11-28T17:41:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.109+0000 7eff9f7fe640 1 -- 172.21.15.98:0/2915031968 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8c031810 con 0x7effa00c5c00 2023-11-28T17:41:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.109+0000 7eff9f7fe640 1 -- 172.21.15.98:0/2915031968 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c031ba0 con 0x7effa00c5c00 2023-11-28T17:41:27.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.110+0000 7effa7031640 1 -- 172.21.15.98:0/2915031968 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effa00c5c00 msgr2=0x7effa00c44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:27.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.110+0000 7effa7031640 1 --2- 172.21.15.98:0/2915031968 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effa00c5c00 0x7effa00c44e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0097a0 tx=0x7eff8c02b8b0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.110+0000 7effa7031640 1 -- 172.21.15.98:0/2915031968 shutdown_connections 2023-11-28T17:41:27.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.110+0000 7effa7031640 1 --2- 172.21.15.98:0/2915031968 >> v2:172.21.15.113:3300/0 conn(0x7effa00c4a20 0x7effa00c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.110+0000 7effa7031640 1 --2- 172.21.15.98:0/2915031968 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effa00c5c00 0x7effa00c44e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.110+0000 7effa7031640 1 -- 172.21.15.98:0/2915031968 >> 172.21.15.98:0/2915031968 conn(0x7effa006a4d0 msgr2=0x7effa006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:27.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.110+0000 7effa7031640 1 -- 172.21.15.98:0/2915031968 shutdown_connections 2023-11-28T17:41:27.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.110+0000 7effa7031640 1 -- 172.21.15.98:0/2915031968 wait complete. 2023-11-28T17:41:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.112+0000 7effa7031640 1 Processor -- start 2023-11-28T17:41:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.112+0000 7effa7031640 1 -- start start 2023-11-28T17:41:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.113+0000 7effa7031640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effa00c4a20 0x7effa0148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.113+0000 7effa7031640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effa00c5c00 0x7effa0071f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.113+0000 7effa7031640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effa0150c00 0x7effa01f0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.113+0000 7effa7031640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa0149a70 con 0x7effa00c5c00 2023-11-28T17:41:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.113+0000 7effa7031640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa014a070 con 0x7effa00c4a20 2023-11-28T17:41:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.113+0000 7effa7031640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa01f2a70 con 0x7effa0150c00 2023-11-28T17:41:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.114+0000 7eff9ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effa00c5c00 0x7effa0071f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.114+0000 7eff9ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effa00c5c00 0x7effa0071f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49600/0 (socket says 172.21.15.98:49600) 2023-11-28T17:41:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.114+0000 7eff9ffff640 1 -- 172.21.15.98:0/2064198138 learned_addr learned my addr 172.21.15.98:0/2064198138 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.114+0000 7effa4dcd640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effa00c4a20 0x7effa0148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.114+0000 7effa55ce640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effa0150c00 0x7effa01f0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.114+0000 7eff9ffff640 1 -- 172.21.15.98:0/2064198138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effa0150c00 msgr2=0x7effa01f0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.114+0000 7eff9ffff640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effa0150c00 0x7effa01f0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.114+0000 7eff9ffff640 1 -- 172.21.15.98:0/2064198138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effa00c4a20 msgr2=0x7effa0148950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.114+0000 7eff9ffff640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effa00c4a20 0x7effa0148950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.114+0000 7eff9ffff640 1 -- 172.21.15.98:0/2064198138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9800bd40 con 0x7effa00c5c00 2023-11-28T17:41:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.114+0000 7effa55ce640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effa0150c00 0x7effa01f0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.114+0000 7effa4dcd640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effa00c4a20 0x7effa0148950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.114+0000 7eff9ffff640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effa00c5c00 0x7effa0071f10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7eff8c02bec0 tx=0x7eff8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:27.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.115+0000 7eff9d7fa640 1 -- 172.21.15.98:0/2064198138 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c03ad60 con 0x7effa00c5c00 2023-11-28T17:41:27.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.115+0000 7eff9d7fa640 1 -- 172.21.15.98:0/2064198138 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8c0423f0 con 0x7effa00c5c00 2023-11-28T17:41:27.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.115+0000 7effa7031640 1 -- 172.21.15.98:0/2064198138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff8c009400 con 0x7effa00c5c00 2023-11-28T17:41:27.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.115+0000 7eff9d7fa640 1 -- 172.21.15.98:0/2064198138 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0427a0 con 0x7effa00c5c00 2023-11-28T17:41:27.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.115+0000 7effa7031640 1 -- 172.21.15.98:0/2064198138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa01f3030 con 0x7effa00c5c00 2023-11-28T17:41:27.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.116+0000 7effa7031640 1 -- 172.21.15.98:0/2064198138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa0067bf0 con 0x7effa00c5c00 2023-11-28T17:41:27.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.117+0000 7eff9d7fa640 1 -- 172.21.15.98:0/2064198138 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7eff8c042900 con 0x7effa00c5c00 2023-11-28T17:41:27.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.118+0000 7eff9d7fa640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7eff8406b2a0 0x7eff8406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:27.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.118+0000 7eff9d7fa640 1 -- 172.21.15.98:0/2064198138 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff8c0abc50 con 0x7effa00c5c00 2023-11-28T17:41:27.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.118+0000 7effa4dcd640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7eff8406b2a0 0x7eff8406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:27.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.119+0000 7effa4dcd640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7eff8406b2a0 0x7eff8406d760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7eff94000f60 tx=0x7eff94009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.124+0000 7eff9d7fa640 1 -- 172.21.15.98:0/2064198138 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff8c076a20 con 0x7effa00c5c00 2023-11-28T17:41:27.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.298+0000 7effa7031640 1 -- 172.21.15.98:0/2064198138 --> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7effa01574e0 con 0x7eff8406b2a0 2023-11-28T17:41:27.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.324+0000 7eff9d7fa640 1 -- 172.21.15.98:0/2064198138 <== mgr.4098 v2:172.21.15.98:6832/2274392793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264383 (secure 0 0 0) 0x7eff94001c90 con 0x7eff8406b2a0 2023-11-28T17:41:27.329 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:27.331 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-11-28T17:41:27.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.330+0000 7effa7031640 1 -- 172.21.15.98:0/2064198138 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7eff8406b2a0 msgr2=0x7eff8406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:27.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.330+0000 7effa7031640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7eff8406b2a0 0x7eff8406d760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7eff94000f60 tx=0x7eff94009040 comp rx=0 tx=0).stop 2023-11-28T17:41:27.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.330+0000 7effa7031640 1 -- 172.21.15.98:0/2064198138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effa00c5c00 msgr2=0x7effa0071f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:27.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.330+0000 7effa7031640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effa00c5c00 0x7effa0071f10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7eff8c02bec0 tx=0x7eff8c0046c0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.330+0000 7effa7031640 1 -- 172.21.15.98:0/2064198138 shutdown_connections 2023-11-28T17:41:27.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.330+0000 7effa7031640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7effa0150c00 0x7effa01f0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.330+0000 7effa7031640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effa00c5c00 0x7effa0071f10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.330+0000 7effa7031640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7eff8406b2a0 0x7eff8406d760 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.330+0000 7effa7031640 1 --2- 172.21.15.98:0/2064198138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7effa00c4a20 0x7effa0148950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.330+0000 7effa7031640 1 -- 172.21.15.98:0/2064198138 >> 172.21.15.98:0/2064198138 conn(0x7effa006a4d0 msgr2=0x7effa014ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:27.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.330+0000 7effa7031640 1 -- 172.21.15.98:0/2064198138 shutdown_connections 2023-11-28T17:41:27.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.331+0000 7effa7031640 1 -- 172.21.15.98:0/2064198138 wait complete. 2023-11-28T17:41:27.351 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-11-28T17:41:26.238674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218048,"kb_used_data":1920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756672,"statfs":{"total":773094113280,"available":772870832128,"internally_reserved":0,"allocated":1966080,"data_stored":1797812,"data_compressed":65062,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.039744"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339217+0000","last_change":"2023-11-28T17:41:17.238145+0000","last_active":"2023-11-28T17:41:18.339217+0000","last_peered":"2023-11-28T17:41:18.339217+0000","last_clean":"2023-11-28T17:41:18.339217+0000","last_became_active":"2023-11-28T17:41:17.237776+0000","last_became_peered":"2023-11-28T17:41:17.237776+0000","last_unstale":"2023-11-28T17:41:18.339217+0000","last_undegraded":"2023-11-28T17:41:18.339217+0000","last_fullsized":"2023-11-28T17:41:18.339217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:41:06.902665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632799+0000","last_change":"2023-11-28T17:41:17.231913+0000","last_active":"2023-11-28T17:41:19.632799+0000","last_peered":"2023-11-28T17:41:19.632799+0000","last_clean":"2023-11-28T17:41:19.632799+0000","last_became_active":"2023-11-28T17:41:17.231622+0000","last_became_peered":"2023-11-28T17:41:17.231622+0000","last_unstale":"2023-11-28T17:41:19.632799+0000","last_undegraded":"2023-11-28T17:41:19.632799+0000","last_fullsized":"2023-11-28T17:41:19.632799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:06.911223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649842+0000","last_change":"2023-11-28T17:41:17.230232+0000","last_active":"2023-11-28T17:41:18.649842+0000","last_peered":"2023-11-28T17:41:18.649842+0000","last_clean":"2023-11-28T17:41:18.649842+0000","last_became_active":"2023-11-28T17:41:17.229795+0000","last_became_peered":"2023-11-28T17:41:17.229795+0000","last_unstale":"2023-11-28T17:41:18.649842+0000","last_undegraded":"2023-11-28T17:41:18.649842+0000","last_fullsized":"2023-11-28T17:41:18.649842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:09:09.507547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207139+0000","last_change":"2023-11-28T17:41:17.234948+0000","last_active":"2023-11-28T17:41:18.207139+0000","last_peered":"2023-11-28T17:41:18.207139+0000","last_clean":"2023-11-28T17:41:18.207139+0000","last_became_active":"2023-11-28T17:41:17.233674+0000","last_became_peered":"2023-11-28T17:41:17.233674+0000","last_unstale":"2023-11-28T17:41:18.207139+0000","last_undegraded":"2023-11-28T17:41:18.207139+0000","last_fullsized":"2023-11-28T17:41:18.207139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:45:10.831719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338225+0000","last_change":"2023-11-28T17:41:17.233908+0000","last_active":"2023-11-28T17:41:18.338225+0000","last_peered":"2023-11-28T17:41:18.338225+0000","last_clean":"2023-11-28T17:41:18.338225+0000","last_became_active":"2023-11-28T17:41:17.233589+0000","last_became_peered":"2023-11-28T17:41:17.233589+0000","last_unstale":"2023-11-28T17:41:18.338225+0000","last_undegraded":"2023-11-28T17:41:18.338225+0000","last_fullsized":"2023-11-28T17:41:18.338225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:21:37.379103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206516+0000","last_change":"2023-11-28T17:41:17.239186+0000","last_active":"2023-11-28T17:41:18.206516+0000","last_peered":"2023-11-28T17:41:18.206516+0000","last_clean":"2023-11-28T17:41:18.206516+0000","last_became_active":"2023-11-28T17:41:17.233898+0000","last_became_peered":"2023-11-28T17:41:17.233898+0000","last_unstale":"2023-11-28T17:41:18.206516+0000","last_undegraded":"2023-11-28T17:41:18.206516+0000","last_fullsized":"2023-11-28T17:41:18.206516+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:54.918589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338198+0000","last_change":"2023-11-28T17:41:17.227923+0000","last_active":"2023-11-28T17:41:18.338198+0000","last_peered":"2023-11-28T17:41:18.338198+0000","last_clean":"2023-11-28T17:41:18.338198+0000","last_became_active":"2023-11-28T17:41:17.227642+0000","last_became_peered":"2023-11-28T17:41:17.227642+0000","last_unstale":"2023-11-28T17:41:18.338198+0000","last_undegraded":"2023-11-28T17:41:18.338198+0000","last_fullsized":"2023-11-28T17:41:18.338198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:33:29.400533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567710+0000","last_change":"2023-11-28T17:41:17.239818+0000","last_active":"2023-11-28T17:41:18.567710+0000","last_peered":"2023-11-28T17:41:18.567710+0000","last_clean":"2023-11-28T17:41:18.567710+0000","last_became_active":"2023-11-28T17:41:17.239594+0000","last_became_peered":"2023-11-28T17:41:17.239594+0000","last_unstale":"2023-11-28T17:41:18.567710+0000","last_undegraded":"2023-11-28T17:41:18.567710+0000","last_fullsized":"2023-11-28T17:41:18.567710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:23.969471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339318+0000","last_change":"2023-11-28T17:41:17.243214+0000","last_active":"2023-11-28T17:41:18.339318+0000","last_peered":"2023-11-28T17:41:18.339318+0000","last_clean":"2023-11-28T17:41:18.339318+0000","last_became_active":"2023-11-28T17:41:17.242933+0000","last_became_peered":"2023-11-28T17:41:17.242933+0000","last_unstale":"2023-11-28T17:41:18.339318+0000","last_undegraded":"2023-11-28T17:41:18.339318+0000","last_fullsized":"2023-11-28T17:41:18.339318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:47:47.697137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633703+0000","last_change":"2023-11-28T17:41:17.242507+0000","last_active":"2023-11-28T17:41:19.633703+0000","last_peered":"2023-11-28T17:41:19.633703+0000","last_clean":"2023-11-28T17:41:19.633703+0000","last_became_active":"2023-11-28T17:41:17.242196+0000","last_became_peered":"2023-11-28T17:41:17.242196+0000","last_unstale":"2023-11-28T17:41:19.633703+0000","last_undegraded":"2023-11-28T17:41:19.633703+0000","last_fullsized":"2023-11-28T17:41:19.633703+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:06.103318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633386+0000","last_change":"2023-11-28T17:41:17.244694+0000","last_active":"2023-11-28T17:41:19.633386+0000","last_peered":"2023-11-28T17:41:19.633386+0000","last_clean":"2023-11-28T17:41:19.633386+0000","last_became_active":"2023-11-28T17:41:17.244301+0000","last_became_peered":"2023-11-28T17:41:17.244301+0000","last_unstale":"2023-11-28T17:41:19.633386+0000","last_undegraded":"2023-11-28T17:41:19.633386+0000","last_fullsized":"2023-11-28T17:41:19.633386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:40.030685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208504+0000","last_change":"2023-11-28T17:41:17.242800+0000","last_active":"2023-11-28T17:41:18.208504+0000","last_peered":"2023-11-28T17:41:18.208504+0000","last_clean":"2023-11-28T17:41:18.208504+0000","last_became_active":"2023-11-28T17:41:17.242463+0000","last_became_peered":"2023-11-28T17:41:17.242463+0000","last_unstale":"2023-11-28T17:41:18.208504+0000","last_undegraded":"2023-11-28T17:41:18.208504+0000","last_fullsized":"2023-11-28T17:41:18.208504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:57:34.802348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568335+0000","last_change":"2023-11-28T17:41:17.232485+0000","last_active":"2023-11-28T17:41:18.568335+0000","last_peered":"2023-11-28T17:41:18.568335+0000","last_clean":"2023-11-28T17:41:18.568335+0000","last_became_active":"2023-11-28T17:41:17.232132+0000","last_became_peered":"2023-11-28T17:41:17.232132+0000","last_unstale":"2023-11-28T17:41:18.568335+0000","last_undegraded":"2023-11-28T17:41:18.568335+0000","last_fullsized":"2023-11-28T17:41:18.568335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:49.292565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649479+0000","last_change":"2023-11-28T17:41:17.234514+0000","last_active":"2023-11-28T17:41:18.649479+0000","last_peered":"2023-11-28T17:41:18.649479+0000","last_clean":"2023-11-28T17:41:18.649479+0000","last_became_active":"2023-11-28T17:41:17.234154+0000","last_became_peered":"2023-11-28T17:41:17.234154+0000","last_unstale":"2023-11-28T17:41:18.649479+0000","last_undegraded":"2023-11-28T17:41:18.649479+0000","last_fullsized":"2023-11-28T17:41:18.649479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:22.430159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632129+0000","last_change":"2023-11-28T17:41:17.225905+0000","last_active":"2023-11-28T17:41:19.632129+0000","last_peered":"2023-11-28T17:41:19.632129+0000","last_clean":"2023-11-28T17:41:19.632129+0000","last_became_active":"2023-11-28T17:41:17.223533+0000","last_became_peered":"2023-11-28T17:41:17.223533+0000","last_unstale":"2023-11-28T17:41:19.632129+0000","last_undegraded":"2023-11-28T17:41:19.632129+0000","last_fullsized":"2023-11-28T17:41:19.632129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:32.314586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.205612+0000","last_change":"2023-11-28T17:41:17.236242+0000","last_active":"2023-11-28T17:41:18.205612+0000","last_peered":"2023-11-28T17:41:18.205612+0000","last_clean":"2023-11-28T17:41:18.205612+0000","last_became_active":"2023-11-28T17:41:17.235890+0000","last_became_peered":"2023-11-28T17:41:17.235890+0000","last_unstale":"2023-11-28T17:41:18.205612+0000","last_undegraded":"2023-11-28T17:41:18.205612+0000","last_fullsized":"2023-11-28T17:41:18.205612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:20:09.004702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208885+0000","last_change":"2023-11-28T17:41:17.234610+0000","last_active":"2023-11-28T17:41:18.208885+0000","last_peered":"2023-11-28T17:41:18.208885+0000","last_clean":"2023-11-28T17:41:18.208885+0000","last_became_active":"2023-11-28T17:41:17.234255+0000","last_became_peered":"2023-11-28T17:41:17.234255+0000","last_unstale":"2023-11-28T17:41:18.208885+0000","last_undegraded":"2023-11-28T17:41:18.208885+0000","last_fullsized":"2023-11-28T17:41:18.208885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:25:51.137148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.202895+0000","last_change":"2023-11-28T17:41:17.231838+0000","last_active":"2023-11-28T17:41:18.202895+0000","last_peered":"2023-11-28T17:41:18.202895+0000","last_clean":"2023-11-28T17:41:18.202895+0000","last_became_active":"2023-11-28T17:41:17.231533+0000","last_became_peered":"2023-11-28T17:41:17.231533+0000","last_unstale":"2023-11-28T17:41:18.202895+0000","last_undegraded":"2023-11-28T17:41:18.202895+0000","last_fullsized":"2023-11-28T17:41:18.202895+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:00:56.554112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339038+0000","last_change":"2023-11-28T17:41:17.238888+0000","last_active":"2023-11-28T17:41:18.339038+0000","last_peered":"2023-11-28T17:41:18.339038+0000","last_clean":"2023-11-28T17:41:18.339038+0000","last_became_active":"2023-11-28T17:41:17.236140+0000","last_became_peered":"2023-11-28T17:41:17.236140+0000","last_unstale":"2023-11-28T17:41:18.339038+0000","last_undegraded":"2023-11-28T17:41:18.339038+0000","last_fullsized":"2023-11-28T17:41:18.339038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:20:57.936710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633201+0000","last_change":"2023-11-28T17:41:17.242150+0000","last_active":"2023-11-28T17:41:19.633201+0000","last_peered":"2023-11-28T17:41:19.633201+0000","last_clean":"2023-11-28T17:41:19.633201+0000","last_became_active":"2023-11-28T17:41:17.241832+0000","last_became_peered":"2023-11-28T17:41:17.241832+0000","last_unstale":"2023-11-28T17:41:19.633201+0000","last_undegraded":"2023-11-28T17:41:19.633201+0000","last_fullsized":"2023-11-28T17:41:19.633201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:30:37.837711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649928+0000","last_change":"2023-11-28T17:41:17.233651+0000","last_active":"2023-11-28T17:41:18.649928+0000","last_peered":"2023-11-28T17:41:18.649928+0000","last_clean":"2023-11-28T17:41:18.649928+0000","last_became_active":"2023-11-28T17:41:17.233393+0000","last_became_peered":"2023-11-28T17:41:17.233393+0000","last_unstale":"2023-11-28T17:41:18.649928+0000","last_undegraded":"2023-11-28T17:41:18.649928+0000","last_fullsized":"2023-11-28T17:41:18.649928+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:52:08.152046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206688+0000","last_change":"2023-11-28T17:41:17.239696+0000","last_active":"2023-11-28T17:41:18.206688+0000","last_peered":"2023-11-28T17:41:18.206688+0000","last_clean":"2023-11-28T17:41:18.206688+0000","last_became_active":"2023-11-28T17:41:17.239449+0000","last_became_peered":"2023-11-28T17:41:17.239449+0000","last_unstale":"2023-11-28T17:41:18.206688+0000","last_undegraded":"2023-11-28T17:41:18.206688+0000","last_fullsized":"2023-11-28T17:41:18.206688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:08:43.019689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567770+0000","last_change":"2023-11-28T17:41:17.244917+0000","last_active":"2023-11-28T17:41:18.567770+0000","last_peered":"2023-11-28T17:41:18.567770+0000","last_clean":"2023-11-28T17:41:18.567770+0000","last_became_active":"2023-11-28T17:41:17.244676+0000","last_became_peered":"2023-11-28T17:41:17.244676+0000","last_unstale":"2023-11-28T17:41:18.567770+0000","last_undegraded":"2023-11-28T17:41:18.567770+0000","last_fullsized":"2023-11-28T17:41:18.567770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:08:10.818556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206063+0000","last_change":"2023-11-28T17:41:17.236488+0000","last_active":"2023-11-28T17:41:18.206063+0000","last_peered":"2023-11-28T17:41:18.206063+0000","last_clean":"2023-11-28T17:41:18.206063+0000","last_became_active":"2023-11-28T17:41:17.235554+0000","last_became_peered":"2023-11-28T17:41:17.235554+0000","last_unstale":"2023-11-28T17:41:18.206063+0000","last_undegraded":"2023-11-28T17:41:18.206063+0000","last_fullsized":"2023-11-28T17:41:18.206063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:29:35.500772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568927+0000","last_change":"2023-11-28T17:41:17.245042+0000","last_active":"2023-11-28T17:41:18.568927+0000","last_peered":"2023-11-28T17:41:18.568927+0000","last_clean":"2023-11-28T17:41:18.568927+0000","last_became_active":"2023-11-28T17:41:17.244729+0000","last_became_peered":"2023-11-28T17:41:17.244729+0000","last_unstale":"2023-11-28T17:41:18.568927+0000","last_undegraded":"2023-11-28T17:41:18.568927+0000","last_fullsized":"2023-11-28T17:41:18.568927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:57:00.728653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633914+0000","last_change":"2023-11-28T17:41:17.249367+0000","last_active":"2023-11-28T17:41:19.633914+0000","last_peered":"2023-11-28T17:41:19.633914+0000","last_clean":"2023-11-28T17:41:19.633914+0000","last_became_active":"2023-11-28T17:41:17.249093+0000","last_became_peered":"2023-11-28T17:41:17.249093+0000","last_unstale":"2023-11-28T17:41:19.633914+0000","last_undegraded":"2023-11-28T17:41:19.633914+0000","last_fullsized":"2023-11-28T17:41:19.633914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:12:41.994354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632733+0000","last_change":"2023-11-28T17:41:17.230256+0000","last_active":"2023-11-28T17:41:19.632733+0000","last_peered":"2023-11-28T17:41:19.632733+0000","last_clean":"2023-11-28T17:41:19.632733+0000","last_became_active":"2023-11-28T17:41:17.229551+0000","last_became_peered":"2023-11-28T17:41:17.229551+0000","last_unstale":"2023-11-28T17:41:19.632733+0000","last_undegraded":"2023-11-28T17:41:19.632733+0000","last_fullsized":"2023-11-28T17:41:19.632733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:54:09.421121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568751+0000","last_change":"2023-11-28T17:41:17.241672+0000","last_active":"2023-11-28T17:41:18.568751+0000","last_peered":"2023-11-28T17:41:18.568751+0000","last_clean":"2023-11-28T17:41:18.568751+0000","last_became_active":"2023-11-28T17:41:17.241366+0000","last_became_peered":"2023-11-28T17:41:17.241366+0000","last_unstale":"2023-11-28T17:41:18.568751+0000","last_undegraded":"2023-11-28T17:41:18.568751+0000","last_fullsized":"2023-11-28T17:41:18.568751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:29:01.718362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650047+0000","last_change":"2023-11-28T17:41:17.229925+0000","last_active":"2023-11-28T17:41:18.650047+0000","last_peered":"2023-11-28T17:41:18.650047+0000","last_clean":"2023-11-28T17:41:18.650047+0000","last_became_active":"2023-11-28T17:41:17.229617+0000","last_became_peered":"2023-11-28T17:41:17.229617+0000","last_unstale":"2023-11-28T17:41:18.650047+0000","last_undegraded":"2023-11-28T17:41:18.650047+0000","last_fullsized":"2023-11-28T17:41:18.650047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:40:14.646558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206950+0000","last_change":"2023-11-28T17:41:17.241933+0000","last_active":"2023-11-28T17:41:18.206950+0000","last_peered":"2023-11-28T17:41:18.206950+0000","last_clean":"2023-11-28T17:41:18.206950+0000","last_became_active":"2023-11-28T17:41:17.241634+0000","last_became_peered":"2023-11-28T17:41:17.241634+0000","last_unstale":"2023-11-28T17:41:18.206950+0000","last_undegraded":"2023-11-28T17:41:18.206950+0000","last_fullsized":"2023-11-28T17:41:18.206950+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:44:01.525920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.202444+0000","last_change":"2023-11-28T17:41:17.233872+0000","last_active":"2023-11-28T17:41:18.202444+0000","last_peered":"2023-11-28T17:41:18.202444+0000","last_clean":"2023-11-28T17:41:18.202444+0000","last_became_active":"2023-11-28T17:41:17.233586+0000","last_became_peered":"2023-11-28T17:41:17.233586+0000","last_unstale":"2023-11-28T17:41:18.202444+0000","last_undegraded":"2023-11-28T17:41:18.202444+0000","last_fullsized":"2023-11-28T17:41:18.202444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:56.041470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633270+0000","last_change":"2023-11-28T17:41:17.236164+0000","last_active":"2023-11-28T17:41:19.633270+0000","last_peered":"2023-11-28T17:41:19.633270+0000","last_clean":"2023-11-28T17:41:19.633270+0000","last_became_active":"2023-11-28T17:41:17.235821+0000","last_became_peered":"2023-11-28T17:41:17.235821+0000","last_unstale":"2023-11-28T17:41:19.633270+0000","last_undegraded":"2023-11-28T17:41:19.633270+0000","last_fullsized":"2023-11-28T17:41:19.633270+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:54:16.568394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206920+0000","last_change":"2023-11-28T17:41:17.235816+0000","last_active":"2023-11-28T17:41:18.206920+0000","last_peered":"2023-11-28T17:41:18.206920+0000","last_clean":"2023-11-28T17:41:18.206920+0000","last_became_active":"2023-11-28T17:41:17.235506+0000","last_became_peered":"2023-11-28T17:41:17.235506+0000","last_unstale":"2023-11-28T17:41:18.206920+0000","last_undegraded":"2023-11-28T17:41:18.206920+0000","last_fullsized":"2023-11-28T17:41:18.206920+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:18:00.597727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568406+0000","last_change":"2023-11-28T17:41:16.266076+0000","last_active":"2023-11-28T17:41:18.568406+0000","last_peered":"2023-11-28T17:41:18.568406+0000","last_clean":"2023-11-28T17:41:18.568406+0000","last_became_active":"2023-11-28T17:41:16.265837+0000","last_became_peered":"2023-11-28T17:41:16.265837+0000","last_unstale":"2023-11-28T17:41:18.568406+0000","last_undegraded":"2023-11-28T17:41:18.568406+0000","last_fullsized":"2023-11-28T17:41:18.568406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:01:51.477975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650250+0000","last_change":"2023-11-28T17:41:17.238915+0000","last_active":"2023-11-28T17:41:18.650250+0000","last_peered":"2023-11-28T17:41:18.650250+0000","last_clean":"2023-11-28T17:41:18.650250+0000","last_became_active":"2023-11-28T17:41:17.238581+0000","last_became_peered":"2023-11-28T17:41:17.238581+0000","last_unstale":"2023-11-28T17:41:18.650250+0000","last_undegraded":"2023-11-28T17:41:18.650250+0000","last_fullsized":"2023-11-28T17:41:18.650250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:52:31.640239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:20.275710+0000","last_change":"2023-11-28T17:41:16.261044+0000","last_active":"2023-11-28T17:41:20.275710+0000","last_peered":"2023-11-28T17:41:20.275710+0000","last_clean":"2023-11-28T17:41:20.275710+0000","last_became_active":"2023-11-28T17:41:16.260752+0000","last_became_peered":"2023-11-28T17:41:16.260752+0000","last_unstale":"2023-11-28T17:41:20.275710+0000","last_undegraded":"2023-11-28T17:41:20.275710+0000","last_fullsized":"2023-11-28T17:41:20.275710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:07:01.443140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.227924+0000","last_change":"2023-11-28T17:41:16.261095+0000","last_active":"2023-11-28T17:41:17.227924+0000","last_peered":"2023-11-28T17:41:17.227924+0000","last_clean":"2023-11-28T17:41:17.227924+0000","last_became_active":"2023-11-28T17:41:16.260793+0000","last_became_peered":"2023-11-28T17:41:16.260793+0000","last_unstale":"2023-11-28T17:41:17.227924+0000","last_undegraded":"2023-11-28T17:41:17.227924+0000","last_fullsized":"2023-11-28T17:41:17.227924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:54:16.821314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567897+0000","last_change":"2023-11-28T17:41:17.236517+0000","last_active":"2023-11-28T17:41:18.567897+0000","last_peered":"2023-11-28T17:41:18.567897+0000","last_clean":"2023-11-28T17:41:18.567897+0000","last_became_active":"2023-11-28T17:41:17.235925+0000","last_became_peered":"2023-11-28T17:41:17.235925+0000","last_unstale":"2023-11-28T17:41:18.567897+0000","last_undegraded":"2023-11-28T17:41:18.567897+0000","last_fullsized":"2023-11-28T17:41:18.567897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:45:09.817844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.205947+0000","last_change":"2023-11-28T17:41:16.258176+0000","last_active":"2023-11-28T17:41:18.205947+0000","last_peered":"2023-11-28T17:41:18.205947+0000","last_clean":"2023-11-28T17:41:18.205947+0000","last_became_active":"2023-11-28T17:41:16.257838+0000","last_became_peered":"2023-11-28T17:41:16.257838+0000","last_unstale":"2023-11-28T17:41:18.205947+0000","last_undegraded":"2023-11-28T17:41:18.205947+0000","last_fullsized":"2023-11-28T17:41:18.205947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:31:01.443661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.631980+0000","last_change":"2023-11-28T17:41:17.227823+0000","last_active":"2023-11-28T17:41:19.631980+0000","last_peered":"2023-11-28T17:41:19.631980+0000","last_clean":"2023-11-28T17:41:19.631980+0000","last_became_active":"2023-11-28T17:41:17.227545+0000","last_became_peered":"2023-11-28T17:41:17.227545+0000","last_unstale":"2023-11-28T17:41:19.631980+0000","last_undegraded":"2023-11-28T17:41:19.631980+0000","last_fullsized":"2023-11-28T17:41:19.631980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:20:33.253155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633476+0000","last_change":"2023-11-28T17:41:16.262372+0000","last_active":"2023-11-28T17:41:19.633476+0000","last_peered":"2023-11-28T17:41:19.633476+0000","last_clean":"2023-11-28T17:41:19.633476+0000","last_became_active":"2023-11-28T17:41:16.262029+0000","last_became_peered":"2023-11-28T17:41:16.262029+0000","last_unstale":"2023-11-28T17:41:19.633476+0000","last_undegraded":"2023-11-28T17:41:19.633476+0000","last_fullsized":"2023-11-28T17:41:19.633476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:46:08.491380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207822+0000","last_change":"2023-11-28T17:41:17.235646+0000","last_active":"2023-11-28T17:41:18.207822+0000","last_peered":"2023-11-28T17:41:18.207822+0000","last_clean":"2023-11-28T17:41:18.207822+0000","last_became_active":"2023-11-28T17:41:17.235330+0000","last_became_peered":"2023-11-28T17:41:17.235330+0000","last_unstale":"2023-11-28T17:41:18.207822+0000","last_undegraded":"2023-11-28T17:41:18.207822+0000","last_fullsized":"2023-11-28T17:41:18.207822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:48:12.717031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207869+0000","last_change":"2023-11-28T17:41:16.262716+0000","last_active":"2023-11-28T17:41:18.207869+0000","last_peered":"2023-11-28T17:41:18.207869+0000","last_clean":"2023-11-28T17:41:18.207869+0000","last_became_active":"2023-11-28T17:41:16.262347+0000","last_became_peered":"2023-11-28T17:41:16.262347+0000","last_unstale":"2023-11-28T17:41:18.207869+0000","last_undegraded":"2023-11-28T17:41:18.207869+0000","last_fullsized":"2023-11-28T17:41:18.207869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:50.685367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650261+0000","last_change":"2023-11-28T17:41:17.245852+0000","last_active":"2023-11-28T17:41:18.650261+0000","last_peered":"2023-11-28T17:41:18.650261+0000","last_clean":"2023-11-28T17:41:18.650261+0000","last_became_active":"2023-11-28T17:41:17.245596+0000","last_became_peered":"2023-11-28T17:41:17.245596+0000","last_unstale":"2023-11-28T17:41:18.650261+0000","last_undegraded":"2023-11-28T17:41:18.650261+0000","last_fullsized":"2023-11-28T17:41:18.650261+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:19:03.484052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.279710+0000","last_change":"2023-11-28T17:41:16.254855+0000","last_active":"2023-11-28T17:41:22.279710+0000","last_peered":"2023-11-28T17:41:22.279710+0000","last_clean":"2023-11-28T17:41:22.279710+0000","last_became_active":"2023-11-28T17:41:16.254557+0000","last_became_peered":"2023-11-28T17:41:16.254557+0000","last_unstale":"2023-11-28T17:41:22.279710+0000","last_undegraded":"2023-11-28T17:41:22.279710+0000","last_fullsized":"2023-11-28T17:41:22.279710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:55:38.565240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208613+0000","last_change":"2023-11-28T17:41:17.249020+0000","last_active":"2023-11-28T17:41:18.208613+0000","last_peered":"2023-11-28T17:41:18.208613+0000","last_clean":"2023-11-28T17:41:18.208613+0000","last_became_active":"2023-11-28T17:41:17.248345+0000","last_became_peered":"2023-11-28T17:41:17.248345+0000","last_unstale":"2023-11-28T17:41:18.208613+0000","last_undegraded":"2023-11-28T17:41:18.208613+0000","last_fullsized":"2023-11-28T17:41:18.208613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:05:09.688116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.279784+0000","last_change":"2023-11-28T17:41:16.256485+0000","last_active":"2023-11-28T17:41:22.279784+0000","last_peered":"2023-11-28T17:41:22.279784+0000","last_clean":"2023-11-28T17:41:22.279784+0000","last_became_active":"2023-11-28T17:41:16.256165+0000","last_became_peered":"2023-11-28T17:41:16.256165+0000","last_unstale":"2023-11-28T17:41:22.279784+0000","last_undegraded":"2023-11-28T17:41:22.279784+0000","last_fullsized":"2023-11-28T17:41:22.279784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:35:26.805435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338198+0000","last_change":"2023-11-28T17:41:17.230246+0000","last_active":"2023-11-28T17:41:18.338198+0000","last_peered":"2023-11-28T17:41:18.338198+0000","last_clean":"2023-11-28T17:41:18.338198+0000","last_became_active":"2023-11-28T17:41:17.229981+0000","last_became_peered":"2023-11-28T17:41:17.229981+0000","last_unstale":"2023-11-28T17:41:18.338198+0000","last_undegraded":"2023-11-28T17:41:18.338198+0000","last_fullsized":"2023-11-28T17:41:18.338198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:15:48.047641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.278649+0000","last_change":"2023-11-28T17:41:16.250894+0000","last_active":"2023-11-28T17:41:22.278649+0000","last_peered":"2023-11-28T17:41:22.278649+0000","last_clean":"2023-11-28T17:41:22.278649+0000","last_became_active":"2023-11-28T17:41:16.250547+0000","last_became_peered":"2023-11-28T17:41:16.250547+0000","last_unstale":"2023-11-28T17:41:22.278649+0000","last_undegraded":"2023-11-28T17:41:22.278649+0000","last_fullsized":"2023-11-28T17:41:22.278649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:58:47.412734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649511+0000","last_change":"2023-11-28T17:41:17.237500+0000","last_active":"2023-11-28T17:41:18.649511+0000","last_peered":"2023-11-28T17:41:18.649511+0000","last_clean":"2023-11-28T17:41:18.649511+0000","last_became_active":"2023-11-28T17:41:17.237211+0000","last_became_peered":"2023-11-28T17:41:17.237211+0000","last_unstale":"2023-11-28T17:41:18.649511+0000","last_undegraded":"2023-11-28T17:41:18.649511+0000","last_fullsized":"2023-11-28T17:41:18.649511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:13.675182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207345+0000","last_change":"2023-11-28T17:41:16.258147+0000","last_active":"2023-11-28T17:41:18.207345+0000","last_peered":"2023-11-28T17:41:18.207345+0000","last_clean":"2023-11-28T17:41:18.207345+0000","last_became_active":"2023-11-28T17:41:16.257540+0000","last_became_peered":"2023-11-28T17:41:16.257540+0000","last_unstale":"2023-11-28T17:41:18.207345+0000","last_undegraded":"2023-11-28T17:41:18.207345+0000","last_fullsized":"2023-11-28T17:41:18.207345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:51:11.394507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338684+0000","last_change":"2023-11-28T17:41:17.247768+0000","last_active":"2023-11-28T17:41:18.338684+0000","last_peered":"2023-11-28T17:41:18.338684+0000","last_clean":"2023-11-28T17:41:18.338684+0000","last_became_active":"2023-11-28T17:41:17.247475+0000","last_became_peered":"2023-11-28T17:41:17.247475+0000","last_unstale":"2023-11-28T17:41:18.338684+0000","last_undegraded":"2023-11-28T17:41:18.338684+0000","last_fullsized":"2023-11-28T17:41:18.338684+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:18.555573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632851+0000","last_change":"2023-11-28T17:41:16.262927+0000","last_active":"2023-11-28T17:41:19.632851+0000","last_peered":"2023-11-28T17:41:19.632851+0000","last_clean":"2023-11-28T17:41:19.632851+0000","last_became_active":"2023-11-28T17:41:16.262556+0000","last_became_peered":"2023-11-28T17:41:16.262556+0000","last_unstale":"2023-11-28T17:41:19.632851+0000","last_undegraded":"2023-11-28T17:41:19.632851+0000","last_fullsized":"2023-11-28T17:41:19.632851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:35:54.672978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632921+0000","last_change":"2023-11-28T17:41:17.239625+0000","last_active":"2023-11-28T17:41:19.632921+0000","last_peered":"2023-11-28T17:41:19.632921+0000","last_clean":"2023-11-28T17:41:19.632921+0000","last_became_active":"2023-11-28T17:41:17.238215+0000","last_became_peered":"2023-11-28T17:41:17.238215+0000","last_unstale":"2023-11-28T17:41:19.632921+0000","last_undegraded":"2023-11-28T17:41:19.632921+0000","last_fullsized":"2023-11-28T17:41:19.632921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:33:31.879592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.284665+0000","last_change":"2023-11-28T17:41:16.250674+0000","last_active":"2023-11-28T17:41:22.284665+0000","last_peered":"2023-11-28T17:41:22.284665+0000","last_clean":"2023-11-28T17:41:22.284665+0000","last_became_active":"2023-11-28T17:41:16.250312+0000","last_became_peered":"2023-11-28T17:41:16.250312+0000","last_unstale":"2023-11-28T17:41:22.284665+0000","last_undegraded":"2023-11-28T17:41:22.284665+0000","last_fullsized":"2023-11-28T17:41:22.284665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:04:05.342081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338315+0000","last_change":"2023-11-28T17:41:17.247675+0000","last_active":"2023-11-28T17:41:18.338315+0000","last_peered":"2023-11-28T17:41:18.338315+0000","last_clean":"2023-11-28T17:41:18.338315+0000","last_became_active":"2023-11-28T17:41:17.247346+0000","last_became_peered":"2023-11-28T17:41:17.247346+0000","last_unstale":"2023-11-28T17:41:18.338315+0000","last_undegraded":"2023-11-28T17:41:18.338315+0000","last_fullsized":"2023-11-28T17:41:18.338315+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:45:03.326583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.209094+0000","last_change":"2023-11-28T17:41:16.260893+0000","last_active":"2023-11-28T17:41:18.209094+0000","last_peered":"2023-11-28T17:41:18.209094+0000","last_clean":"2023-11-28T17:41:18.209094+0000","last_became_active":"2023-11-28T17:41:16.260637+0000","last_became_peered":"2023-11-28T17:41:16.260637+0000","last_unstale":"2023-11-28T17:41:18.209094+0000","last_undegraded":"2023-11-28T17:41:18.209094+0000","last_fullsized":"2023-11-28T17:41:18.209094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:07:24.032989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338811+0000","last_change":"2023-11-28T17:41:17.249202+0000","last_active":"2023-11-28T17:41:18.338811+0000","last_peered":"2023-11-28T17:41:18.338811+0000","last_clean":"2023-11-28T17:41:18.338811+0000","last_became_active":"2023-11-28T17:41:17.248187+0000","last_became_peered":"2023-11-28T17:41:17.248187+0000","last_unstale":"2023-11-28T17:41:18.338811+0000","last_undegraded":"2023-11-28T17:41:18.338811+0000","last_fullsized":"2023-11-28T17:41:18.338811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:23.973069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.277051+0000","last_change":"2023-11-28T17:41:16.244938+0000","last_active":"2023-11-28T17:41:22.277051+0000","last_peered":"2023-11-28T17:41:22.277051+0000","last_clean":"2023-11-28T17:41:22.277051+0000","last_became_active":"2023-11-28T17:41:16.244586+0000","last_became_peered":"2023-11-28T17:41:16.244586+0000","last_unstale":"2023-11-28T17:41:22.277051+0000","last_undegraded":"2023-11-28T17:41:22.277051+0000","last_fullsized":"2023-11-28T17:41:22.277051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:38.421966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339409+0000","last_change":"2023-11-28T17:41:17.248488+0000","last_active":"2023-11-28T17:41:18.339409+0000","last_peered":"2023-11-28T17:41:18.339409+0000","last_clean":"2023-11-28T17:41:18.339409+0000","last_became_active":"2023-11-28T17:41:17.248170+0000","last_became_peered":"2023-11-28T17:41:17.248170+0000","last_unstale":"2023-11-28T17:41:18.339409+0000","last_undegraded":"2023-11-28T17:41:18.339409+0000","last_fullsized":"2023-11-28T17:41:18.339409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:41.188370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632665+0000","last_change":"2023-11-28T17:41:16.246192+0000","last_active":"2023-11-28T17:41:19.632665+0000","last_peered":"2023-11-28T17:41:19.632665+0000","last_clean":"2023-11-28T17:41:19.632665+0000","last_became_active":"2023-11-28T17:41:16.245379+0000","last_became_peered":"2023-11-28T17:41:16.245379+0000","last_unstale":"2023-11-28T17:41:19.632665+0000","last_undegraded":"2023-11-28T17:41:19.632665+0000","last_fullsized":"2023-11-28T17:41:19.632665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:23:08.664344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207475+0000","last_change":"2023-11-28T17:41:17.234273+0000","last_active":"2023-11-28T17:41:18.207475+0000","last_peered":"2023-11-28T17:41:18.207475+0000","last_clean":"2023-11-28T17:41:18.207475+0000","last_became_active":"2023-11-28T17:41:17.233756+0000","last_became_peered":"2023-11-28T17:41:17.233756+0000","last_unstale":"2023-11-28T17:41:18.207475+0000","last_undegraded":"2023-11-28T17:41:18.207475+0000","last_fullsized":"2023-11-28T17:41:18.207475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:18:40.539329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.279280+0000","last_change":"2023-11-28T17:41:16.257952+0000","last_active":"2023-11-28T17:41:17.279280+0000","last_peered":"2023-11-28T17:41:17.279280+0000","last_clean":"2023-11-28T17:41:17.279280+0000","last_became_active":"2023-11-28T17:41:16.257659+0000","last_became_peered":"2023-11-28T17:41:16.257659+0000","last_unstale":"2023-11-28T17:41:17.279280+0000","last_undegraded":"2023-11-28T17:41:17.279280+0000","last_fullsized":"2023-11-28T17:41:17.279280+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:28:12.131592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338433+0000","last_change":"2023-11-28T17:41:17.239069+0000","last_active":"2023-11-28T17:41:18.338433+0000","last_peered":"2023-11-28T17:41:18.338433+0000","last_clean":"2023-11-28T17:41:18.338433+0000","last_became_active":"2023-11-28T17:41:17.238548+0000","last_became_peered":"2023-11-28T17:41:17.238548+0000","last_unstale":"2023-11-28T17:41:18.338433+0000","last_undegraded":"2023-11-28T17:41:18.338433+0000","last_fullsized":"2023-11-28T17:41:18.338433+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:12:56.686814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206759+0000","last_change":"2023-11-28T17:41:12.444311+0000","last_active":"2023-11-28T17:41:17.206759+0000","last_peered":"2023-11-28T17:41:17.206759+0000","last_clean":"2023-11-28T17:41:17.206759+0000","last_became_active":"2023-11-28T17:41:12.442193+0000","last_became_peered":"2023-11-28T17:41:12.442193+0000","last_unstale":"2023-11-28T17:41:17.206759+0000","last_undegraded":"2023-11-28T17:41:17.206759+0000","last_fullsized":"2023-11-28T17:41:17.206759+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:10.865936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:10.865936+0000","last_clean_scrub_stamp":"2023-11-28T17:41:10.865936+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:12:48.912392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206457+0000","last_change":"2023-11-28T17:41:16.253023+0000","last_active":"2023-11-28T17:41:18.206457+0000","last_peered":"2023-11-28T17:41:18.206457+0000","last_clean":"2023-11-28T17:41:18.206457+0000","last_became_active":"2023-11-28T17:41:16.245730+0000","last_became_peered":"2023-11-28T17:41:16.245730+0000","last_unstale":"2023-11-28T17:41:18.206457+0000","last_undegraded":"2023-11-28T17:41:18.206457+0000","last_fullsized":"2023-11-28T17:41:18.206457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:05.970917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568226+0000","last_change":"2023-11-28T17:41:17.246677+0000","last_active":"2023-11-28T17:41:18.568226+0000","last_peered":"2023-11-28T17:41:18.568226+0000","last_clean":"2023-11-28T17:41:18.568226+0000","last_became_active":"2023-11-28T17:41:17.246373+0000","last_became_peered":"2023-11-28T17:41:17.246373+0000","last_unstale":"2023-11-28T17:41:18.568226+0000","last_undegraded":"2023-11-28T17:41:18.568226+0000","last_fullsized":"2023-11-28T17:41:18.568226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:22:18.623331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206504+0000","last_change":"2023-11-28T17:41:16.272843+0000","last_active":"2023-11-28T17:41:17.206504+0000","last_peered":"2023-11-28T17:41:17.206504+0000","last_clean":"2023-11-28T17:41:17.206504+0000","last_became_active":"2023-11-28T17:41:16.272527+0000","last_became_peered":"2023-11-28T17:41:16.272527+0000","last_unstale":"2023-11-28T17:41:17.206504+0000","last_undegraded":"2023-11-28T17:41:17.206504+0000","last_fullsized":"2023-11-28T17:41:17.206504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:55:52.185953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649479+0000","last_change":"2023-11-28T17:41:17.240890+0000","last_active":"2023-11-28T17:41:18.649479+0000","last_peered":"2023-11-28T17:41:18.649479+0000","last_clean":"2023-11-28T17:41:18.649479+0000","last_became_active":"2023-11-28T17:41:17.240571+0000","last_became_peered":"2023-11-28T17:41:17.240571+0000","last_unstale":"2023-11-28T17:41:18.649479+0000","last_undegraded":"2023-11-28T17:41:18.649479+0000","last_fullsized":"2023-11-28T17:41:18.649479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:58.086322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.279241+0000","last_change":"2023-11-28T17:41:16.251925+0000","last_active":"2023-11-28T17:41:17.279241+0000","last_peered":"2023-11-28T17:41:17.279241+0000","last_clean":"2023-11-28T17:41:17.279241+0000","last_became_active":"2023-11-28T17:41:16.251482+0000","last_became_peered":"2023-11-28T17:41:16.251482+0000","last_unstale":"2023-11-28T17:41:17.279241+0000","last_undegraded":"2023-11-28T17:41:17.279241+0000","last_fullsized":"2023-11-28T17:41:17.279241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:07:09.590250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632279+0000","last_change":"2023-11-28T17:41:17.223810+0000","last_active":"2023-11-28T17:41:19.632279+0000","last_peered":"2023-11-28T17:41:19.632279+0000","last_clean":"2023-11-28T17:41:19.632279+0000","last_became_active":"2023-11-28T17:41:17.223244+0000","last_became_peered":"2023-11-28T17:41:17.223244+0000","last_unstale":"2023-11-28T17:41:19.632279+0000","last_undegraded":"2023-11-28T17:41:19.632279+0000","last_fullsized":"2023-11-28T17:41:19.632279+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:31:22.132844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207190+0000","last_change":"2023-11-28T17:41:16.252402+0000","last_active":"2023-11-28T17:41:18.207190+0000","last_peered":"2023-11-28T17:41:18.207190+0000","last_clean":"2023-11-28T17:41:18.207190+0000","last_became_active":"2023-11-28T17:41:16.252061+0000","last_became_peered":"2023-11-28T17:41:16.252061+0000","last_unstale":"2023-11-28T17:41:18.207190+0000","last_undegraded":"2023-11-28T17:41:18.207190+0000","last_fullsized":"2023-11-28T17:41:18.207190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:20:11.890036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207754+0000","last_change":"2023-11-28T17:41:17.234320+0000","last_active":"2023-11-28T17:41:18.207754+0000","last_peered":"2023-11-28T17:41:18.207754+0000","last_clean":"2023-11-28T17:41:18.207754+0000","last_became_active":"2023-11-28T17:41:17.233738+0000","last_became_peered":"2023-11-28T17:41:17.233738+0000","last_unstale":"2023-11-28T17:41:18.207754+0000","last_undegraded":"2023-11-28T17:41:18.207754+0000","last_fullsized":"2023-11-28T17:41:18.207754+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:31:10.257906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.204406+0000","last_change":"2023-11-28T17:41:16.245394+0000","last_active":"2023-11-28T17:41:17.204406+0000","last_peered":"2023-11-28T17:41:17.204406+0000","last_clean":"2023-11-28T17:41:17.204406+0000","last_became_active":"2023-11-28T17:41:16.242985+0000","last_became_peered":"2023-11-28T17:41:16.242985+0000","last_unstale":"2023-11-28T17:41:17.204406+0000","last_undegraded":"2023-11-28T17:41:17.204406+0000","last_fullsized":"2023-11-28T17:41:17.204406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:48.728960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207612+0000","last_change":"2023-11-28T17:41:17.233987+0000","last_active":"2023-11-28T17:41:18.207612+0000","last_peered":"2023-11-28T17:41:18.207612+0000","last_clean":"2023-11-28T17:41:18.207612+0000","last_became_active":"2023-11-28T17:41:17.233430+0000","last_became_peered":"2023-11-28T17:41:17.233430+0000","last_unstale":"2023-11-28T17:41:18.207612+0000","last_undegraded":"2023-11-28T17:41:18.207612+0000","last_fullsized":"2023-11-28T17:41:18.207612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:51:44.816033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.213771+0000","last_change":"2023-11-28T17:41:16.273047+0000","last_active":"2023-11-28T17:41:17.213771+0000","last_peered":"2023-11-28T17:41:17.213771+0000","last_clean":"2023-11-28T17:41:17.213771+0000","last_became_active":"2023-11-28T17:41:16.272667+0000","last_became_peered":"2023-11-28T17:41:16.272667+0000","last_unstale":"2023-11-28T17:41:17.213771+0000","last_undegraded":"2023-11-28T17:41:17.213771+0000","last_fullsized":"2023-11-28T17:41:17.213771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:19:37.472121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632439+0000","last_change":"2023-11-28T17:41:17.232465+0000","last_active":"2023-11-28T17:41:19.632439+0000","last_peered":"2023-11-28T17:41:19.632439+0000","last_clean":"2023-11-28T17:41:19.632439+0000","last_became_active":"2023-11-28T17:41:17.230496+0000","last_became_peered":"2023-11-28T17:41:17.230496+0000","last_unstale":"2023-11-28T17:41:19.632439+0000","last_undegraded":"2023-11-28T17:41:19.632439+0000","last_fullsized":"2023-11-28T17:41:19.632439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:52:08.590259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206197+0000","last_change":"2023-11-28T17:41:16.258717+0000","last_active":"2023-11-28T17:41:17.206197+0000","last_peered":"2023-11-28T17:41:17.206197+0000","last_clean":"2023-11-28T17:41:17.206197+0000","last_became_active":"2023-11-28T17:41:16.258388+0000","last_became_peered":"2023-11-28T17:41:16.258388+0000","last_unstale":"2023-11-28T17:41:17.206197+0000","last_undegraded":"2023-11-28T17:41:17.206197+0000","last_fullsized":"2023-11-28T17:41:17.206197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:17.845137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568998+0000","last_change":"2023-11-28T17:41:17.239189+0000","last_active":"2023-11-28T17:41:18.568998+0000","last_peered":"2023-11-28T17:41:18.568998+0000","last_clean":"2023-11-28T17:41:18.568998+0000","last_became_active":"2023-11-28T17:41:17.238829+0000","last_became_peered":"2023-11-28T17:41:17.238829+0000","last_unstale":"2023-11-28T17:41:18.568998+0000","last_undegraded":"2023-11-28T17:41:18.568998+0000","last_fullsized":"2023-11-28T17:41:18.568998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:10.923572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.276656+0000","last_change":"2023-11-28T17:41:16.268334+0000","last_active":"2023-11-28T17:41:22.276656+0000","last_peered":"2023-11-28T17:41:22.276656+0000","last_clean":"2023-11-28T17:41:22.276656+0000","last_became_active":"2023-11-28T17:41:16.268015+0000","last_became_peered":"2023-11-28T17:41:16.268015+0000","last_unstale":"2023-11-28T17:41:22.276656+0000","last_undegraded":"2023-11-28T17:41:22.276656+0000","last_fullsized":"2023-11-28T17:41:22.276656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:57:22.031553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650340+0000","last_change":"2023-11-28T17:41:17.225161+0000","last_active":"2023-11-28T17:41:18.650340+0000","last_peered":"2023-11-28T17:41:18.650340+0000","last_clean":"2023-11-28T17:41:18.650340+0000","last_became_active":"2023-11-28T17:41:17.224420+0000","last_became_peered":"2023-11-28T17:41:17.224420+0000","last_unstale":"2023-11-28T17:41:18.650340+0000","last_undegraded":"2023-11-28T17:41:18.650340+0000","last_fullsized":"2023-11-28T17:41:18.650340+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:25:11.426980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206160+0000","last_change":"2023-11-28T17:41:16.272884+0000","last_active":"2023-11-28T17:41:17.206160+0000","last_peered":"2023-11-28T17:41:17.206160+0000","last_clean":"2023-11-28T17:41:17.206160+0000","last_became_active":"2023-11-28T17:41:16.272557+0000","last_became_peered":"2023-11-28T17:41:16.272557+0000","last_unstale":"2023-11-28T17:41:17.206160+0000","last_undegraded":"2023-11-28T17:41:17.206160+0000","last_fullsized":"2023-11-28T17:41:17.206160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:56.634559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206149+0000","last_change":"2023-11-28T17:41:17.238145+0000","last_active":"2023-11-28T17:41:18.206149+0000","last_peered":"2023-11-28T17:41:18.206149+0000","last_clean":"2023-11-28T17:41:18.206149+0000","last_became_active":"2023-11-28T17:41:17.237774+0000","last_became_peered":"2023-11-28T17:41:17.237774+0000","last_unstale":"2023-11-28T17:41:18.206149+0000","last_undegraded":"2023-11-28T17:41:18.206149+0000","last_fullsized":"2023-11-28T17:41:18.206149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:44.723180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.204215+0000","last_change":"2023-11-28T17:41:16.262318+0000","last_active":"2023-11-28T17:41:17.204215+0000","last_peered":"2023-11-28T17:41:17.204215+0000","last_clean":"2023-11-28T17:41:17.204215+0000","last_became_active":"2023-11-28T17:41:16.262014+0000","last_became_peered":"2023-11-28T17:41:16.262014+0000","last_unstale":"2023-11-28T17:41:17.204215+0000","last_undegraded":"2023-11-28T17:41:17.204215+0000","last_fullsized":"2023-11-28T17:41:17.204215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:17:15.470393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568031+0000","last_change":"2023-11-28T17:41:17.229233+0000","last_active":"2023-11-28T17:41:18.568031+0000","last_peered":"2023-11-28T17:41:18.568031+0000","last_clean":"2023-11-28T17:41:18.568031+0000","last_became_active":"2023-11-28T17:41:17.228885+0000","last_became_peered":"2023-11-28T17:41:17.228885+0000","last_unstale":"2023-11-28T17:41:18.568031+0000","last_undegraded":"2023-11-28T17:41:18.568031+0000","last_fullsized":"2023-11-28T17:41:18.568031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:10:24.247307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.203545+0000","last_change":"2023-11-28T17:41:16.246533+0000","last_active":"2023-11-28T17:41:17.203545+0000","last_peered":"2023-11-28T17:41:17.203545+0000","last_clean":"2023-11-28T17:41:17.203545+0000","last_became_active":"2023-11-28T17:41:16.244067+0000","last_became_peered":"2023-11-28T17:41:16.244067+0000","last_unstale":"2023-11-28T17:41:17.203545+0000","last_undegraded":"2023-11-28T17:41:17.203545+0000","last_fullsized":"2023-11-28T17:41:17.203545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:02:20.862297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568303+0000","last_change":"2023-11-28T17:41:17.236159+0000","last_active":"2023-11-28T17:41:18.568303+0000","last_peered":"2023-11-28T17:41:18.568303+0000","last_clean":"2023-11-28T17:41:18.568303+0000","last_became_active":"2023-11-28T17:41:17.235700+0000","last_became_peered":"2023-11-28T17:41:17.235700+0000","last_unstale":"2023-11-28T17:41:18.568303+0000","last_undegraded":"2023-11-28T17:41:18.568303+0000","last_fullsized":"2023-11-28T17:41:18.568303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:21.900805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.271288+0000","last_change":"2023-11-28T17:41:16.268969+0000","last_active":"2023-11-28T17:41:22.271288+0000","last_peered":"2023-11-28T17:41:22.271288+0000","last_clean":"2023-11-28T17:41:22.271288+0000","last_became_active":"2023-11-28T17:41:16.268717+0000","last_became_peered":"2023-11-28T17:41:16.268717+0000","last_unstale":"2023-11-28T17:41:22.271288+0000","last_undegraded":"2023-11-28T17:41:22.271288+0000","last_fullsized":"2023-11-28T17:41:22.271288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:03:29.120624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633192+0000","last_change":"2023-11-28T17:41:17.244376+0000","last_active":"2023-11-28T17:41:19.633192+0000","last_peered":"2023-11-28T17:41:19.633192+0000","last_clean":"2023-11-28T17:41:19.633192+0000","last_became_active":"2023-11-28T17:41:17.244034+0000","last_became_peered":"2023-11-28T17:41:17.244034+0000","last_unstale":"2023-11-28T17:41:19.633192+0000","last_undegraded":"2023-11-28T17:41:19.633192+0000","last_fullsized":"2023-11-28T17:41:19.633192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:15:56.198732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208198+0000","last_change":"2023-11-28T17:41:17.233749+0000","last_active":"2023-11-28T17:41:18.208198+0000","last_peered":"2023-11-28T17:41:18.208198+0000","last_clean":"2023-11-28T17:41:18.208198+0000","last_became_active":"2023-11-28T17:41:17.232992+0000","last_became_peered":"2023-11-28T17:41:17.232992+0000","last_unstale":"2023-11-28T17:41:18.208198+0000","last_undegraded":"2023-11-28T17:41:18.208198+0000","last_fullsized":"2023-11-28T17:41:18.208198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:00:43.698244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.276885+0000","last_change":"2023-11-28T17:41:16.269465+0000","last_active":"2023-11-28T17:41:22.276885+0000","last_peered":"2023-11-28T17:41:22.276885+0000","last_clean":"2023-11-28T17:41:22.276885+0000","last_became_active":"2023-11-28T17:41:16.269147+0000","last_became_peered":"2023-11-28T17:41:16.269147+0000","last_unstale":"2023-11-28T17:41:22.276885+0000","last_undegraded":"2023-11-28T17:41:22.276885+0000","last_fullsized":"2023-11-28T17:41:22.276885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:57:58.121232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207346+0000","last_change":"2023-11-28T17:41:17.243498+0000","last_active":"2023-11-28T17:41:18.207346+0000","last_peered":"2023-11-28T17:41:18.207346+0000","last_clean":"2023-11-28T17:41:18.207346+0000","last_became_active":"2023-11-28T17:41:17.243169+0000","last_became_peered":"2023-11-28T17:41:17.243169+0000","last_unstale":"2023-11-28T17:41:18.207346+0000","last_undegraded":"2023-11-28T17:41:18.207346+0000","last_fullsized":"2023-11-28T17:41:18.207346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:11:31.509921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.275947+0000","last_change":"2023-11-28T17:41:16.270231+0000","last_active":"2023-11-28T17:41:22.275947+0000","last_peered":"2023-11-28T17:41:22.275947+0000","last_clean":"2023-11-28T17:41:22.275947+0000","last_became_active":"2023-11-28T17:41:16.269938+0000","last_became_peered":"2023-11-28T17:41:16.269938+0000","last_unstale":"2023-11-28T17:41:22.275947+0000","last_undegraded":"2023-11-28T17:41:22.275947+0000","last_fullsized":"2023-11-28T17:41:22.275947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:45.654658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650357+0000","last_change":"2023-11-28T17:41:17.245856+0000","last_active":"2023-11-28T17:41:18.650357+0000","last_peered":"2023-11-28T17:41:18.650357+0000","last_clean":"2023-11-28T17:41:18.650357+0000","last_became_active":"2023-11-28T17:41:17.245624+0000","last_became_peered":"2023-11-28T17:41:17.245624+0000","last_unstale":"2023-11-28T17:41:18.650357+0000","last_undegraded":"2023-11-28T17:41:18.650357+0000","last_fullsized":"2023-11-28T17:41:18.650357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:03:10.206506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568922+0000","last_change":"2023-11-28T17:41:16.260989+0000","last_active":"2023-11-28T17:41:18.568922+0000","last_peered":"2023-11-28T17:41:18.568922+0000","last_clean":"2023-11-28T17:41:18.568922+0000","last_became_active":"2023-11-28T17:41:16.260686+0000","last_became_peered":"2023-11-28T17:41:16.260686+0000","last_unstale":"2023-11-28T17:41:18.568922+0000","last_undegraded":"2023-11-28T17:41:18.568922+0000","last_fullsized":"2023-11-28T17:41:18.568922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:58:44.193791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568672+0000","last_change":"2023-11-28T17:41:17.240803+0000","last_active":"2023-11-28T17:41:18.568672+0000","last_peered":"2023-11-28T17:41:18.568672+0000","last_clean":"2023-11-28T17:41:18.568672+0000","last_became_active":"2023-11-28T17:41:17.240442+0000","last_became_peered":"2023-11-28T17:41:17.240442+0000","last_unstale":"2023-11-28T17:41:18.568672+0000","last_undegraded":"2023-11-28T17:41:18.568672+0000","last_fullsized":"2023-11-28T17:41:18.568672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:28:21.223788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.279104+0000","last_change":"2023-11-28T17:41:16.266520+0000","last_active":"2023-11-28T17:41:22.279104+0000","last_peered":"2023-11-28T17:41:22.279104+0000","last_clean":"2023-11-28T17:41:22.279104+0000","last_became_active":"2023-11-28T17:41:16.266245+0000","last_became_peered":"2023-11-28T17:41:16.266245+0000","last_unstale":"2023-11-28T17:41:22.279104+0000","last_undegraded":"2023-11-28T17:41:22.279104+0000","last_fullsized":"2023-11-28T17:41:22.279104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:18:28.732421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65062,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":290816,"data_stored":667482,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1060000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.73}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.804}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5109999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]}]},{"osd":6,"up_from":15,"seq":64424509445,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":78144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0110000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8340000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.278}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3279999999999998}]}]},{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":76622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.323}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4490000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5920000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":76658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.377}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.482}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.756}]}]},{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":78144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.427}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0110000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7190000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.913}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.827}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":76692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4910000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.796}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0579999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0990000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":299008,"data_stored":667538,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.909}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0369999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9930000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1869999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":76532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8080000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T17:41:27.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:41:27.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.589+0000 7f133f18a640 1 Processor -- start 2023-11-28T17:41:27.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.589+0000 7f133f18a640 1 -- start start 2023-11-28T17:41:27.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.591+0000 7f133f18a640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1338153ea0 0x7f13380cc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:27.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.591+0000 7f133f18a640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f133815d1d0 0x7f133815d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:27.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.591+0000 7f133f18a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13380cc290 con 0x7f13380c4720 2023-11-28T17:41:27.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.591+0000 7f133f18a640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f133815db80 con 0x7f133815d1d0 2023-11-28T17:41:27.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.591+0000 7f133f18a640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133815dd60 con 0x7f1338153ea0 2023-11-28T17:41:27.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.591+0000 7f1337fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f13380c4720 0x7f13380ca410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37550/0 (socket says 172.21.15.98:37550) 2023-11-28T17:41:27.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.591+0000 7f1337fff640 1 -- 172.21.15.98:0/844342716 learned_addr learned my addr 172.21.15.98:0/844342716 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:27.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f133cf26640 1 --2- 172.21.15.98:0/844342716 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1338153ea0 0x7f13380cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:27.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f133d727640 1 --2- 172.21.15.98:0/844342716 >> v2:172.21.15.113:3300/0 conn(0x7f133815d1d0 0x7f133815d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:27.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 -- 172.21.15.98:0/844342716 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588640589 0 0) 0x7f13380cc290 con 0x7f13380c4720 2023-11-28T17:41:27.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 -- 172.21.15.98:0/844342716 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1324003520 con 0x7f13380c4720 2023-11-28T17:41:27.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 -- 172.21.15.98:0/844342716 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3479914696 0 0) 0x7f1324003520 con 0x7f13380c4720 2023-11-28T17:41:27.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 -- 172.21.15.98:0/844342716 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1338153ea0 msgr2=0x7f13380cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:27.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 --2- 172.21.15.98:0/844342716 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1338153ea0 0x7f13380cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 -- 172.21.15.98:0/844342716 >> v2:172.21.15.113:3300/0 conn(0x7f133815d1d0 msgr2=0x7f133815d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:27.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 --2- 172.21.15.98:0/844342716 >> v2:172.21.15.113:3300/0 conn(0x7f133815d1d0 0x7f133815d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 -- 172.21.15.98:0/844342716 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133815e040 con 0x7f13380c4720 2023-11-28T17:41:27.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 -- 172.21.15.98:0/844342716 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f1320001670 con 0x7f13380c4720 2023-11-28T17:41:27.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 -- 172.21.15.98:0/844342716 >> v1:172.21.15.98:6789/0 conn(0x7f13380c4720 legacy=0x7f13380ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:27.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 --2- 172.21.15.98:0/844342716 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1338153ea0 0x7f1324003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:27.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 --2- 172.21.15.98:0/844342716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1324003ef0 0x7f13240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:27.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 --2- 172.21.15.98:0/844342716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1324004d00 0x7f1324005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:27.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 -- 172.21.15.98:0/844342716 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133815db80 con 0x7f1324003ef0 2023-11-28T17:41:27.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 -- 172.21.15.98:0/844342716 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133815dd60 con 0x7f1324004d00 2023-11-28T17:41:27.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f13377fe640 1 -- 172.21.15.98:0/844342716 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f132400b9d0 con 0x7f1338153ea0 2023-11-28T17:41:27.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.592+0000 7f1337fff640 1 --2- 172.21.15.98:0/844342716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1324003ef0 0x7f13240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:27.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.593+0000 7f133d727640 1 --2- 172.21.15.98:0/844342716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1324004d00 0x7f1324005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:27.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.593+0000 7f133cf26640 1 --2- 172.21.15.98:0/844342716 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1338153ea0 0x7f1324003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:27.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.593+0000 7f1337fff640 1 -- 172.21.15.98:0/844342716 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1338153ea0 msgr2=0x7f1324003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:27.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.593+0000 7f1337fff640 1 --2- 172.21.15.98:0/844342716 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1338153ea0 0x7f1324003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.593+0000 7f1337fff640 1 -- 172.21.15.98:0/844342716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1324004d00 msgr2=0x7f1324005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:27.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.593+0000 7f1337fff640 1 --2- 172.21.15.98:0/844342716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1324004d00 0x7f1324005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.593+0000 7f1337fff640 1 -- 172.21.15.98:0/844342716 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1324003520 con 0x7f1324003ef0 2023-11-28T17:41:27.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.593+0000 7f133d727640 1 --2- 172.21.15.98:0/844342716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1324004d00 0x7f1324005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:27.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.594+0000 7f1337fff640 1 --2- 172.21.15.98:0/844342716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1324003ef0 0x7f13240047c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f132000c210 tx=0x7f13240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd1543c08a06ee6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:27.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.594+0000 7f13377fe640 1 -- 172.21.15.98:0/844342716 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132002cae0 con 0x7f1324003ef0 2023-11-28T17:41:27.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.594+0000 7f13377fe640 1 -- 172.21.15.98:0/844342716 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132002cc40 con 0x7f1324003ef0 2023-11-28T17:41:27.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.594+0000 7f13377fe640 1 -- 172.21.15.98:0/844342716 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1320005af0 con 0x7f1324003ef0 2023-11-28T17:41:27.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.594+0000 7f133f18a640 1 -- 172.21.15.98:0/844342716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1324003ef0 msgr2=0x7f13240047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:27.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.594+0000 7f133f18a640 1 --2- 172.21.15.98:0/844342716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1324003ef0 0x7f13240047c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f132000c210 tx=0x7f13240057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.594+0000 7f133d727640 1 -- 172.21.15.98:0/844342716 reap_dead start 2023-11-28T17:41:27.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.595+0000 7f133f18a640 1 -- 172.21.15.98:0/844342716 shutdown_connections 2023-11-28T17:41:27.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.595+0000 7f133f18a640 1 -- 172.21.15.98:0/844342716 >> 172.21.15.98:0/844342716 conn(0x7f133806aef0 msgr2=0x7f133806b300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:27.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.595+0000 7f133f18a640 1 -- 172.21.15.98:0/844342716 shutdown_connections 2023-11-28T17:41:27.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.595+0000 7f133f18a640 1 -- 172.21.15.98:0/844342716 wait complete. 2023-11-28T17:41:27.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.596+0000 7f133f18a640 1 Processor -- start 2023-11-28T17:41:27.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.596+0000 7f133f18a640 1 -- start start 2023-11-28T17:41:27.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.597+0000 7f133f18a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1324003ef0 0x7f1338144110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:27.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.597+0000 7f133f18a640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13381476b0 0x7f133815c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:27.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.597+0000 7f133f18a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f133815c540 0x7f13381f1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:27.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.597+0000 7f133f18a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1338145560 con 0x7f1324003ef0 2023-11-28T17:41:27.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.597+0000 7f133f18a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13380cffb0 con 0x7f133815c540 2023-11-28T17:41:27.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.597+0000 7f133f18a640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13381457d0 con 0x7f13381476b0 2023-11-28T17:41:27.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.597+0000 7f133cf26640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1324003ef0 0x7f1338144110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:27.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.597+0000 7f133cf26640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1324003ef0 0x7f1338144110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49608/0 (socket says 172.21.15.98:49608) 2023-11-28T17:41:27.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.598+0000 7f133cf26640 1 -- 172.21.15.98:0/1661436057 learned_addr learned my addr 172.21.15.98:0/1661436057 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:27.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.598+0000 7f133cf26640 1 -- 172.21.15.98:0/1661436057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13381476b0 msgr2=0x7f133815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:27.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.598+0000 7f133cf26640 1 --2- 172.21.15.98:0/1661436057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13381476b0 0x7f133815c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.598+0000 7f133cf26640 1 -- 172.21.15.98:0/1661436057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f133815c540 msgr2=0x7f13381f1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:27.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.598+0000 7f133cf26640 1 --2- 172.21.15.98:0/1661436057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f133815c540 0x7f13381f1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.598+0000 7f133cf26640 1 -- 172.21.15.98:0/1661436057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1338145ab0 con 0x7f1324003ef0 2023-11-28T17:41:27.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.598+0000 7f133cf26640 1 --2- 172.21.15.98:0/1661436057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1324003ef0 0x7f1338144110 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1328007e60 tx=0x7f132800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:27.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.598+0000 7f13357fa640 1 -- 172.21.15.98:0/1661436057 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13280145f0 con 0x7f1324003ef0 2023-11-28T17:41:27.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.598+0000 7f13357fa640 1 -- 172.21.15.98:0/1661436057 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1328014750 con 0x7f1324003ef0 2023-11-28T17:41:27.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.598+0000 7f13357fa640 1 -- 172.21.15.98:0/1661436057 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328014ae0 con 0x7f1324003ef0 2023-11-28T17:41:27.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.598+0000 7f133f18a640 1 -- 172.21.15.98:0/1661436057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1338145d90 con 0x7f1324003ef0 2023-11-28T17:41:27.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.598+0000 7f133f18a640 1 -- 172.21.15.98:0/1661436057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1338147220 con 0x7f1324003ef0 2023-11-28T17:41:27.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.600+0000 7f133f18a640 1 -- 172.21.15.98:0/1661436057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1338145f20 con 0x7f1324003ef0 2023-11-28T17:41:27.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.600+0000 7f13357fa640 1 -- 172.21.15.98:0/1661436057 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f1328014c40 con 0x7f1324003ef0 2023-11-28T17:41:27.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.602+0000 7f13357fa640 1 --2- 172.21.15.98:0/1661436057 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f131806b2d0 0x7f131806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:27.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.602+0000 7f13357fa640 1 -- 172.21.15.98:0/1661436057 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f132808a8b0 con 0x7f1324003ef0 2023-11-28T17:41:27.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.602+0000 7f1337fff640 1 --2- 172.21.15.98:0/1661436057 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f131806b2d0 0x7f131806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:27.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.602+0000 7f1337fff640 1 --2- 172.21.15.98:0/1661436057 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f131806b2d0 0x7f131806d790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1320034000 tx=0x7f132002e000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:27.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.606+0000 7f13357fa640 1 -- 172.21.15.98:0/1661436057 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1328055680 con 0x7f1324003ef0 2023-11-28T17:41:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.780+0000 7f133f18a640 1 -- 172.21.15.98:0/1661436057 --> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1338145940 con 0x7f131806b2d0 2023-11-28T17:41:27.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.800+0000 7f13357fa640 1 -- 172.21.15.98:0/1661436057 <== mgr.4098 v2:172.21.15.98:6832/2274392793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264383 (secure 0 0 0) 0x7f1338145940 con 0x7f131806b2d0 2023-11-28T17:41:27.805 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:27.806 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-11-28T17:41:27.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.804+0000 7f133f18a640 1 -- 172.21.15.98:0/1661436057 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f131806b2d0 msgr2=0x7f131806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:27.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.805+0000 7f133f18a640 1 --2- 172.21.15.98:0/1661436057 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f131806b2d0 0x7f131806d790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1320034000 tx=0x7f132002e000 comp rx=0 tx=0).stop 2023-11-28T17:41:27.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.805+0000 7f133f18a640 1 -- 172.21.15.98:0/1661436057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1324003ef0 msgr2=0x7f1338144110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:27.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.805+0000 7f133f18a640 1 --2- 172.21.15.98:0/1661436057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1324003ef0 0x7f1338144110 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1328007e60 tx=0x7f132800f520 comp rx=0 tx=0).stop 2023-11-28T17:41:27.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.805+0000 7f133f18a640 1 -- 172.21.15.98:0/1661436057 shutdown_connections 2023-11-28T17:41:27.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.805+0000 7f133f18a640 1 --2- 172.21.15.98:0/1661436057 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f131806b2d0 0x7f131806d790 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.805+0000 7f133f18a640 1 --2- 172.21.15.98:0/1661436057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f133815c540 0x7f13381f1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.805+0000 7f133f18a640 1 --2- 172.21.15.98:0/1661436057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13381476b0 0x7f133815c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.805+0000 7f133f18a640 1 --2- 172.21.15.98:0/1661436057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1324003ef0 0x7f1338144110 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:27.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.805+0000 7f133f18a640 1 -- 172.21.15.98:0/1661436057 >> 172.21.15.98:0/1661436057 conn(0x7f133806aef0 msgr2=0x7f133806b300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:27.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.805+0000 7f133f18a640 1 -- 172.21.15.98:0/1661436057 shutdown_connections 2023-11-28T17:41:27.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:27.805+0000 7f133f18a640 1 -- 172.21.15.98:0/1661436057 wait complete. 2023-11-28T17:41:27.823 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-11-28T17:41:26.238674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218048,"kb_used_data":1920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756672,"statfs":{"total":773094113280,"available":772870832128,"internally_reserved":0,"allocated":1966080,"data_stored":1797812,"data_compressed":65062,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.039744"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339217+0000","last_change":"2023-11-28T17:41:17.238145+0000","last_active":"2023-11-28T17:41:18.339217+0000","last_peered":"2023-11-28T17:41:18.339217+0000","last_clean":"2023-11-28T17:41:18.339217+0000","last_became_active":"2023-11-28T17:41:17.237776+0000","last_became_peered":"2023-11-28T17:41:17.237776+0000","last_unstale":"2023-11-28T17:41:18.339217+0000","last_undegraded":"2023-11-28T17:41:18.339217+0000","last_fullsized":"2023-11-28T17:41:18.339217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:41:06.902665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632799+0000","last_change":"2023-11-28T17:41:17.231913+0000","last_active":"2023-11-28T17:41:19.632799+0000","last_peered":"2023-11-28T17:41:19.632799+0000","last_clean":"2023-11-28T17:41:19.632799+0000","last_became_active":"2023-11-28T17:41:17.231622+0000","last_became_peered":"2023-11-28T17:41:17.231622+0000","last_unstale":"2023-11-28T17:41:19.632799+0000","last_undegraded":"2023-11-28T17:41:19.632799+0000","last_fullsized":"2023-11-28T17:41:19.632799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:06.911223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649842+0000","last_change":"2023-11-28T17:41:17.230232+0000","last_active":"2023-11-28T17:41:18.649842+0000","last_peered":"2023-11-28T17:41:18.649842+0000","last_clean":"2023-11-28T17:41:18.649842+0000","last_became_active":"2023-11-28T17:41:17.229795+0000","last_became_peered":"2023-11-28T17:41:17.229795+0000","last_unstale":"2023-11-28T17:41:18.649842+0000","last_undegraded":"2023-11-28T17:41:18.649842+0000","last_fullsized":"2023-11-28T17:41:18.649842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:09:09.507547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207139+0000","last_change":"2023-11-28T17:41:17.234948+0000","last_active":"2023-11-28T17:41:18.207139+0000","last_peered":"2023-11-28T17:41:18.207139+0000","last_clean":"2023-11-28T17:41:18.207139+0000","last_became_active":"2023-11-28T17:41:17.233674+0000","last_became_peered":"2023-11-28T17:41:17.233674+0000","last_unstale":"2023-11-28T17:41:18.207139+0000","last_undegraded":"2023-11-28T17:41:18.207139+0000","last_fullsized":"2023-11-28T17:41:18.207139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:45:10.831719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338225+0000","last_change":"2023-11-28T17:41:17.233908+0000","last_active":"2023-11-28T17:41:18.338225+0000","last_peered":"2023-11-28T17:41:18.338225+0000","last_clean":"2023-11-28T17:41:18.338225+0000","last_became_active":"2023-11-28T17:41:17.233589+0000","last_became_peered":"2023-11-28T17:41:17.233589+0000","last_unstale":"2023-11-28T17:41:18.338225+0000","last_undegraded":"2023-11-28T17:41:18.338225+0000","last_fullsized":"2023-11-28T17:41:18.338225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:21:37.379103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206516+0000","last_change":"2023-11-28T17:41:17.239186+0000","last_active":"2023-11-28T17:41:18.206516+0000","last_peered":"2023-11-28T17:41:18.206516+0000","last_clean":"2023-11-28T17:41:18.206516+0000","last_became_active":"2023-11-28T17:41:17.233898+0000","last_became_peered":"2023-11-28T17:41:17.233898+0000","last_unstale":"2023-11-28T17:41:18.206516+0000","last_undegraded":"2023-11-28T17:41:18.206516+0000","last_fullsized":"2023-11-28T17:41:18.206516+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:54.918589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338198+0000","last_change":"2023-11-28T17:41:17.227923+0000","last_active":"2023-11-28T17:41:18.338198+0000","last_peered":"2023-11-28T17:41:18.338198+0000","last_clean":"2023-11-28T17:41:18.338198+0000","last_became_active":"2023-11-28T17:41:17.227642+0000","last_became_peered":"2023-11-28T17:41:17.227642+0000","last_unstale":"2023-11-28T17:41:18.338198+0000","last_undegraded":"2023-11-28T17:41:18.338198+0000","last_fullsized":"2023-11-28T17:41:18.338198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:33:29.400533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567710+0000","last_change":"2023-11-28T17:41:17.239818+0000","last_active":"2023-11-28T17:41:18.567710+0000","last_peered":"2023-11-28T17:41:18.567710+0000","last_clean":"2023-11-28T17:41:18.567710+0000","last_became_active":"2023-11-28T17:41:17.239594+0000","last_became_peered":"2023-11-28T17:41:17.239594+0000","last_unstale":"2023-11-28T17:41:18.567710+0000","last_undegraded":"2023-11-28T17:41:18.567710+0000","last_fullsized":"2023-11-28T17:41:18.567710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:23.969471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339318+0000","last_change":"2023-11-28T17:41:17.243214+0000","last_active":"2023-11-28T17:41:18.339318+0000","last_peered":"2023-11-28T17:41:18.339318+0000","last_clean":"2023-11-28T17:41:18.339318+0000","last_became_active":"2023-11-28T17:41:17.242933+0000","last_became_peered":"2023-11-28T17:41:17.242933+0000","last_unstale":"2023-11-28T17:41:18.339318+0000","last_undegraded":"2023-11-28T17:41:18.339318+0000","last_fullsized":"2023-11-28T17:41:18.339318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:47:47.697137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633703+0000","last_change":"2023-11-28T17:41:17.242507+0000","last_active":"2023-11-28T17:41:19.633703+0000","last_peered":"2023-11-28T17:41:19.633703+0000","last_clean":"2023-11-28T17:41:19.633703+0000","last_became_active":"2023-11-28T17:41:17.242196+0000","last_became_peered":"2023-11-28T17:41:17.242196+0000","last_unstale":"2023-11-28T17:41:19.633703+0000","last_undegraded":"2023-11-28T17:41:19.633703+0000","last_fullsized":"2023-11-28T17:41:19.633703+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:06.103318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633386+0000","last_change":"2023-11-28T17:41:17.244694+0000","last_active":"2023-11-28T17:41:19.633386+0000","last_peered":"2023-11-28T17:41:19.633386+0000","last_clean":"2023-11-28T17:41:19.633386+0000","last_became_active":"2023-11-28T17:41:17.244301+0000","last_became_peered":"2023-11-28T17:41:17.244301+0000","last_unstale":"2023-11-28T17:41:19.633386+0000","last_undegraded":"2023-11-28T17:41:19.633386+0000","last_fullsized":"2023-11-28T17:41:19.633386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:40.030685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208504+0000","last_change":"2023-11-28T17:41:17.242800+0000","last_active":"2023-11-28T17:41:18.208504+0000","last_peered":"2023-11-28T17:41:18.208504+0000","last_clean":"2023-11-28T17:41:18.208504+0000","last_became_active":"2023-11-28T17:41:17.242463+0000","last_became_peered":"2023-11-28T17:41:17.242463+0000","last_unstale":"2023-11-28T17:41:18.208504+0000","last_undegraded":"2023-11-28T17:41:18.208504+0000","last_fullsized":"2023-11-28T17:41:18.208504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:57:34.802348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568335+0000","last_change":"2023-11-28T17:41:17.232485+0000","last_active":"2023-11-28T17:41:18.568335+0000","last_peered":"2023-11-28T17:41:18.568335+0000","last_clean":"2023-11-28T17:41:18.568335+0000","last_became_active":"2023-11-28T17:41:17.232132+0000","last_became_peered":"2023-11-28T17:41:17.232132+0000","last_unstale":"2023-11-28T17:41:18.568335+0000","last_undegraded":"2023-11-28T17:41:18.568335+0000","last_fullsized":"2023-11-28T17:41:18.568335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:49.292565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649479+0000","last_change":"2023-11-28T17:41:17.234514+0000","last_active":"2023-11-28T17:41:18.649479+0000","last_peered":"2023-11-28T17:41:18.649479+0000","last_clean":"2023-11-28T17:41:18.649479+0000","last_became_active":"2023-11-28T17:41:17.234154+0000","last_became_peered":"2023-11-28T17:41:17.234154+0000","last_unstale":"2023-11-28T17:41:18.649479+0000","last_undegraded":"2023-11-28T17:41:18.649479+0000","last_fullsized":"2023-11-28T17:41:18.649479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:22.430159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632129+0000","last_change":"2023-11-28T17:41:17.225905+0000","last_active":"2023-11-28T17:41:19.632129+0000","last_peered":"2023-11-28T17:41:19.632129+0000","last_clean":"2023-11-28T17:41:19.632129+0000","last_became_active":"2023-11-28T17:41:17.223533+0000","last_became_peered":"2023-11-28T17:41:17.223533+0000","last_unstale":"2023-11-28T17:41:19.632129+0000","last_undegraded":"2023-11-28T17:41:19.632129+0000","last_fullsized":"2023-11-28T17:41:19.632129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:32.314586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.205612+0000","last_change":"2023-11-28T17:41:17.236242+0000","last_active":"2023-11-28T17:41:18.205612+0000","last_peered":"2023-11-28T17:41:18.205612+0000","last_clean":"2023-11-28T17:41:18.205612+0000","last_became_active":"2023-11-28T17:41:17.235890+0000","last_became_peered":"2023-11-28T17:41:17.235890+0000","last_unstale":"2023-11-28T17:41:18.205612+0000","last_undegraded":"2023-11-28T17:41:18.205612+0000","last_fullsized":"2023-11-28T17:41:18.205612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:20:09.004702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208885+0000","last_change":"2023-11-28T17:41:17.234610+0000","last_active":"2023-11-28T17:41:18.208885+0000","last_peered":"2023-11-28T17:41:18.208885+0000","last_clean":"2023-11-28T17:41:18.208885+0000","last_became_active":"2023-11-28T17:41:17.234255+0000","last_became_peered":"2023-11-28T17:41:17.234255+0000","last_unstale":"2023-11-28T17:41:18.208885+0000","last_undegraded":"2023-11-28T17:41:18.208885+0000","last_fullsized":"2023-11-28T17:41:18.208885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:25:51.137148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.202895+0000","last_change":"2023-11-28T17:41:17.231838+0000","last_active":"2023-11-28T17:41:18.202895+0000","last_peered":"2023-11-28T17:41:18.202895+0000","last_clean":"2023-11-28T17:41:18.202895+0000","last_became_active":"2023-11-28T17:41:17.231533+0000","last_became_peered":"2023-11-28T17:41:17.231533+0000","last_unstale":"2023-11-28T17:41:18.202895+0000","last_undegraded":"2023-11-28T17:41:18.202895+0000","last_fullsized":"2023-11-28T17:41:18.202895+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:00:56.554112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339038+0000","last_change":"2023-11-28T17:41:17.238888+0000","last_active":"2023-11-28T17:41:18.339038+0000","last_peered":"2023-11-28T17:41:18.339038+0000","last_clean":"2023-11-28T17:41:18.339038+0000","last_became_active":"2023-11-28T17:41:17.236140+0000","last_became_peered":"2023-11-28T17:41:17.236140+0000","last_unstale":"2023-11-28T17:41:18.339038+0000","last_undegraded":"2023-11-28T17:41:18.339038+0000","last_fullsized":"2023-11-28T17:41:18.339038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:20:57.936710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633201+0000","last_change":"2023-11-28T17:41:17.242150+0000","last_active":"2023-11-28T17:41:19.633201+0000","last_peered":"2023-11-28T17:41:19.633201+0000","last_clean":"2023-11-28T17:41:19.633201+0000","last_became_active":"2023-11-28T17:41:17.241832+0000","last_became_peered":"2023-11-28T17:41:17.241832+0000","last_unstale":"2023-11-28T17:41:19.633201+0000","last_undegraded":"2023-11-28T17:41:19.633201+0000","last_fullsized":"2023-11-28T17:41:19.633201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:30:37.837711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649928+0000","last_change":"2023-11-28T17:41:17.233651+0000","last_active":"2023-11-28T17:41:18.649928+0000","last_peered":"2023-11-28T17:41:18.649928+0000","last_clean":"2023-11-28T17:41:18.649928+0000","last_became_active":"2023-11-28T17:41:17.233393+0000","last_became_peered":"2023-11-28T17:41:17.233393+0000","last_unstale":"2023-11-28T17:41:18.649928+0000","last_undegraded":"2023-11-28T17:41:18.649928+0000","last_fullsized":"2023-11-28T17:41:18.649928+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:52:08.152046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206688+0000","last_change":"2023-11-28T17:41:17.239696+0000","last_active":"2023-11-28T17:41:18.206688+0000","last_peered":"2023-11-28T17:41:18.206688+0000","last_clean":"2023-11-28T17:41:18.206688+0000","last_became_active":"2023-11-28T17:41:17.239449+0000","last_became_peered":"2023-11-28T17:41:17.239449+0000","last_unstale":"2023-11-28T17:41:18.206688+0000","last_undegraded":"2023-11-28T17:41:18.206688+0000","last_fullsized":"2023-11-28T17:41:18.206688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:08:43.019689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567770+0000","last_change":"2023-11-28T17:41:17.244917+0000","last_active":"2023-11-28T17:41:18.567770+0000","last_peered":"2023-11-28T17:41:18.567770+0000","last_clean":"2023-11-28T17:41:18.567770+0000","last_became_active":"2023-11-28T17:41:17.244676+0000","last_became_peered":"2023-11-28T17:41:17.244676+0000","last_unstale":"2023-11-28T17:41:18.567770+0000","last_undegraded":"2023-11-28T17:41:18.567770+0000","last_fullsized":"2023-11-28T17:41:18.567770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:08:10.818556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206063+0000","last_change":"2023-11-28T17:41:17.236488+0000","last_active":"2023-11-28T17:41:18.206063+0000","last_peered":"2023-11-28T17:41:18.206063+0000","last_clean":"2023-11-28T17:41:18.206063+0000","last_became_active":"2023-11-28T17:41:17.235554+0000","last_became_peered":"2023-11-28T17:41:17.235554+0000","last_unstale":"2023-11-28T17:41:18.206063+0000","last_undegraded":"2023-11-28T17:41:18.206063+0000","last_fullsized":"2023-11-28T17:41:18.206063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:29:35.500772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568927+0000","last_change":"2023-11-28T17:41:17.245042+0000","last_active":"2023-11-28T17:41:18.568927+0000","last_peered":"2023-11-28T17:41:18.568927+0000","last_clean":"2023-11-28T17:41:18.568927+0000","last_became_active":"2023-11-28T17:41:17.244729+0000","last_became_peered":"2023-11-28T17:41:17.244729+0000","last_unstale":"2023-11-28T17:41:18.568927+0000","last_undegraded":"2023-11-28T17:41:18.568927+0000","last_fullsized":"2023-11-28T17:41:18.568927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:57:00.728653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633914+0000","last_change":"2023-11-28T17:41:17.249367+0000","last_active":"2023-11-28T17:41:19.633914+0000","last_peered":"2023-11-28T17:41:19.633914+0000","last_clean":"2023-11-28T17:41:19.633914+0000","last_became_active":"2023-11-28T17:41:17.249093+0000","last_became_peered":"2023-11-28T17:41:17.249093+0000","last_unstale":"2023-11-28T17:41:19.633914+0000","last_undegraded":"2023-11-28T17:41:19.633914+0000","last_fullsized":"2023-11-28T17:41:19.633914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:12:41.994354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632733+0000","last_change":"2023-11-28T17:41:17.230256+0000","last_active":"2023-11-28T17:41:19.632733+0000","last_peered":"2023-11-28T17:41:19.632733+0000","last_clean":"2023-11-28T17:41:19.632733+0000","last_became_active":"2023-11-28T17:41:17.229551+0000","last_became_peered":"2023-11-28T17:41:17.229551+0000","last_unstale":"2023-11-28T17:41:19.632733+0000","last_undegraded":"2023-11-28T17:41:19.632733+0000","last_fullsized":"2023-11-28T17:41:19.632733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:54:09.421121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568751+0000","last_change":"2023-11-28T17:41:17.241672+0000","last_active":"2023-11-28T17:41:18.568751+0000","last_peered":"2023-11-28T17:41:18.568751+0000","last_clean":"2023-11-28T17:41:18.568751+0000","last_became_active":"2023-11-28T17:41:17.241366+0000","last_became_peered":"2023-11-28T17:41:17.241366+0000","last_unstale":"2023-11-28T17:41:18.568751+0000","last_undegraded":"2023-11-28T17:41:18.568751+0000","last_fullsized":"2023-11-28T17:41:18.568751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:29:01.718362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650047+0000","last_change":"2023-11-28T17:41:17.229925+0000","last_active":"2023-11-28T17:41:18.650047+0000","last_peered":"2023-11-28T17:41:18.650047+0000","last_clean":"2023-11-28T17:41:18.650047+0000","last_became_active":"2023-11-28T17:41:17.229617+0000","last_became_peered":"2023-11-28T17:41:17.229617+0000","last_unstale":"2023-11-28T17:41:18.650047+0000","last_undegraded":"2023-11-28T17:41:18.650047+0000","last_fullsized":"2023-11-28T17:41:18.650047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:40:14.646558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206950+0000","last_change":"2023-11-28T17:41:17.241933+0000","last_active":"2023-11-28T17:41:18.206950+0000","last_peered":"2023-11-28T17:41:18.206950+0000","last_clean":"2023-11-28T17:41:18.206950+0000","last_became_active":"2023-11-28T17:41:17.241634+0000","last_became_peered":"2023-11-28T17:41:17.241634+0000","last_unstale":"2023-11-28T17:41:18.206950+0000","last_undegraded":"2023-11-28T17:41:18.206950+0000","last_fullsized":"2023-11-28T17:41:18.206950+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:44:01.525920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.202444+0000","last_change":"2023-11-28T17:41:17.233872+0000","last_active":"2023-11-28T17:41:18.202444+0000","last_peered":"2023-11-28T17:41:18.202444+0000","last_clean":"2023-11-28T17:41:18.202444+0000","last_became_active":"2023-11-28T17:41:17.233586+0000","last_became_peered":"2023-11-28T17:41:17.233586+0000","last_unstale":"2023-11-28T17:41:18.202444+0000","last_undegraded":"2023-11-28T17:41:18.202444+0000","last_fullsized":"2023-11-28T17:41:18.202444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:56.041470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633270+0000","last_change":"2023-11-28T17:41:17.236164+0000","last_active":"2023-11-28T17:41:19.633270+0000","last_peered":"2023-11-28T17:41:19.633270+0000","last_clean":"2023-11-28T17:41:19.633270+0000","last_became_active":"2023-11-28T17:41:17.235821+0000","last_became_peered":"2023-11-28T17:41:17.235821+0000","last_unstale":"2023-11-28T17:41:19.633270+0000","last_undegraded":"2023-11-28T17:41:19.633270+0000","last_fullsized":"2023-11-28T17:41:19.633270+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:54:16.568394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206920+0000","last_change":"2023-11-28T17:41:17.235816+0000","last_active":"2023-11-28T17:41:18.206920+0000","last_peered":"2023-11-28T17:41:18.206920+0000","last_clean":"2023-11-28T17:41:18.206920+0000","last_became_active":"2023-11-28T17:41:17.235506+0000","last_became_peered":"2023-11-28T17:41:17.235506+0000","last_unstale":"2023-11-28T17:41:18.206920+0000","last_undegraded":"2023-11-28T17:41:18.206920+0000","last_fullsized":"2023-11-28T17:41:18.206920+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:18:00.597727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568406+0000","last_change":"2023-11-28T17:41:16.266076+0000","last_active":"2023-11-28T17:41:18.568406+0000","last_peered":"2023-11-28T17:41:18.568406+0000","last_clean":"2023-11-28T17:41:18.568406+0000","last_became_active":"2023-11-28T17:41:16.265837+0000","last_became_peered":"2023-11-28T17:41:16.265837+0000","last_unstale":"2023-11-28T17:41:18.568406+0000","last_undegraded":"2023-11-28T17:41:18.568406+0000","last_fullsized":"2023-11-28T17:41:18.568406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:01:51.477975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650250+0000","last_change":"2023-11-28T17:41:17.238915+0000","last_active":"2023-11-28T17:41:18.650250+0000","last_peered":"2023-11-28T17:41:18.650250+0000","last_clean":"2023-11-28T17:41:18.650250+0000","last_became_active":"2023-11-28T17:41:17.238581+0000","last_became_peered":"2023-11-28T17:41:17.238581+0000","last_unstale":"2023-11-28T17:41:18.650250+0000","last_undegraded":"2023-11-28T17:41:18.650250+0000","last_fullsized":"2023-11-28T17:41:18.650250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:52:31.640239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:20.275710+0000","last_change":"2023-11-28T17:41:16.261044+0000","last_active":"2023-11-28T17:41:20.275710+0000","last_peered":"2023-11-28T17:41:20.275710+0000","last_clean":"2023-11-28T17:41:20.275710+0000","last_became_active":"2023-11-28T17:41:16.260752+0000","last_became_peered":"2023-11-28T17:41:16.260752+0000","last_unstale":"2023-11-28T17:41:20.275710+0000","last_undegraded":"2023-11-28T17:41:20.275710+0000","last_fullsized":"2023-11-28T17:41:20.275710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:07:01.443140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.227924+0000","last_change":"2023-11-28T17:41:16.261095+0000","last_active":"2023-11-28T17:41:17.227924+0000","last_peered":"2023-11-28T17:41:17.227924+0000","last_clean":"2023-11-28T17:41:17.227924+0000","last_became_active":"2023-11-28T17:41:16.260793+0000","last_became_peered":"2023-11-28T17:41:16.260793+0000","last_unstale":"2023-11-28T17:41:17.227924+0000","last_undegraded":"2023-11-28T17:41:17.227924+0000","last_fullsized":"2023-11-28T17:41:17.227924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:54:16.821314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567897+0000","last_change":"2023-11-28T17:41:17.236517+0000","last_active":"2023-11-28T17:41:18.567897+0000","last_peered":"2023-11-28T17:41:18.567897+0000","last_clean":"2023-11-28T17:41:18.567897+0000","last_became_active":"2023-11-28T17:41:17.235925+0000","last_became_peered":"2023-11-28T17:41:17.235925+0000","last_unstale":"2023-11-28T17:41:18.567897+0000","last_undegraded":"2023-11-28T17:41:18.567897+0000","last_fullsized":"2023-11-28T17:41:18.567897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:45:09.817844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.205947+0000","last_change":"2023-11-28T17:41:16.258176+0000","last_active":"2023-11-28T17:41:18.205947+0000","last_peered":"2023-11-28T17:41:18.205947+0000","last_clean":"2023-11-28T17:41:18.205947+0000","last_became_active":"2023-11-28T17:41:16.257838+0000","last_became_peered":"2023-11-28T17:41:16.257838+0000","last_unstale":"2023-11-28T17:41:18.205947+0000","last_undegraded":"2023-11-28T17:41:18.205947+0000","last_fullsized":"2023-11-28T17:41:18.205947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:31:01.443661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.631980+0000","last_change":"2023-11-28T17:41:17.227823+0000","last_active":"2023-11-28T17:41:19.631980+0000","last_peered":"2023-11-28T17:41:19.631980+0000","last_clean":"2023-11-28T17:41:19.631980+0000","last_became_active":"2023-11-28T17:41:17.227545+0000","last_became_peered":"2023-11-28T17:41:17.227545+0000","last_unstale":"2023-11-28T17:41:19.631980+0000","last_undegraded":"2023-11-28T17:41:19.631980+0000","last_fullsized":"2023-11-28T17:41:19.631980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:20:33.253155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633476+0000","last_change":"2023-11-28T17:41:16.262372+0000","last_active":"2023-11-28T17:41:19.633476+0000","last_peered":"2023-11-28T17:41:19.633476+0000","last_clean":"2023-11-28T17:41:19.633476+0000","last_became_active":"2023-11-28T17:41:16.262029+0000","last_became_peered":"2023-11-28T17:41:16.262029+0000","last_unstale":"2023-11-28T17:41:19.633476+0000","last_undegraded":"2023-11-28T17:41:19.633476+0000","last_fullsized":"2023-11-28T17:41:19.633476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:46:08.491380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207822+0000","last_change":"2023-11-28T17:41:17.235646+0000","last_active":"2023-11-28T17:41:18.207822+0000","last_peered":"2023-11-28T17:41:18.207822+0000","last_clean":"2023-11-28T17:41:18.207822+0000","last_became_active":"2023-11-28T17:41:17.235330+0000","last_became_peered":"2023-11-28T17:41:17.235330+0000","last_unstale":"2023-11-28T17:41:18.207822+0000","last_undegraded":"2023-11-28T17:41:18.207822+0000","last_fullsized":"2023-11-28T17:41:18.207822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:48:12.717031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207869+0000","last_change":"2023-11-28T17:41:16.262716+0000","last_active":"2023-11-28T17:41:18.207869+0000","last_peered":"2023-11-28T17:41:18.207869+0000","last_clean":"2023-11-28T17:41:18.207869+0000","last_became_active":"2023-11-28T17:41:16.262347+0000","last_became_peered":"2023-11-28T17:41:16.262347+0000","last_unstale":"2023-11-28T17:41:18.207869+0000","last_undegraded":"2023-11-28T17:41:18.207869+0000","last_fullsized":"2023-11-28T17:41:18.207869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:50.685367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650261+0000","last_change":"2023-11-28T17:41:17.245852+0000","last_active":"2023-11-28T17:41:18.650261+0000","last_peered":"2023-11-28T17:41:18.650261+0000","last_clean":"2023-11-28T17:41:18.650261+0000","last_became_active":"2023-11-28T17:41:17.245596+0000","last_became_peered":"2023-11-28T17:41:17.245596+0000","last_unstale":"2023-11-28T17:41:18.650261+0000","last_undegraded":"2023-11-28T17:41:18.650261+0000","last_fullsized":"2023-11-28T17:41:18.650261+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:19:03.484052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.279710+0000","last_change":"2023-11-28T17:41:16.254855+0000","last_active":"2023-11-28T17:41:22.279710+0000","last_peered":"2023-11-28T17:41:22.279710+0000","last_clean":"2023-11-28T17:41:22.279710+0000","last_became_active":"2023-11-28T17:41:16.254557+0000","last_became_peered":"2023-11-28T17:41:16.254557+0000","last_unstale":"2023-11-28T17:41:22.279710+0000","last_undegraded":"2023-11-28T17:41:22.279710+0000","last_fullsized":"2023-11-28T17:41:22.279710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:55:38.565240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208613+0000","last_change":"2023-11-28T17:41:17.249020+0000","last_active":"2023-11-28T17:41:18.208613+0000","last_peered":"2023-11-28T17:41:18.208613+0000","last_clean":"2023-11-28T17:41:18.208613+0000","last_became_active":"2023-11-28T17:41:17.248345+0000","last_became_peered":"2023-11-28T17:41:17.248345+0000","last_unstale":"2023-11-28T17:41:18.208613+0000","last_undegraded":"2023-11-28T17:41:18.208613+0000","last_fullsized":"2023-11-28T17:41:18.208613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:05:09.688116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.279784+0000","last_change":"2023-11-28T17:41:16.256485+0000","last_active":"2023-11-28T17:41:22.279784+0000","last_peered":"2023-11-28T17:41:22.279784+0000","last_clean":"2023-11-28T17:41:22.279784+0000","last_became_active":"2023-11-28T17:41:16.256165+0000","last_became_peered":"2023-11-28T17:41:16.256165+0000","last_unstale":"2023-11-28T17:41:22.279784+0000","last_undegraded":"2023-11-28T17:41:22.279784+0000","last_fullsized":"2023-11-28T17:41:22.279784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:35:26.805435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338198+0000","last_change":"2023-11-28T17:41:17.230246+0000","last_active":"2023-11-28T17:41:18.338198+0000","last_peered":"2023-11-28T17:41:18.338198+0000","last_clean":"2023-11-28T17:41:18.338198+0000","last_became_active":"2023-11-28T17:41:17.229981+0000","last_became_peered":"2023-11-28T17:41:17.229981+0000","last_unstale":"2023-11-28T17:41:18.338198+0000","last_undegraded":"2023-11-28T17:41:18.338198+0000","last_fullsized":"2023-11-28T17:41:18.338198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:15:48.047641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.278649+0000","last_change":"2023-11-28T17:41:16.250894+0000","last_active":"2023-11-28T17:41:22.278649+0000","last_peered":"2023-11-28T17:41:22.278649+0000","last_clean":"2023-11-28T17:41:22.278649+0000","last_became_active":"2023-11-28T17:41:16.250547+0000","last_became_peered":"2023-11-28T17:41:16.250547+0000","last_unstale":"2023-11-28T17:41:22.278649+0000","last_undegraded":"2023-11-28T17:41:22.278649+0000","last_fullsized":"2023-11-28T17:41:22.278649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:58:47.412734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649511+0000","last_change":"2023-11-28T17:41:17.237500+0000","last_active":"2023-11-28T17:41:18.649511+0000","last_peered":"2023-11-28T17:41:18.649511+0000","last_clean":"2023-11-28T17:41:18.649511+0000","last_became_active":"2023-11-28T17:41:17.237211+0000","last_became_peered":"2023-11-28T17:41:17.237211+0000","last_unstale":"2023-11-28T17:41:18.649511+0000","last_undegraded":"2023-11-28T17:41:18.649511+0000","last_fullsized":"2023-11-28T17:41:18.649511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:13.675182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207345+0000","last_change":"2023-11-28T17:41:16.258147+0000","last_active":"2023-11-28T17:41:18.207345+0000","last_peered":"2023-11-28T17:41:18.207345+0000","last_clean":"2023-11-28T17:41:18.207345+0000","last_became_active":"2023-11-28T17:41:16.257540+0000","last_became_peered":"2023-11-28T17:41:16.257540+0000","last_unstale":"2023-11-28T17:41:18.207345+0000","last_undegraded":"2023-11-28T17:41:18.207345+0000","last_fullsized":"2023-11-28T17:41:18.207345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:51:11.394507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338684+0000","last_change":"2023-11-28T17:41:17.247768+0000","last_active":"2023-11-28T17:41:18.338684+0000","last_peered":"2023-11-28T17:41:18.338684+0000","last_clean":"2023-11-28T17:41:18.338684+0000","last_became_active":"2023-11-28T17:41:17.247475+0000","last_became_peered":"2023-11-28T17:41:17.247475+0000","last_unstale":"2023-11-28T17:41:18.338684+0000","last_undegraded":"2023-11-28T17:41:18.338684+0000","last_fullsized":"2023-11-28T17:41:18.338684+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:18.555573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632851+0000","last_change":"2023-11-28T17:41:16.262927+0000","last_active":"2023-11-28T17:41:19.632851+0000","last_peered":"2023-11-28T17:41:19.632851+0000","last_clean":"2023-11-28T17:41:19.632851+0000","last_became_active":"2023-11-28T17:41:16.262556+0000","last_became_peered":"2023-11-28T17:41:16.262556+0000","last_unstale":"2023-11-28T17:41:19.632851+0000","last_undegraded":"2023-11-28T17:41:19.632851+0000","last_fullsized":"2023-11-28T17:41:19.632851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:35:54.672978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632921+0000","last_change":"2023-11-28T17:41:17.239625+0000","last_active":"2023-11-28T17:41:19.632921+0000","last_peered":"2023-11-28T17:41:19.632921+0000","last_clean":"2023-11-28T17:41:19.632921+0000","last_became_active":"2023-11-28T17:41:17.238215+0000","last_became_peered":"2023-11-28T17:41:17.238215+0000","last_unstale":"2023-11-28T17:41:19.632921+0000","last_undegraded":"2023-11-28T17:41:19.632921+0000","last_fullsized":"2023-11-28T17:41:19.632921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:33:31.879592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.284665+0000","last_change":"2023-11-28T17:41:16.250674+0000","last_active":"2023-11-28T17:41:22.284665+0000","last_peered":"2023-11-28T17:41:22.284665+0000","last_clean":"2023-11-28T17:41:22.284665+0000","last_became_active":"2023-11-28T17:41:16.250312+0000","last_became_peered":"2023-11-28T17:41:16.250312+0000","last_unstale":"2023-11-28T17:41:22.284665+0000","last_undegraded":"2023-11-28T17:41:22.284665+0000","last_fullsized":"2023-11-28T17:41:22.284665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:04:05.342081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338315+0000","last_change":"2023-11-28T17:41:17.247675+0000","last_active":"2023-11-28T17:41:18.338315+0000","last_peered":"2023-11-28T17:41:18.338315+0000","last_clean":"2023-11-28T17:41:18.338315+0000","last_became_active":"2023-11-28T17:41:17.247346+0000","last_became_peered":"2023-11-28T17:41:17.247346+0000","last_unstale":"2023-11-28T17:41:18.338315+0000","last_undegraded":"2023-11-28T17:41:18.338315+0000","last_fullsized":"2023-11-28T17:41:18.338315+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:45:03.326583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.209094+0000","last_change":"2023-11-28T17:41:16.260893+0000","last_active":"2023-11-28T17:41:18.209094+0000","last_peered":"2023-11-28T17:41:18.209094+0000","last_clean":"2023-11-28T17:41:18.209094+0000","last_became_active":"2023-11-28T17:41:16.260637+0000","last_became_peered":"2023-11-28T17:41:16.260637+0000","last_unstale":"2023-11-28T17:41:18.209094+0000","last_undegraded":"2023-11-28T17:41:18.209094+0000","last_fullsized":"2023-11-28T17:41:18.209094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:07:24.032989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338811+0000","last_change":"2023-11-28T17:41:17.249202+0000","last_active":"2023-11-28T17:41:18.338811+0000","last_peered":"2023-11-28T17:41:18.338811+0000","last_clean":"2023-11-28T17:41:18.338811+0000","last_became_active":"2023-11-28T17:41:17.248187+0000","last_became_peered":"2023-11-28T17:41:17.248187+0000","last_unstale":"2023-11-28T17:41:18.338811+0000","last_undegraded":"2023-11-28T17:41:18.338811+0000","last_fullsized":"2023-11-28T17:41:18.338811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:23.973069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.277051+0000","last_change":"2023-11-28T17:41:16.244938+0000","last_active":"2023-11-28T17:41:22.277051+0000","last_peered":"2023-11-28T17:41:22.277051+0000","last_clean":"2023-11-28T17:41:22.277051+0000","last_became_active":"2023-11-28T17:41:16.244586+0000","last_became_peered":"2023-11-28T17:41:16.244586+0000","last_unstale":"2023-11-28T17:41:22.277051+0000","last_undegraded":"2023-11-28T17:41:22.277051+0000","last_fullsized":"2023-11-28T17:41:22.277051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:38.421966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339409+0000","last_change":"2023-11-28T17:41:17.248488+0000","last_active":"2023-11-28T17:41:18.339409+0000","last_peered":"2023-11-28T17:41:18.339409+0000","last_clean":"2023-11-28T17:41:18.339409+0000","last_became_active":"2023-11-28T17:41:17.248170+0000","last_became_peered":"2023-11-28T17:41:17.248170+0000","last_unstale":"2023-11-28T17:41:18.339409+0000","last_undegraded":"2023-11-28T17:41:18.339409+0000","last_fullsized":"2023-11-28T17:41:18.339409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:41.188370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632665+0000","last_change":"2023-11-28T17:41:16.246192+0000","last_active":"2023-11-28T17:41:19.632665+0000","last_peered":"2023-11-28T17:41:19.632665+0000","last_clean":"2023-11-28T17:41:19.632665+0000","last_became_active":"2023-11-28T17:41:16.245379+0000","last_became_peered":"2023-11-28T17:41:16.245379+0000","last_unstale":"2023-11-28T17:41:19.632665+0000","last_undegraded":"2023-11-28T17:41:19.632665+0000","last_fullsized":"2023-11-28T17:41:19.632665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:23:08.664344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207475+0000","last_change":"2023-11-28T17:41:17.234273+0000","last_active":"2023-11-28T17:41:18.207475+0000","last_peered":"2023-11-28T17:41:18.207475+0000","last_clean":"2023-11-28T17:41:18.207475+0000","last_became_active":"2023-11-28T17:41:17.233756+0000","last_became_peered":"2023-11-28T17:41:17.233756+0000","last_unstale":"2023-11-28T17:41:18.207475+0000","last_undegraded":"2023-11-28T17:41:18.207475+0000","last_fullsized":"2023-11-28T17:41:18.207475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:18:40.539329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.279280+0000","last_change":"2023-11-28T17:41:16.257952+0000","last_active":"2023-11-28T17:41:17.279280+0000","last_peered":"2023-11-28T17:41:17.279280+0000","last_clean":"2023-11-28T17:41:17.279280+0000","last_became_active":"2023-11-28T17:41:16.257659+0000","last_became_peered":"2023-11-28T17:41:16.257659+0000","last_unstale":"2023-11-28T17:41:17.279280+0000","last_undegraded":"2023-11-28T17:41:17.279280+0000","last_fullsized":"2023-11-28T17:41:17.279280+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:28:12.131592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338433+0000","last_change":"2023-11-28T17:41:17.239069+0000","last_active":"2023-11-28T17:41:18.338433+0000","last_peered":"2023-11-28T17:41:18.338433+0000","last_clean":"2023-11-28T17:41:18.338433+0000","last_became_active":"2023-11-28T17:41:17.238548+0000","last_became_peered":"2023-11-28T17:41:17.238548+0000","last_unstale":"2023-11-28T17:41:18.338433+0000","last_undegraded":"2023-11-28T17:41:18.338433+0000","last_fullsized":"2023-11-28T17:41:18.338433+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:12:56.686814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206759+0000","last_change":"2023-11-28T17:41:12.444311+0000","last_active":"2023-11-28T17:41:17.206759+0000","last_peered":"2023-11-28T17:41:17.206759+0000","last_clean":"2023-11-28T17:41:17.206759+0000","last_became_active":"2023-11-28T17:41:12.442193+0000","last_became_peered":"2023-11-28T17:41:12.442193+0000","last_unstale":"2023-11-28T17:41:17.206759+0000","last_undegraded":"2023-11-28T17:41:17.206759+0000","last_fullsized":"2023-11-28T17:41:17.206759+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:10.865936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:10.865936+0000","last_clean_scrub_stamp":"2023-11-28T17:41:10.865936+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:12:48.912392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206457+0000","last_change":"2023-11-28T17:41:16.253023+0000","last_active":"2023-11-28T17:41:18.206457+0000","last_peered":"2023-11-28T17:41:18.206457+0000","last_clean":"2023-11-28T17:41:18.206457+0000","last_became_active":"2023-11-28T17:41:16.245730+0000","last_became_peered":"2023-11-28T17:41:16.245730+0000","last_unstale":"2023-11-28T17:41:18.206457+0000","last_undegraded":"2023-11-28T17:41:18.206457+0000","last_fullsized":"2023-11-28T17:41:18.206457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:05.970917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568226+0000","last_change":"2023-11-28T17:41:17.246677+0000","last_active":"2023-11-28T17:41:18.568226+0000","last_peered":"2023-11-28T17:41:18.568226+0000","last_clean":"2023-11-28T17:41:18.568226+0000","last_became_active":"2023-11-28T17:41:17.246373+0000","last_became_peered":"2023-11-28T17:41:17.246373+0000","last_unstale":"2023-11-28T17:41:18.568226+0000","last_undegraded":"2023-11-28T17:41:18.568226+0000","last_fullsized":"2023-11-28T17:41:18.568226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:22:18.623331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206504+0000","last_change":"2023-11-28T17:41:16.272843+0000","last_active":"2023-11-28T17:41:17.206504+0000","last_peered":"2023-11-28T17:41:17.206504+0000","last_clean":"2023-11-28T17:41:17.206504+0000","last_became_active":"2023-11-28T17:41:16.272527+0000","last_became_peered":"2023-11-28T17:41:16.272527+0000","last_unstale":"2023-11-28T17:41:17.206504+0000","last_undegraded":"2023-11-28T17:41:17.206504+0000","last_fullsized":"2023-11-28T17:41:17.206504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:55:52.185953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649479+0000","last_change":"2023-11-28T17:41:17.240890+0000","last_active":"2023-11-28T17:41:18.649479+0000","last_peered":"2023-11-28T17:41:18.649479+0000","last_clean":"2023-11-28T17:41:18.649479+0000","last_became_active":"2023-11-28T17:41:17.240571+0000","last_became_peered":"2023-11-28T17:41:17.240571+0000","last_unstale":"2023-11-28T17:41:18.649479+0000","last_undegraded":"2023-11-28T17:41:18.649479+0000","last_fullsized":"2023-11-28T17:41:18.649479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:58.086322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.279241+0000","last_change":"2023-11-28T17:41:16.251925+0000","last_active":"2023-11-28T17:41:17.279241+0000","last_peered":"2023-11-28T17:41:17.279241+0000","last_clean":"2023-11-28T17:41:17.279241+0000","last_became_active":"2023-11-28T17:41:16.251482+0000","last_became_peered":"2023-11-28T17:41:16.251482+0000","last_unstale":"2023-11-28T17:41:17.279241+0000","last_undegraded":"2023-11-28T17:41:17.279241+0000","last_fullsized":"2023-11-28T17:41:17.279241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:07:09.590250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632279+0000","last_change":"2023-11-28T17:41:17.223810+0000","last_active":"2023-11-28T17:41:19.632279+0000","last_peered":"2023-11-28T17:41:19.632279+0000","last_clean":"2023-11-28T17:41:19.632279+0000","last_became_active":"2023-11-28T17:41:17.223244+0000","last_became_peered":"2023-11-28T17:41:17.223244+0000","last_unstale":"2023-11-28T17:41:19.632279+0000","last_undegraded":"2023-11-28T17:41:19.632279+0000","last_fullsized":"2023-11-28T17:41:19.632279+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:31:22.132844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207190+0000","last_change":"2023-11-28T17:41:16.252402+0000","last_active":"2023-11-28T17:41:18.207190+0000","last_peered":"2023-11-28T17:41:18.207190+0000","last_clean":"2023-11-28T17:41:18.207190+0000","last_became_active":"2023-11-28T17:41:16.252061+0000","last_became_peered":"2023-11-28T17:41:16.252061+0000","last_unstale":"2023-11-28T17:41:18.207190+0000","last_undegraded":"2023-11-28T17:41:18.207190+0000","last_fullsized":"2023-11-28T17:41:18.207190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:20:11.890036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207754+0000","last_change":"2023-11-28T17:41:17.234320+0000","last_active":"2023-11-28T17:41:18.207754+0000","last_peered":"2023-11-28T17:41:18.207754+0000","last_clean":"2023-11-28T17:41:18.207754+0000","last_became_active":"2023-11-28T17:41:17.233738+0000","last_became_peered":"2023-11-28T17:41:17.233738+0000","last_unstale":"2023-11-28T17:41:18.207754+0000","last_undegraded":"2023-11-28T17:41:18.207754+0000","last_fullsized":"2023-11-28T17:41:18.207754+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:31:10.257906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.204406+0000","last_change":"2023-11-28T17:41:16.245394+0000","last_active":"2023-11-28T17:41:17.204406+0000","last_peered":"2023-11-28T17:41:17.204406+0000","last_clean":"2023-11-28T17:41:17.204406+0000","last_became_active":"2023-11-28T17:41:16.242985+0000","last_became_peered":"2023-11-28T17:41:16.242985+0000","last_unstale":"2023-11-28T17:41:17.204406+0000","last_undegraded":"2023-11-28T17:41:17.204406+0000","last_fullsized":"2023-11-28T17:41:17.204406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:48.728960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207612+0000","last_change":"2023-11-28T17:41:17.233987+0000","last_active":"2023-11-28T17:41:18.207612+0000","last_peered":"2023-11-28T17:41:18.207612+0000","last_clean":"2023-11-28T17:41:18.207612+0000","last_became_active":"2023-11-28T17:41:17.233430+0000","last_became_peered":"2023-11-28T17:41:17.233430+0000","last_unstale":"2023-11-28T17:41:18.207612+0000","last_undegraded":"2023-11-28T17:41:18.207612+0000","last_fullsized":"2023-11-28T17:41:18.207612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:51:44.816033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.213771+0000","last_change":"2023-11-28T17:41:16.273047+0000","last_active":"2023-11-28T17:41:17.213771+0000","last_peered":"2023-11-28T17:41:17.213771+0000","last_clean":"2023-11-28T17:41:17.213771+0000","last_became_active":"2023-11-28T17:41:16.272667+0000","last_became_peered":"2023-11-28T17:41:16.272667+0000","last_unstale":"2023-11-28T17:41:17.213771+0000","last_undegraded":"2023-11-28T17:41:17.213771+0000","last_fullsized":"2023-11-28T17:41:17.213771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:19:37.472121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632439+0000","last_change":"2023-11-28T17:41:17.232465+0000","last_active":"2023-11-28T17:41:19.632439+0000","last_peered":"2023-11-28T17:41:19.632439+0000","last_clean":"2023-11-28T17:41:19.632439+0000","last_became_active":"2023-11-28T17:41:17.230496+0000","last_became_peered":"2023-11-28T17:41:17.230496+0000","last_unstale":"2023-11-28T17:41:19.632439+0000","last_undegraded":"2023-11-28T17:41:19.632439+0000","last_fullsized":"2023-11-28T17:41:19.632439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:52:08.590259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206197+0000","last_change":"2023-11-28T17:41:16.258717+0000","last_active":"2023-11-28T17:41:17.206197+0000","last_peered":"2023-11-28T17:41:17.206197+0000","last_clean":"2023-11-28T17:41:17.206197+0000","last_became_active":"2023-11-28T17:41:16.258388+0000","last_became_peered":"2023-11-28T17:41:16.258388+0000","last_unstale":"2023-11-28T17:41:17.206197+0000","last_undegraded":"2023-11-28T17:41:17.206197+0000","last_fullsized":"2023-11-28T17:41:17.206197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:17.845137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568998+0000","last_change":"2023-11-28T17:41:17.239189+0000","last_active":"2023-11-28T17:41:18.568998+0000","last_peered":"2023-11-28T17:41:18.568998+0000","last_clean":"2023-11-28T17:41:18.568998+0000","last_became_active":"2023-11-28T17:41:17.238829+0000","last_became_peered":"2023-11-28T17:41:17.238829+0000","last_unstale":"2023-11-28T17:41:18.568998+0000","last_undegraded":"2023-11-28T17:41:18.568998+0000","last_fullsized":"2023-11-28T17:41:18.568998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:10.923572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.276656+0000","last_change":"2023-11-28T17:41:16.268334+0000","last_active":"2023-11-28T17:41:22.276656+0000","last_peered":"2023-11-28T17:41:22.276656+0000","last_clean":"2023-11-28T17:41:22.276656+0000","last_became_active":"2023-11-28T17:41:16.268015+0000","last_became_peered":"2023-11-28T17:41:16.268015+0000","last_unstale":"2023-11-28T17:41:22.276656+0000","last_undegraded":"2023-11-28T17:41:22.276656+0000","last_fullsized":"2023-11-28T17:41:22.276656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:57:22.031553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650340+0000","last_change":"2023-11-28T17:41:17.225161+0000","last_active":"2023-11-28T17:41:18.650340+0000","last_peered":"2023-11-28T17:41:18.650340+0000","last_clean":"2023-11-28T17:41:18.650340+0000","last_became_active":"2023-11-28T17:41:17.224420+0000","last_became_peered":"2023-11-28T17:41:17.224420+0000","last_unstale":"2023-11-28T17:41:18.650340+0000","last_undegraded":"2023-11-28T17:41:18.650340+0000","last_fullsized":"2023-11-28T17:41:18.650340+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:25:11.426980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206160+0000","last_change":"2023-11-28T17:41:16.272884+0000","last_active":"2023-11-28T17:41:17.206160+0000","last_peered":"2023-11-28T17:41:17.206160+0000","last_clean":"2023-11-28T17:41:17.206160+0000","last_became_active":"2023-11-28T17:41:16.272557+0000","last_became_peered":"2023-11-28T17:41:16.272557+0000","last_unstale":"2023-11-28T17:41:17.206160+0000","last_undegraded":"2023-11-28T17:41:17.206160+0000","last_fullsized":"2023-11-28T17:41:17.206160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:56.634559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206149+0000","last_change":"2023-11-28T17:41:17.238145+0000","last_active":"2023-11-28T17:41:18.206149+0000","last_peered":"2023-11-28T17:41:18.206149+0000","last_clean":"2023-11-28T17:41:18.206149+0000","last_became_active":"2023-11-28T17:41:17.237774+0000","last_became_peered":"2023-11-28T17:41:17.237774+0000","last_unstale":"2023-11-28T17:41:18.206149+0000","last_undegraded":"2023-11-28T17:41:18.206149+0000","last_fullsized":"2023-11-28T17:41:18.206149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:44.723180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.204215+0000","last_change":"2023-11-28T17:41:16.262318+0000","last_active":"2023-11-28T17:41:17.204215+0000","last_peered":"2023-11-28T17:41:17.204215+0000","last_clean":"2023-11-28T17:41:17.204215+0000","last_became_active":"2023-11-28T17:41:16.262014+0000","last_became_peered":"2023-11-28T17:41:16.262014+0000","last_unstale":"2023-11-28T17:41:17.204215+0000","last_undegraded":"2023-11-28T17:41:17.204215+0000","last_fullsized":"2023-11-28T17:41:17.204215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:17:15.470393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568031+0000","last_change":"2023-11-28T17:41:17.229233+0000","last_active":"2023-11-28T17:41:18.568031+0000","last_peered":"2023-11-28T17:41:18.568031+0000","last_clean":"2023-11-28T17:41:18.568031+0000","last_became_active":"2023-11-28T17:41:17.228885+0000","last_became_peered":"2023-11-28T17:41:17.228885+0000","last_unstale":"2023-11-28T17:41:18.568031+0000","last_undegraded":"2023-11-28T17:41:18.568031+0000","last_fullsized":"2023-11-28T17:41:18.568031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:10:24.247307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.203545+0000","last_change":"2023-11-28T17:41:16.246533+0000","last_active":"2023-11-28T17:41:17.203545+0000","last_peered":"2023-11-28T17:41:17.203545+0000","last_clean":"2023-11-28T17:41:17.203545+0000","last_became_active":"2023-11-28T17:41:16.244067+0000","last_became_peered":"2023-11-28T17:41:16.244067+0000","last_unstale":"2023-11-28T17:41:17.203545+0000","last_undegraded":"2023-11-28T17:41:17.203545+0000","last_fullsized":"2023-11-28T17:41:17.203545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:02:20.862297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568303+0000","last_change":"2023-11-28T17:41:17.236159+0000","last_active":"2023-11-28T17:41:18.568303+0000","last_peered":"2023-11-28T17:41:18.568303+0000","last_clean":"2023-11-28T17:41:18.568303+0000","last_became_active":"2023-11-28T17:41:17.235700+0000","last_became_peered":"2023-11-28T17:41:17.235700+0000","last_unstale":"2023-11-28T17:41:18.568303+0000","last_undegraded":"2023-11-28T17:41:18.568303+0000","last_fullsized":"2023-11-28T17:41:18.568303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:21.900805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.271288+0000","last_change":"2023-11-28T17:41:16.268969+0000","last_active":"2023-11-28T17:41:22.271288+0000","last_peered":"2023-11-28T17:41:22.271288+0000","last_clean":"2023-11-28T17:41:22.271288+0000","last_became_active":"2023-11-28T17:41:16.268717+0000","last_became_peered":"2023-11-28T17:41:16.268717+0000","last_unstale":"2023-11-28T17:41:22.271288+0000","last_undegraded":"2023-11-28T17:41:22.271288+0000","last_fullsized":"2023-11-28T17:41:22.271288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:03:29.120624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633192+0000","last_change":"2023-11-28T17:41:17.244376+0000","last_active":"2023-11-28T17:41:19.633192+0000","last_peered":"2023-11-28T17:41:19.633192+0000","last_clean":"2023-11-28T17:41:19.633192+0000","last_became_active":"2023-11-28T17:41:17.244034+0000","last_became_peered":"2023-11-28T17:41:17.244034+0000","last_unstale":"2023-11-28T17:41:19.633192+0000","last_undegraded":"2023-11-28T17:41:19.633192+0000","last_fullsized":"2023-11-28T17:41:19.633192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:15:56.198732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208198+0000","last_change":"2023-11-28T17:41:17.233749+0000","last_active":"2023-11-28T17:41:18.208198+0000","last_peered":"2023-11-28T17:41:18.208198+0000","last_clean":"2023-11-28T17:41:18.208198+0000","last_became_active":"2023-11-28T17:41:17.232992+0000","last_became_peered":"2023-11-28T17:41:17.232992+0000","last_unstale":"2023-11-28T17:41:18.208198+0000","last_undegraded":"2023-11-28T17:41:18.208198+0000","last_fullsized":"2023-11-28T17:41:18.208198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:00:43.698244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.276885+0000","last_change":"2023-11-28T17:41:16.269465+0000","last_active":"2023-11-28T17:41:22.276885+0000","last_peered":"2023-11-28T17:41:22.276885+0000","last_clean":"2023-11-28T17:41:22.276885+0000","last_became_active":"2023-11-28T17:41:16.269147+0000","last_became_peered":"2023-11-28T17:41:16.269147+0000","last_unstale":"2023-11-28T17:41:22.276885+0000","last_undegraded":"2023-11-28T17:41:22.276885+0000","last_fullsized":"2023-11-28T17:41:22.276885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:57:58.121232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207346+0000","last_change":"2023-11-28T17:41:17.243498+0000","last_active":"2023-11-28T17:41:18.207346+0000","last_peered":"2023-11-28T17:41:18.207346+0000","last_clean":"2023-11-28T17:41:18.207346+0000","last_became_active":"2023-11-28T17:41:17.243169+0000","last_became_peered":"2023-11-28T17:41:17.243169+0000","last_unstale":"2023-11-28T17:41:18.207346+0000","last_undegraded":"2023-11-28T17:41:18.207346+0000","last_fullsized":"2023-11-28T17:41:18.207346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:11:31.509921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.275947+0000","last_change":"2023-11-28T17:41:16.270231+0000","last_active":"2023-11-28T17:41:22.275947+0000","last_peered":"2023-11-28T17:41:22.275947+0000","last_clean":"2023-11-28T17:41:22.275947+0000","last_became_active":"2023-11-28T17:41:16.269938+0000","last_became_peered":"2023-11-28T17:41:16.269938+0000","last_unstale":"2023-11-28T17:41:22.275947+0000","last_undegraded":"2023-11-28T17:41:22.275947+0000","last_fullsized":"2023-11-28T17:41:22.275947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:45.654658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650357+0000","last_change":"2023-11-28T17:41:17.245856+0000","last_active":"2023-11-28T17:41:18.650357+0000","last_peered":"2023-11-28T17:41:18.650357+0000","last_clean":"2023-11-28T17:41:18.650357+0000","last_became_active":"2023-11-28T17:41:17.245624+0000","last_became_peered":"2023-11-28T17:41:17.245624+0000","last_unstale":"2023-11-28T17:41:18.650357+0000","last_undegraded":"2023-11-28T17:41:18.650357+0000","last_fullsized":"2023-11-28T17:41:18.650357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:03:10.206506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568922+0000","last_change":"2023-11-28T17:41:16.260989+0000","last_active":"2023-11-28T17:41:18.568922+0000","last_peered":"2023-11-28T17:41:18.568922+0000","last_clean":"2023-11-28T17:41:18.568922+0000","last_became_active":"2023-11-28T17:41:16.260686+0000","last_became_peered":"2023-11-28T17:41:16.260686+0000","last_unstale":"2023-11-28T17:41:18.568922+0000","last_undegraded":"2023-11-28T17:41:18.568922+0000","last_fullsized":"2023-11-28T17:41:18.568922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:58:44.193791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568672+0000","last_change":"2023-11-28T17:41:17.240803+0000","last_active":"2023-11-28T17:41:18.568672+0000","last_peered":"2023-11-28T17:41:18.568672+0000","last_clean":"2023-11-28T17:41:18.568672+0000","last_became_active":"2023-11-28T17:41:17.240442+0000","last_became_peered":"2023-11-28T17:41:17.240442+0000","last_unstale":"2023-11-28T17:41:18.568672+0000","last_undegraded":"2023-11-28T17:41:18.568672+0000","last_fullsized":"2023-11-28T17:41:18.568672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:28:21.223788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.279104+0000","last_change":"2023-11-28T17:41:16.266520+0000","last_active":"2023-11-28T17:41:22.279104+0000","last_peered":"2023-11-28T17:41:22.279104+0000","last_clean":"2023-11-28T17:41:22.279104+0000","last_became_active":"2023-11-28T17:41:16.266245+0000","last_became_peered":"2023-11-28T17:41:16.266245+0000","last_unstale":"2023-11-28T17:41:22.279104+0000","last_undegraded":"2023-11-28T17:41:22.279104+0000","last_fullsized":"2023-11-28T17:41:22.279104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:18:28.732421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65062,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":290816,"data_stored":667482,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1060000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.73}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.804}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5109999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]}]},{"osd":6,"up_from":15,"seq":64424509445,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":78144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0110000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3720000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8340000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.278}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3279999999999998}]}]},{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":76622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.323}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4490000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5920000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":76658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.377}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.482}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.756}]}]},{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":78144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.427}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0110000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7190000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.913}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.827}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":76692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4910000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.98}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.796}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0579999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0990000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":299008,"data_stored":667538,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.909}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0369999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9930000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1869999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":76532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8080000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T17:41:27.830 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-28T17:41:27.830 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-28T17:41:27.830 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-28T17:41:27.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T17:41:28.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.073+0000 7f9c3084a640 1 Processor -- start 2023-11-28T17:41:28.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.073+0000 7f9c3084a640 1 -- start start 2023-11-28T17:41:28.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.075+0000 7f9c3084a640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9c2c154060 0x7f9c2c0bff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:28.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.075+0000 7f9c3084a640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c2c06afe0 0x7f9c2c0c0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:28.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.075+0000 7f9c3084a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9c2c0c0d70 0x7f9c2c06dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:28.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.075+0000 7f9c3084a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2c0c11e0 con 0x7f9c2c0c0d70 2023-11-28T17:41:28.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.075+0000 7f9c3084a640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2c159c60 con 0x7f9c2c154060 2023-11-28T17:41:28.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.075+0000 7f9c3084a640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c154440 con 0x7f9c2c06afe0 2023-11-28T17:41:28.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.075+0000 7f9c2b59e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9c2c0c0d70 0x7f9c2c06dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:28.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.075+0000 7f9c2b59e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9c2c0c0d70 0x7f9c2c06dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49618/0 (socket says 172.21.15.98:49618) 2023-11-28T17:41:28.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.075+0000 7f9c2b59e640 1 -- 172.21.15.98:0/1791451085 learned_addr learned my addr 172.21.15.98:0/1791451085 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:28.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.075+0000 7f9c2ad9d640 1 --2- 172.21.15.98:0/1791451085 >> v2:172.21.15.113:3300/0 conn(0x7f9c2c154060 0x7f9c2c0bff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:28.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.075+0000 7f9c2a59c640 1 --2- 172.21.15.98:0/1791451085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c2c06afe0 0x7f9c2c0c0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:28.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.075+0000 7f9c2b59e640 1 -- 172.21.15.98:0/1791451085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c2c06afe0 msgr2=0x7f9c2c0c0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:28.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.075+0000 7f9c2b59e640 1 --2- 172.21.15.98:0/1791451085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c2c06afe0 0x7f9c2c0c0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:28.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.076+0000 7f9c2b59e640 1 -- 172.21.15.98:0/1791451085 >> v2:172.21.15.113:3300/0 conn(0x7f9c2c154060 msgr2=0x7f9c2c0bff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:28.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.076+0000 7f9c2b59e640 1 --2- 172.21.15.98:0/1791451085 >> v2:172.21.15.113:3300/0 conn(0x7f9c2c154060 0x7f9c2c0bff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:28.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.076+0000 7f9c2b59e640 1 -- 172.21.15.98:0/1791451085 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c159e80 con 0x7f9c2c0c0d70 2023-11-28T17:41:28.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.077+0000 7f9c2b59e640 1 --2- 172.21.15.98:0/1791451085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c2c0c0d70 0x7f9c2c06dd30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c06fa20 tx=0x7f9c2402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e83941266cf0845 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:28.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.077+0000 7f9c29d9b640 1 -- 172.21.15.98:0/1791451085 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2402ca70 con 0x7f9c2c0c0d70 2023-11-28T17:41:28.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.077+0000 7f9c29d9b640 1 -- 172.21.15.98:0/1791451085 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2402cbd0 con 0x7f9c2c0c0d70 2023-11-28T17:41:28.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.077+0000 7f9c29d9b640 1 -- 172.21.15.98:0/1791451085 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c240343f0 con 0x7f9c2c0c0d70 2023-11-28T17:41:28.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.077+0000 7f9c3084a640 1 -- 172.21.15.98:0/1791451085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c2c0c0d70 msgr2=0x7f9c2c06dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:28.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.077+0000 7f9c3084a640 1 --2- 172.21.15.98:0/1791451085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c2c0c0d70 0x7f9c2c06dd30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c06fa20 tx=0x7f9c2402bed0 comp rx=0 tx=0).stop 2023-11-28T17:41:28.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.077+0000 7f9c3084a640 1 -- 172.21.15.98:0/1791451085 shutdown_connections 2023-11-28T17:41:28.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.077+0000 7f9c3084a640 1 --2- 172.21.15.98:0/1791451085 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c2c06afe0 0x7f9c2c0c0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:28.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.078+0000 7f9c3084a640 1 --2- 172.21.15.98:0/1791451085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c2c0c0d70 0x7f9c2c06dd30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:28.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.078+0000 7f9c3084a640 1 --2- 172.21.15.98:0/1791451085 >> v2:172.21.15.113:3300/0 conn(0x7f9c2c154060 0x7f9c2c0bff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:28.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.078+0000 7f9c3084a640 1 -- 172.21.15.98:0/1791451085 >> 172.21.15.98:0/1791451085 conn(0x7f9c2c066230 msgr2=0x7f9c2c06a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:28.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.078+0000 7f9c3084a640 1 -- 172.21.15.98:0/1791451085 shutdown_connections 2023-11-28T17:41:28.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.078+0000 7f9c3084a640 1 -- 172.21.15.98:0/1791451085 wait complete. 2023-11-28T17:41:28.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.079+0000 7f9c3084a640 1 Processor -- start 2023-11-28T17:41:28.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.079+0000 7f9c3084a640 1 -- start start 2023-11-28T17:41:28.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.080+0000 7f9c3084a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c2c06afe0 0x7f9c2c0710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:28.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.080+0000 7f9c3084a640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c2c0c0d70 0x7f9c2c0ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:28.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.080+0000 7f9c3084a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c2c154060 0x7f9c2c1528d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:28.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.080+0000 7f9c3084a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c157730 con 0x7f9c2c06afe0 2023-11-28T17:41:28.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.080+0000 7f9c3084a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c152440 con 0x7f9c2c154060 2023-11-28T17:41:28.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.080+0000 7f9c3084a640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c1556f0 con 0x7f9c2c0c0d70 2023-11-28T17:41:28.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.080+0000 7f9c2ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c2c06afe0 0x7f9c2c0710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:28.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.080+0000 7f9c2ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c2c06afe0 0x7f9c2c0710a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49630/0 (socket says 172.21.15.98:49630) 2023-11-28T17:41:28.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.080+0000 7f9c2ad9d640 1 -- 172.21.15.98:0/3125013201 learned_addr learned my addr 172.21.15.98:0/3125013201 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:28.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.081+0000 7f9c2ad9d640 1 -- 172.21.15.98:0/3125013201 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c2c0c0d70 msgr2=0x7f9c2c0ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:28.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.081+0000 7f9c2ad9d640 1 --2- 172.21.15.98:0/3125013201 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c2c0c0d70 0x7f9c2c0ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:28.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.081+0000 7f9c2b59e640 1 --2- 172.21.15.98:0/3125013201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c2c154060 0x7f9c2c1528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:28.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.081+0000 7f9c2ad9d640 1 -- 172.21.15.98:0/3125013201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c2c154060 msgr2=0x7f9c2c1528d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:28.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.081+0000 7f9c2ad9d640 1 --2- 172.21.15.98:0/3125013201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c2c154060 0x7f9c2c1528d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:28.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.081+0000 7f9c2ad9d640 1 -- 172.21.15.98:0/3125013201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c24009b30 con 0x7f9c2c06afe0 2023-11-28T17:41:28.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.081+0000 7f9c2ad9d640 1 --2- 172.21.15.98:0/3125013201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c2c06afe0 0x7f9c2c0710a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c00d5f0 tx=0x7f9c1c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:28.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.081+0000 7f9c137fe640 1 -- 172.21.15.98:0/3125013201 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c0125f0 con 0x7f9c2c06afe0 2023-11-28T17:41:28.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.081+0000 7f9c137fe640 1 -- 172.21.15.98:0/3125013201 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1c012750 con 0x7f9c2c06afe0 2023-11-28T17:41:28.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.081+0000 7f9c137fe640 1 -- 172.21.15.98:0/3125013201 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c012b00 con 0x7f9c2c06afe0 2023-11-28T17:41:28.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.081+0000 7f9c3084a640 1 -- 172.21.15.98:0/3125013201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2c151b30 con 0x7f9c2c06afe0 2023-11-28T17:41:28.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.081+0000 7f9c3084a640 1 -- 172.21.15.98:0/3125013201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c2c071bc0 con 0x7f9c2c06afe0 2023-11-28T17:41:28.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.082+0000 7f9c3084a640 1 -- 172.21.15.98:0/3125013201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2c074db0 con 0x7f9c2c06afe0 2023-11-28T17:41:28.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.084+0000 7f9c137fe640 1 -- 172.21.15.98:0/3125013201 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f9c1c012c60 con 0x7f9c2c06afe0 2023-11-28T17:41:28.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.084+0000 7f9c137fe640 1 --2- 172.21.15.98:0/3125013201 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9c1806b2d0 0x7f9c1806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:28.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.085+0000 7f9c137fe640 1 -- 172.21.15.98:0/3125013201 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c1c08a180 con 0x7f9c2c06afe0 2023-11-28T17:41:28.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.085+0000 7f9c2a59c640 1 --2- 172.21.15.98:0/3125013201 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9c1806b2d0 0x7f9c1806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:28.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.085+0000 7f9c2a59c640 1 --2- 172.21.15.98:0/3125013201 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9c1806b2d0 0x7f9c1806d790 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c0c6450 tx=0x7f9c20008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:28.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.089+0000 7f9c137fe640 1 -- 172.21.15.98:0/3125013201 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9c1c054ec0 con 0x7f9c2c06afe0 2023-11-28T17:41:28.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.323+0000 7f9c3084a640 1 -- 172.21.15.98:0/3125013201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9c2c0727b0 con 0x7f9c2c06afe0 2023-11-28T17:41:28.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.323+0000 7f9c137fe640 1 -- 172.21.15.98:0/3125013201 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f9c1c0584e0 con 0x7f9c2c06afe0 2023-11-28T17:41:28.325 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:28.325 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-11-28T17:41:28.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.326+0000 7f9c3084a640 1 -- 172.21.15.98:0/3125013201 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9c1806b2d0 msgr2=0x7f9c1806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:28.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.326+0000 7f9c3084a640 1 --2- 172.21.15.98:0/3125013201 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9c1806b2d0 0x7f9c1806d790 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c0c6450 tx=0x7f9c20008040 comp rx=0 tx=0).stop 2023-11-28T17:41:28.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.326+0000 7f9c3084a640 1 -- 172.21.15.98:0/3125013201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c2c06afe0 msgr2=0x7f9c2c0710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:28.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.326+0000 7f9c3084a640 1 --2- 172.21.15.98:0/3125013201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c2c06afe0 0x7f9c2c0710a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c00d5f0 tx=0x7f9c1c00dab0 comp rx=0 tx=0).stop 2023-11-28T17:41:28.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.326+0000 7f9c3084a640 1 -- 172.21.15.98:0/3125013201 shutdown_connections 2023-11-28T17:41:28.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.326+0000 7f9c3084a640 1 --2- 172.21.15.98:0/3125013201 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9c1806b2d0 0x7f9c1806d790 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:28.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.326+0000 7f9c3084a640 1 --2- 172.21.15.98:0/3125013201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c2c154060 0x7f9c2c1528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:28.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.326+0000 7f9c3084a640 1 --2- 172.21.15.98:0/3125013201 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c2c0c0d70 0x7f9c2c0ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:28.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.326+0000 7f9c3084a640 1 --2- 172.21.15.98:0/3125013201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c2c06afe0 0x7f9c2c0710a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:28.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.326+0000 7f9c3084a640 1 -- 172.21.15.98:0/3125013201 >> 172.21.15.98:0/3125013201 conn(0x7f9c2c066230 msgr2=0x7f9c2c0afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:28.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.326+0000 7f9c3084a640 1 -- 172.21.15.98:0/3125013201 shutdown_connections 2023-11-28T17:41:28.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:28.326+0000 7f9c3084a640 1 -- 172.21.15.98:0/3125013201 wait complete. 2023-11-28T17:41:30.878 INFO:tasks.ceph.mds.b.smithi113.stderr:2023-11-28T17:41:30.877+0000 7f6aad4d5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T17:41:30.879 INFO:tasks.ceph.mds.b.smithi113.stderr:2023-11-28T17:41:30.877+0000 7f6aad4d5640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-11-28T17:41:31.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T17:41:31.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.551+0000 7fb8848c9640 1 Processor -- start 2023-11-28T17:41:31.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.551+0000 7fb8848c9640 1 -- start start 2023-11-28T17:41:31.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.553+0000 7fb8848c9640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb880154040 0x7fb88015dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:31.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.553+0000 7fb8848c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb880159c50 con 0x7fb8800c9000 2023-11-28T17:41:31.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.553+0000 7fb8848c9640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8800cab90 con 0x7fb8800c9420 2023-11-28T17:41:31.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.553+0000 7fb8848c9640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8800c2ad0 con 0x7fb880154040 2023-11-28T17:41:31.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.553+0000 7fb877fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb8800c9000 0x7fb8800c7ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35634/0 (socket says 172.21.15.98:35634) 2023-11-28T17:41:31.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.553+0000 7fb877fff640 1 -- 172.21.15.98:0/819249900 learned_addr learned my addr 172.21.15.98:0/819249900 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:31.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.553+0000 7fb87e59c640 1 --2- 172.21.15.98:0/819249900 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb880154040 0x7fb88015dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:31.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.553+0000 7fb87dd9b640 1 -- 172.21.15.98:0/819249900 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321702410 0 0) 0x7fb880159c50 con 0x7fb8800c9000 2023-11-28T17:41:31.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.553+0000 7fb87dd9b640 1 -- 172.21.15.98:0/819249900 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb86c003520 con 0x7fb8800c9000 2023-11-28T17:41:31.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb87dd9b640 1 -- 172.21.15.98:0/819249900 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2562835540 0 0) 0x7fb86c003520 con 0x7fb8800c9000 2023-11-28T17:41:31.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb87dd9b640 1 -- 172.21.15.98:0/819249900 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb880154040 msgr2=0x7fb88015dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:31.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb87dd9b640 1 --2- 172.21.15.98:0/819249900 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb880154040 0x7fb88015dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:31.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb87dd9b640 1 -- 172.21.15.98:0/819249900 >> v1:172.21.15.113:6789/0 conn(0x7fb8800c9420 legacy=0x7fb88015a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:31.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb87dd9b640 1 -- 172.21.15.98:0/819249900 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8800c9db0 con 0x7fb8800c9000 2023-11-28T17:41:31.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb87dd9b640 1 -- 172.21.15.98:0/819249900 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7fb868001670 con 0x7fb8800c9000 2023-11-28T17:41:31.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb87dd9b640 1 -- 172.21.15.98:0/819249900 >> v1:172.21.15.98:6789/0 conn(0x7fb8800c9000 legacy=0x7fb8800c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:31.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb87dd9b640 1 --2- 172.21.15.98:0/819249900 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb880154040 0x7fb86c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:31.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb87dd9b640 1 --2- 172.21.15.98:0/819249900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb86c003f70 0x7fb86c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:31.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb87dd9b640 1 --2- 172.21.15.98:0/819249900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb86c004d00 0x7fb86c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:31.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb87dd9b640 1 -- 172.21.15.98:0/819249900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8800c2ad0 con 0x7fb86c003f70 2023-11-28T17:41:31.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb87dd9b640 1 -- 172.21.15.98:0/819249900 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c00bb50 con 0x7fb86c004d00 2023-11-28T17:41:31.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb87dd9b640 1 -- 172.21.15.98:0/819249900 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c00bd10 con 0x7fb880154040 2023-11-28T17:41:31.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb877fff640 1 --2- 172.21.15.98:0/819249900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb86c003f70 0x7fb86c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:31.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb87e59c640 1 --2- 172.21.15.98:0/819249900 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb880154040 0x7fb86c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:31.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.554+0000 7fb87ed9d640 1 --2- 172.21.15.98:0/819249900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb86c004d00 0x7fb86c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:31.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.555+0000 7fb877fff640 1 -- 172.21.15.98:0/819249900 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb880154040 msgr2=0x7fb86c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:31.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.555+0000 7fb877fff640 1 --2- 172.21.15.98:0/819249900 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb880154040 0x7fb86c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:31.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.555+0000 7fb877fff640 1 -- 172.21.15.98:0/819249900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb86c004d00 msgr2=0x7fb86c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:31.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.555+0000 7fb877fff640 1 --2- 172.21.15.98:0/819249900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb86c004d00 0x7fb86c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:31.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.555+0000 7fb877fff640 1 -- 172.21.15.98:0/819249900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c003520 con 0x7fb86c003f70 2023-11-28T17:41:31.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.556+0000 7fb877fff640 1 --2- 172.21.15.98:0/819249900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb86c003f70 0x7fb86c0047c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb86800c0e0 tx=0x7fb86c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d826799bab5d694d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:31.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.556+0000 7fb87dd9b640 1 -- 172.21.15.98:0/819249900 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868005ad0 con 0x7fb86c003f70 2023-11-28T17:41:31.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.556+0000 7fb87dd9b640 1 -- 172.21.15.98:0/819249900 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb868005c50 con 0x7fb86c003f70 2023-11-28T17:41:31.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.556+0000 7fb87dd9b640 1 -- 172.21.15.98:0/819249900 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868007070 con 0x7fb86c003f70 2023-11-28T17:41:31.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.557+0000 7fb8848c9640 1 -- 172.21.15.98:0/819249900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb86c003f70 msgr2=0x7fb86c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:31.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.557+0000 7fb8848c9640 1 --2- 172.21.15.98:0/819249900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb86c003f70 0x7fb86c0047c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb86800c0e0 tx=0x7fb86c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:31.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.557+0000 7fb87ed9d640 1 -- 172.21.15.98:0/819249900 reap_dead start 2023-11-28T17:41:31.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.557+0000 7fb8848c9640 1 -- 172.21.15.98:0/819249900 shutdown_connections 2023-11-28T17:41:31.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.557+0000 7fb8848c9640 1 -- 172.21.15.98:0/819249900 >> 172.21.15.98:0/819249900 conn(0x7fb88006a4d0 msgr2=0x7fb88006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:31.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.557+0000 7fb8848c9640 1 -- 172.21.15.98:0/819249900 shutdown_connections 2023-11-28T17:41:31.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.557+0000 7fb8848c9640 1 -- 172.21.15.98:0/819249900 wait complete. 2023-11-28T17:41:31.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.558+0000 7fb8848c9640 1 Processor -- start 2023-11-28T17:41:31.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.559+0000 7fb8848c9640 1 -- start start 2023-11-28T17:41:31.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.559+0000 7fb8848c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb86c003f70 0x7fb8800cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:31.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.559+0000 7fb8848c9640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8800c3310 0x7fb880157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:31.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.559+0000 7fb8848c9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8800c3790 0x7fb880157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:31.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.559+0000 7fb8848c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8800c4fa0 con 0x7fb86c003f70 2023-11-28T17:41:31.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.560+0000 7fb8848c9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8800c4390 con 0x7fb8800c3790 2023-11-28T17:41:31.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.560+0000 7fb8848c9640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8801e4910 con 0x7fb8800c3310 2023-11-28T17:41:31.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.560+0000 7fb87e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb86c003f70 0x7fb8800cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:31.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.560+0000 7fb87e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb86c003f70 0x7fb8800cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50506/0 (socket says 172.21.15.98:50506) 2023-11-28T17:41:31.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.560+0000 7fb87e59c640 1 -- 172.21.15.98:0/2344398312 learned_addr learned my addr 172.21.15.98:0/2344398312 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:31.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.560+0000 7fb877fff640 1 --2- 172.21.15.98:0/2344398312 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8800c3310 0x7fb880157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:31.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.560+0000 7fb87e59c640 1 -- 172.21.15.98:0/2344398312 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8800c3310 msgr2=0x7fb880157a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:31.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.560+0000 7fb87e59c640 1 --2- 172.21.15.98:0/2344398312 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8800c3310 0x7fb880157a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:31.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.560+0000 7fb87e59c640 1 -- 172.21.15.98:0/2344398312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8800c3790 msgr2=0x7fb880157fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:31.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.560+0000 7fb87ed9d640 1 --2- 172.21.15.98:0/2344398312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8800c3790 0x7fb880157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:31.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.560+0000 7fb87e59c640 1 --2- 172.21.15.98:0/2344398312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8800c3790 0x7fb880157fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:31.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.560+0000 7fb87e59c640 1 -- 172.21.15.98:0/2344398312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8801e3d40 con 0x7fb86c003f70 2023-11-28T17:41:31.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.560+0000 7fb87e59c640 1 --2- 172.21.15.98:0/2344398312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb86c003f70 0x7fb8800cb400 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb870011630 tx=0x7fb870011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:31.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.561+0000 7fb876ffd640 1 -- 172.21.15.98:0/2344398312 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87000f8c0 con 0x7fb86c003f70 2023-11-28T17:41:31.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.561+0000 7fb876ffd640 1 -- 172.21.15.98:0/2344398312 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87000fa20 con 0x7fb86c003f70 2023-11-28T17:41:31.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.561+0000 7fb876ffd640 1 -- 172.21.15.98:0/2344398312 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87000fd90 con 0x7fb86c003f70 2023-11-28T17:41:31.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.561+0000 7fb8848c9640 1 -- 172.21.15.98:0/2344398312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8801e3fe0 con 0x7fb86c003f70 2023-11-28T17:41:31.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.561+0000 7fb8848c9640 1 -- 172.21.15.98:0/2344398312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8801e45a0 con 0x7fb86c003f70 2023-11-28T17:41:31.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.562+0000 7fb8848c9640 1 -- 172.21.15.98:0/2344398312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb880067bf0 con 0x7fb86c003f70 2023-11-28T17:41:31.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.563+0000 7fb876ffd640 1 -- 172.21.15.98:0/2344398312 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb870015460 con 0x7fb86c003f70 2023-11-28T17:41:31.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.564+0000 7fb876ffd640 1 --2- 172.21.15.98:0/2344398312 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb86006b2d0 0x7fb86006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:31.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.564+0000 7fb876ffd640 1 -- 172.21.15.98:0/2344398312 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb87008a430 con 0x7fb86c003f70 2023-11-28T17:41:31.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.564+0000 7fb877fff640 1 --2- 172.21.15.98:0/2344398312 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb86006b2d0 0x7fb86006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:31.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.564+0000 7fb877fff640 1 --2- 172.21.15.98:0/2344398312 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb86006b2d0 0x7fb86006d790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb868034000 tx=0x7fb868009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:31.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.568+0000 7fb876ffd640 1 -- 172.21.15.98:0/2344398312 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb870055290 con 0x7fb86c003f70 2023-11-28T17:41:31.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.790+0000 7fb8848c9640 1 -- 172.21.15.98:0/2344398312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb880074f60 con 0x7fb86c003f70 2023-11-28T17:41:31.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.791+0000 7fb876ffd640 1 -- 172.21.15.98:0/2344398312 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7fb8700588b0 con 0x7fb86c003f70 2023-11-28T17:41:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:31.793 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-11-28T17:41:31.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.793+0000 7fb8848c9640 1 -- 172.21.15.98:0/2344398312 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb86006b2d0 msgr2=0x7fb86006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:31.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.793+0000 7fb8848c9640 1 --2- 172.21.15.98:0/2344398312 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb86006b2d0 0x7fb86006d790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb868034000 tx=0x7fb868009030 comp rx=0 tx=0).stop 2023-11-28T17:41:31.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.793+0000 7fb8848c9640 1 -- 172.21.15.98:0/2344398312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb86c003f70 msgr2=0x7fb8800cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:31.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.793+0000 7fb8848c9640 1 --2- 172.21.15.98:0/2344398312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb86c003f70 0x7fb8800cb400 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb870011630 tx=0x7fb870011af0 comp rx=0 tx=0).stop 2023-11-28T17:41:31.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.793+0000 7fb8848c9640 1 -- 172.21.15.98:0/2344398312 shutdown_connections 2023-11-28T17:41:31.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.793+0000 7fb8848c9640 1 --2- 172.21.15.98:0/2344398312 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb86006b2d0 0x7fb86006d790 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:31.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.793+0000 7fb8848c9640 1 --2- 172.21.15.98:0/2344398312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8800c3790 0x7fb880157fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:31.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.793+0000 7fb8848c9640 1 --2- 172.21.15.98:0/2344398312 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8800c3310 0x7fb880157a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:31.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.793+0000 7fb8848c9640 1 --2- 172.21.15.98:0/2344398312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb86c003f70 0x7fb8800cb400 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:31.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.793+0000 7fb8848c9640 1 -- 172.21.15.98:0/2344398312 >> 172.21.15.98:0/2344398312 conn(0x7fb88006a4d0 msgr2=0x7fb880150c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:31.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.793+0000 7fb8848c9640 1 -- 172.21.15.98:0/2344398312 shutdown_connections 2023-11-28T17:41:31.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:31.793+0000 7fb8848c9640 1 -- 172.21.15.98:0/2344398312 wait complete. 2023-11-28T17:41:34.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T17:41:35.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.031+0000 7f1ab0a09640 1 Processor -- start 2023-11-28T17:41:35.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.032+0000 7f1ab0a09640 1 -- start start 2023-11-28T17:41:35.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.033+0000 7f1ab0a09640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1aac153bb0 0x7f1aac15a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:35.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.033+0000 7f1ab0a09640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1aac0ca390 0x7f1aac0ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:35.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.033+0000 7f1ab0a09640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1aac0cf280 0x7f1aac0cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:35.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.033+0000 7f1ab0a09640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aac15d120 con 0x7f1aac153bb0 2023-11-28T17:41:35.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.033+0000 7f1ab0a09640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aac15d330 con 0x7f1aac0ca390 2023-11-28T17:41:35.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.033+0000 7f1ab0a09640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac15d610 con 0x7f1aac0cf280 2023-11-28T17:41:35.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.034+0000 7f1aab7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1aac153bb0 0x7f1aac15a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:35.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.034+0000 7f1aab7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1aac153bb0 0x7f1aac15a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50516/0 (socket says 172.21.15.98:50516) 2023-11-28T17:41:35.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.034+0000 7f1aab7fe640 1 -- 172.21.15.98:0/67353853 learned_addr learned my addr 172.21.15.98:0/67353853 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:35.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.034+0000 7f1aabfff640 1 --2- 172.21.15.98:0/67353853 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1aac0cf280 0x7f1aac0cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:35.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.034+0000 7f1aaaffd640 1 --2- 172.21.15.98:0/67353853 >> v2:172.21.15.113:3300/0 conn(0x7f1aac0ca390 0x7f1aac0ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:35.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.034+0000 7f1aab7fe640 1 -- 172.21.15.98:0/67353853 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1aac0cf280 msgr2=0x7f1aac0cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:35.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.034+0000 7f1aab7fe640 1 --2- 172.21.15.98:0/67353853 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1aac0cf280 0x7f1aac0cf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:35.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.034+0000 7f1aab7fe640 1 -- 172.21.15.98:0/67353853 >> v2:172.21.15.113:3300/0 conn(0x7f1aac0ca390 msgr2=0x7f1aac0ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:35.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.034+0000 7f1aab7fe640 1 --2- 172.21.15.98:0/67353853 >> v2:172.21.15.113:3300/0 conn(0x7f1aac0ca390 0x7f1aac0ca7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:35.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.034+0000 7f1aab7fe640 1 -- 172.21.15.98:0/67353853 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aac15d8f0 con 0x7f1aac153bb0 2023-11-28T17:41:35.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.035+0000 7f1aab7fe640 1 --2- 172.21.15.98:0/67353853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aac153bb0 0x7f1aac15a6d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1aac0ca320 tx=0x7f1a9c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448d1bac85f49247 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:35.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.035+0000 7f1aaa7fc640 1 -- 172.21.15.98:0/67353853 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c030860 con 0x7f1aac153bb0 2023-11-28T17:41:35.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.035+0000 7f1aaa7fc640 1 -- 172.21.15.98:0/67353853 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9c0309c0 con 0x7f1aac153bb0 2023-11-28T17:41:35.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.035+0000 7f1aaa7fc640 1 -- 172.21.15.98:0/67353853 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c030d50 con 0x7f1aac153bb0 2023-11-28T17:41:35.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.036+0000 7f1ab0a09640 1 -- 172.21.15.98:0/67353853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aac153bb0 msgr2=0x7f1aac15a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:35.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.036+0000 7f1ab0a09640 1 --2- 172.21.15.98:0/67353853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aac153bb0 0x7f1aac15a6d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1aac0ca320 tx=0x7f1a9c02b9a0 comp rx=0 tx=0).stop 2023-11-28T17:41:35.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.036+0000 7f1ab0a09640 1 -- 172.21.15.98:0/67353853 shutdown_connections 2023-11-28T17:41:35.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.036+0000 7f1ab0a09640 1 --2- 172.21.15.98:0/67353853 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1aac0cf280 0x7f1aac0cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:35.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.036+0000 7f1ab0a09640 1 --2- 172.21.15.98:0/67353853 >> v2:172.21.15.113:3300/0 conn(0x7f1aac0ca390 0x7f1aac0ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:35.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.036+0000 7f1ab0a09640 1 --2- 172.21.15.98:0/67353853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aac153bb0 0x7f1aac15a6d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:35.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.036+0000 7f1ab0a09640 1 -- 172.21.15.98:0/67353853 >> 172.21.15.98:0/67353853 conn(0x7f1aac071c00 msgr2=0x7f1aac072010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:35.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.036+0000 7f1ab0a09640 1 -- 172.21.15.98:0/67353853 shutdown_connections 2023-11-28T17:41:35.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.036+0000 7f1ab0a09640 1 -- 172.21.15.98:0/67353853 wait complete. 2023-11-28T17:41:35.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.038+0000 7f1ab0a09640 1 Processor -- start 2023-11-28T17:41:35.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.038+0000 7f1ab0a09640 1 -- start start 2023-11-28T17:41:35.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.039+0000 7f1ab0a09640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aac0ca390 0x7f1aac1519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:35.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.039+0000 7f1ab0a09640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1aac0cf280 0x7f1aac1f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:35.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.039+0000 7f1ab0a09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aac153bb0 0x7f1aac151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:35.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.039+0000 7f1ab0a09640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac14fcf0 con 0x7f1aac153bb0 2023-11-28T17:41:35.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.039+0000 7f1ab0a09640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac14b760 con 0x7f1aac0ca390 2023-11-28T17:41:35.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.039+0000 7f1ab0a09640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac144760 con 0x7f1aac0cf280 2023-11-28T17:41:35.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.040+0000 7f1aabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aac153bb0 0x7f1aac151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:35.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.040+0000 7f1aabfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aac153bb0 0x7f1aac151170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50524/0 (socket says 172.21.15.98:50524) 2023-11-28T17:41:35.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.040+0000 7f1aabfff640 1 -- 172.21.15.98:0/571018620 learned_addr learned my addr 172.21.15.98:0/571018620 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:35.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.040+0000 7f1aab7fe640 1 --2- 172.21.15.98:0/571018620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aac0ca390 0x7f1aac1519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:35.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.040+0000 7f1aaaffd640 1 --2- 172.21.15.98:0/571018620 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1aac0cf280 0x7f1aac1f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:35.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.040+0000 7f1aabfff640 1 -- 172.21.15.98:0/571018620 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1aac0cf280 msgr2=0x7f1aac1f3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:35.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.040+0000 7f1aabfff640 1 --2- 172.21.15.98:0/571018620 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1aac0cf280 0x7f1aac1f3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:35.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.040+0000 7f1aabfff640 1 -- 172.21.15.98:0/571018620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aac0ca390 msgr2=0x7f1aac1519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:35.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.040+0000 7f1aabfff640 1 --2- 172.21.15.98:0/571018620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aac0ca390 0x7f1aac1519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:35.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.040+0000 7f1aabfff640 1 -- 172.21.15.98:0/571018620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c0094d0 con 0x7f1aac153bb0 2023-11-28T17:41:35.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.041+0000 7f1aabfff640 1 --2- 172.21.15.98:0/571018620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aac153bb0 0x7f1aac151170 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400c820 tx=0x7f1aa400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:35.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.041+0000 7f1aa3fff640 1 -- 172.21.15.98:0/571018620 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4004020 con 0x7f1aac153bb0 2023-11-28T17:41:35.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.041+0000 7f1aa3fff640 1 -- 172.21.15.98:0/571018620 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa4004180 con 0x7f1aac153bb0 2023-11-28T17:41:35.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.041+0000 7f1aa3fff640 1 -- 172.21.15.98:0/571018620 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa40077d0 con 0x7f1aac153bb0 2023-11-28T17:41:35.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.041+0000 7f1ab0a09640 1 -- 172.21.15.98:0/571018620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aac14f4b0 con 0x7f1aac153bb0 2023-11-28T17:41:35.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.041+0000 7f1ab0a09640 1 -- 172.21.15.98:0/571018620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aac1f3c90 con 0x7f1aac153bb0 2023-11-28T17:41:35.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.042+0000 7f1ab0a09640 1 -- 172.21.15.98:0/571018620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aac14f790 con 0x7f1aac153bb0 2023-11-28T17:41:35.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.043+0000 7f1aa3fff640 1 -- 172.21.15.98:0/571018620 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f1aa4007930 con 0x7f1aac153bb0 2023-11-28T17:41:35.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.044+0000 7f1aa3fff640 1 --2- 172.21.15.98:0/571018620 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1a8c06b2a0 0x7f1a8c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:35.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.044+0000 7f1aa3fff640 1 -- 172.21.15.98:0/571018620 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1aa4089980 con 0x7f1aac153bb0 2023-11-28T17:41:35.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.044+0000 7f1aab7fe640 1 --2- 172.21.15.98:0/571018620 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1a8c06b2a0 0x7f1a8c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:35.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.045+0000 7f1aab7fe640 1 --2- 172.21.15.98:0/571018620 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1a8c06b2a0 0x7f1a8c06d760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c036000 tx=0x7f1a9c03b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:35.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.050+0000 7f1aa3fff640 1 -- 172.21.15.98:0/571018620 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1aa40546c0 con 0x7f1aac153bb0 2023-11-28T17:41:35.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.283+0000 7f1ab0a09640 1 -- 172.21.15.98:0/571018620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1aac0ca060 con 0x7f1aac153bb0 2023-11-28T17:41:35.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.284+0000 7f1aa3fff640 1 -- 172.21.15.98:0/571018620 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f1aa401e080 con 0x7f1aac153bb0 2023-11-28T17:41:35.286 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:35.286 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-11-28T17:41:35.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.287+0000 7f1ab0a09640 1 -- 172.21.15.98:0/571018620 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1a8c06b2a0 msgr2=0x7f1a8c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:35.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.287+0000 7f1ab0a09640 1 --2- 172.21.15.98:0/571018620 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1a8c06b2a0 0x7f1a8c06d760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c036000 tx=0x7f1a9c03b040 comp rx=0 tx=0).stop 2023-11-28T17:41:35.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.287+0000 7f1ab0a09640 1 -- 172.21.15.98:0/571018620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aac153bb0 msgr2=0x7f1aac151170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:35.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.287+0000 7f1ab0a09640 1 --2- 172.21.15.98:0/571018620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aac153bb0 0x7f1aac151170 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400c820 tx=0x7f1aa400cce0 comp rx=0 tx=0).stop 2023-11-28T17:41:35.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.287+0000 7f1ab0a09640 1 -- 172.21.15.98:0/571018620 shutdown_connections 2023-11-28T17:41:35.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.287+0000 7f1ab0a09640 1 --2- 172.21.15.98:0/571018620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1aac153bb0 0x7f1aac151170 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:35.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.287+0000 7f1ab0a09640 1 --2- 172.21.15.98:0/571018620 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1aac0cf280 0x7f1aac1f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:35.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.287+0000 7f1ab0a09640 1 --2- 172.21.15.98:0/571018620 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1a8c06b2a0 0x7f1a8c06d760 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:35.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.287+0000 7f1ab0a09640 1 --2- 172.21.15.98:0/571018620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1aac0ca390 0x7f1aac1519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:35.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.287+0000 7f1ab0a09640 1 -- 172.21.15.98:0/571018620 >> 172.21.15.98:0/571018620 conn(0x7f1aac071c00 msgr2=0x7f1aac071fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:35.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.287+0000 7f1ab0a09640 1 -- 172.21.15.98:0/571018620 shutdown_connections 2023-11-28T17:41:35.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:35.287+0000 7f1ab0a09640 1 -- 172.21.15.98:0/571018620 wait complete. 2023-11-28T17:41:35.879 INFO:tasks.ceph.mds.b.smithi113.stderr:2023-11-28T17:41:35.878+0000 7f6aad4d5640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-11-28T17:41:36.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:41:37.007 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:41:38.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T17:41:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.524+0000 7f98e7a15640 1 Processor -- start 2023-11-28T17:41:38.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.524+0000 7f98e7a15640 1 -- start start 2023-11-28T17:41:38.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.526+0000 7f98e7a15640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98e0151d80 0x7f98e0157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.526+0000 7f98e7a15640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f98e015bc10 0x7f98e00c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.526+0000 7f98e7a15640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f98e00c6140 0x7f98e00c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.526+0000 7f98e7a15640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e015b150 con 0x7f98e015bc10 2023-11-28T17:41:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.526+0000 7f98e7a15640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e006e1f0 con 0x7f98e00c6140 2023-11-28T17:41:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.526+0000 7f98e7a15640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e006e4d0 con 0x7f98e0151d80 2023-11-28T17:41:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.526+0000 7f98e4fb0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f98e015bc10 0x7f98e00c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.526+0000 7f98e4fb0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f98e015bc10 0x7f98e00c5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50538/0 (socket says 172.21.15.98:50538) 2023-11-28T17:41:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.526+0000 7f98e4fb0640 1 -- 172.21.15.98:0/2096816081 learned_addr learned my addr 172.21.15.98:0/2096816081 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.526+0000 7f98e57b1640 1 --2- 172.21.15.98:0/2096816081 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98e0151d80 0x7f98e0157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:38.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.527+0000 7f98e5fb2640 1 --2- 172.21.15.98:0/2096816081 >> v2:172.21.15.113:3300/0 conn(0x7f98e00c6140 0x7f98e00c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.527+0000 7f98e4fb0640 1 -- 172.21.15.98:0/2096816081 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98e0151d80 msgr2=0x7f98e0157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.527+0000 7f98e4fb0640 1 --2- 172.21.15.98:0/2096816081 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98e0151d80 0x7f98e0157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.527+0000 7f98e4fb0640 1 -- 172.21.15.98:0/2096816081 >> v2:172.21.15.113:3300/0 conn(0x7f98e00c6140 msgr2=0x7f98e00c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.527+0000 7f98e4fb0640 1 --2- 172.21.15.98:0/2096816081 >> v2:172.21.15.113:3300/0 conn(0x7f98e00c6140 0x7f98e00c4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.527+0000 7f98e4fb0640 1 -- 172.21.15.98:0/2096816081 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e006e7b0 con 0x7f98e015bc10 2023-11-28T17:41:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.528+0000 7f98e4fb0640 1 --2- 172.21.15.98:0/2096816081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98e015bc10 0x7f98e00c5c00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f98e00c78a0 tx=0x7f98d002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c3570087ff22f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.528+0000 7f98d7fff640 1 -- 172.21.15.98:0/2096816081 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d0030860 con 0x7f98e015bc10 2023-11-28T17:41:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.528+0000 7f98d7fff640 1 -- 172.21.15.98:0/2096816081 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d00309c0 con 0x7f98e015bc10 2023-11-28T17:41:38.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.528+0000 7f98d7fff640 1 -- 172.21.15.98:0/2096816081 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d0030d50 con 0x7f98e015bc10 2023-11-28T17:41:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.529+0000 7f98e7a15640 1 -- 172.21.15.98:0/2096816081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98e015bc10 msgr2=0x7f98e00c5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.529+0000 7f98e7a15640 1 --2- 172.21.15.98:0/2096816081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98e015bc10 0x7f98e00c5c00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f98e00c78a0 tx=0x7f98d002b9a0 comp rx=0 tx=0).stop 2023-11-28T17:41:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.529+0000 7f98e7a15640 1 -- 172.21.15.98:0/2096816081 shutdown_connections 2023-11-28T17:41:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.529+0000 7f98e7a15640 1 --2- 172.21.15.98:0/2096816081 >> v2:172.21.15.113:3300/0 conn(0x7f98e00c6140 0x7f98e00c4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.529+0000 7f98e7a15640 1 --2- 172.21.15.98:0/2096816081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98e015bc10 0x7f98e00c5c00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.529+0000 7f98e7a15640 1 --2- 172.21.15.98:0/2096816081 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98e0151d80 0x7f98e0157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.529+0000 7f98e7a15640 1 -- 172.21.15.98:0/2096816081 >> 172.21.15.98:0/2096816081 conn(0x7f98e006a4d0 msgr2=0x7f98e006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.529+0000 7f98e7a15640 1 -- 172.21.15.98:0/2096816081 shutdown_connections 2023-11-28T17:41:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.529+0000 7f98e7a15640 1 -- 172.21.15.98:0/2096816081 wait complete. 2023-11-28T17:41:38.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.531+0000 7f98e7a15640 1 Processor -- start 2023-11-28T17:41:38.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.531+0000 7f98e7a15640 1 -- start start 2023-11-28T17:41:38.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.532+0000 7f98e7a15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98e00c6140 0x7f98e015a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:38.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.532+0000 7f98e7a15640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98e0151d80 0x7f98e01e4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:38.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.532+0000 7f98e7a15640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98e015bc10 0x7f98e01487f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.532+0000 7f98e7a15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e01f3e50 con 0x7f98e00c6140 2023-11-28T17:41:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.532+0000 7f98e7a15640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e0149360 con 0x7f98e0151d80 2023-11-28T17:41:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.532+0000 7f98e7a15640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e01f2ff0 con 0x7f98e015bc10 2023-11-28T17:41:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.532+0000 7f98e57b1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98e00c6140 0x7f98e015a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.532+0000 7f98e57b1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98e00c6140 0x7f98e015a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50546/0 (socket says 172.21.15.98:50546) 2023-11-28T17:41:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.532+0000 7f98e57b1640 1 -- 172.21.15.98:0/228514992 learned_addr learned my addr 172.21.15.98:0/228514992 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.532+0000 7f98e4fb0640 1 --2- 172.21.15.98:0/228514992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98e0151d80 0x7f98e01e4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.532+0000 7f98e5fb2640 1 --2- 172.21.15.98:0/228514992 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98e015bc10 0x7f98e01487f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:38.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.533+0000 7f98e57b1640 1 -- 172.21.15.98:0/228514992 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98e015bc10 msgr2=0x7f98e01487f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.533+0000 7f98e57b1640 1 --2- 172.21.15.98:0/228514992 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98e015bc10 0x7f98e01487f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.533+0000 7f98e57b1640 1 -- 172.21.15.98:0/228514992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98e0151d80 msgr2=0x7f98e01e4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.533+0000 7f98e57b1640 1 --2- 172.21.15.98:0/228514992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98e0151d80 0x7f98e01e4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.533+0000 7f98e57b1640 1 -- 172.21.15.98:0/228514992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d00094d0 con 0x7f98e00c6140 2023-11-28T17:41:38.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.533+0000 7f98e4fb0640 1 --2- 172.21.15.98:0/228514992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98e0151d80 0x7f98e01e4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:38.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.533+0000 7f98e57b1640 1 --2- 172.21.15.98:0/228514992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98e00c6140 0x7f98e015a990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00d800 tx=0x7f98dc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:38.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.533+0000 7f98d5ffb640 1 -- 172.21.15.98:0/228514992 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc00b6c0 con 0x7f98e00c6140 2023-11-28T17:41:38.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.533+0000 7f98d5ffb640 1 -- 172.21.15.98:0/228514992 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98dc00b820 con 0x7f98e00c6140 2023-11-28T17:41:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.533+0000 7f98d5ffb640 1 -- 172.21.15.98:0/228514992 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc00bbb0 con 0x7f98e00c6140 2023-11-28T17:41:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.533+0000 7f98e7a15640 1 -- 172.21.15.98:0/228514992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e01f32d0 con 0x7f98e00c6140 2023-11-28T17:41:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.533+0000 7f98e7a15640 1 -- 172.21.15.98:0/228514992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e01f3890 con 0x7f98e00c6140 2023-11-28T17:41:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.534+0000 7f98e7a15640 1 -- 172.21.15.98:0/228514992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e0067bf0 con 0x7f98e00c6140 2023-11-28T17:41:38.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.535+0000 7f98d5ffb640 1 -- 172.21.15.98:0/228514992 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f98dc00bd10 con 0x7f98e00c6140 2023-11-28T17:41:38.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.536+0000 7f98d5ffb640 1 --2- 172.21.15.98:0/228514992 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f98c406b2a0 0x7f98c406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.536+0000 7f98d5ffb640 1 -- 172.21.15.98:0/228514992 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98dc08a2a0 con 0x7f98e00c6140 2023-11-28T17:41:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.537+0000 7f98e4fb0640 1 --2- 172.21.15.98:0/228514992 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f98c406b2a0 0x7f98c406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:38.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.537+0000 7f98e4fb0640 1 --2- 172.21.15.98:0/228514992 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f98c406b2a0 0x7f98c406d760 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f98d0036000 tx=0x7f98d003b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:38.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.541+0000 7f98d5ffb640 1 -- 172.21.15.98:0/228514992 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98dc054fe0 con 0x7f98e00c6140 2023-11-28T17:41:38.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.768+0000 7f98e7a15640 1 -- 172.21.15.98:0/228514992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f98e006b2d0 con 0x7f98e00c6140 2023-11-28T17:41:38.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.769+0000 7f98d5ffb640 1 -- 172.21.15.98:0/228514992 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f98dc058600 con 0x7f98e00c6140 2023-11-28T17:41:38.771 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:38.771 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-11-28T17:41:38.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.772+0000 7f98e7a15640 1 -- 172.21.15.98:0/228514992 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f98c406b2a0 msgr2=0x7f98c406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:38.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.772+0000 7f98e7a15640 1 --2- 172.21.15.98:0/228514992 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f98c406b2a0 0x7f98c406d760 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f98d0036000 tx=0x7f98d003b040 comp rx=0 tx=0).stop 2023-11-28T17:41:38.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.772+0000 7f98e7a15640 1 -- 172.21.15.98:0/228514992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98e00c6140 msgr2=0x7f98e015a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:38.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.772+0000 7f98e7a15640 1 --2- 172.21.15.98:0/228514992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98e00c6140 0x7f98e015a990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00d800 tx=0x7f98dc00dcc0 comp rx=0 tx=0).stop 2023-11-28T17:41:38.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.772+0000 7f98e7a15640 1 -- 172.21.15.98:0/228514992 shutdown_connections 2023-11-28T17:41:38.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.772+0000 7f98e7a15640 1 --2- 172.21.15.98:0/228514992 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f98e015bc10 0x7f98e01487f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:38.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.772+0000 7f98e7a15640 1 --2- 172.21.15.98:0/228514992 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f98c406b2a0 0x7f98c406d760 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:38.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.772+0000 7f98e7a15640 1 --2- 172.21.15.98:0/228514992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98e0151d80 0x7f98e01e4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:38.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.772+0000 7f98e7a15640 1 --2- 172.21.15.98:0/228514992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98e00c6140 0x7f98e015a990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:38.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.772+0000 7f98e7a15640 1 -- 172.21.15.98:0/228514992 >> 172.21.15.98:0/228514992 conn(0x7f98e006a4d0 msgr2=0x7f98e00c4910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:38.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.773+0000 7f98e7a15640 1 -- 172.21.15.98:0/228514992 shutdown_connections 2023-11-28T17:41:38.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:38.773+0000 7f98e7a15640 1 -- 172.21.15.98:0/228514992 wait complete. 2023-11-28T17:41:40.880 INFO:tasks.ceph.mds.b.smithi113.stderr:2023-11-28T17:41:40.879+0000 7f6aad4d5640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-11-28T17:41:41.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T17:41:42.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.011+0000 7f8195d89640 1 Processor -- start 2023-11-28T17:41:42.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.012+0000 7f8195d89640 1 -- start start 2023-11-28T17:41:42.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.013+0000 7f8195d89640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8190157b60 0x7f819015bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.013+0000 7f8195d89640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f81900c4910 0x7f81900c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.013+0000 7f8195d89640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819015b150 con 0x7f8190151d80 2023-11-28T17:41:42.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.013+0000 7f8195d89640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8190157f90 con 0x7f81900c4910 2023-11-28T17:41:42.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.013+0000 7f8195d89640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819015c150 con 0x7f8190157b60 2023-11-28T17:41:42.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.013+0000 7f818ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f81900c4910 0x7f81900c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.013+0000 7f818ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f81900c4910 0x7f81900c4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36634/0 (socket says 172.21.15.98:36634) 2023-11-28T17:41:42.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.014+0000 7f818ffff640 1 -- 172.21.15.98:0/3569873247 learned_addr learned my addr 172.21.15.98:0/3569873247 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:42.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.014+0000 7f818effd640 1 --2- 172.21.15.98:0/3569873247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8190157b60 0x7f819015bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.014+0000 7f818ffff640 1 -- 172.21.15.98:0/3569873247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8190157b60 msgr2=0x7f819015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:42.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.014+0000 7f818ffff640 1 --2- 172.21.15.98:0/3569873247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8190157b60 0x7f819015bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.014+0000 7f818ffff640 1 -- 172.21.15.98:0/3569873247 >> v1:172.21.15.113:6789/0 conn(0x7f8190151d80 legacy=0x7f81900c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:42.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.014+0000 7f818ffff640 1 -- 172.21.15.98:0/3569873247 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819006e1f0 con 0x7f81900c4910 2023-11-28T17:41:42.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.014+0000 7f818effd640 1 --2- 172.21.15.98:0/3569873247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8190157b60 0x7f819015bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:42.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.015+0000 7f818ffff640 1 --2- 172.21.15.98:0/3569873247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81900c4910 0x7f81900c4da0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8188009e50 tx=0x7f818802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74d6388d6514df0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:42.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.015+0000 7f818e7fc640 1 -- 172.21.15.98:0/3569873247 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818802cb00 con 0x7f81900c4910 2023-11-28T17:41:42.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.015+0000 7f818e7fc640 1 -- 172.21.15.98:0/3569873247 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818802cc60 con 0x7f81900c4910 2023-11-28T17:41:42.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.015+0000 7f818e7fc640 1 -- 172.21.15.98:0/3569873247 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81880343f0 con 0x7f81900c4910 2023-11-28T17:41:42.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.016+0000 7f8195d89640 1 -- 172.21.15.98:0/3569873247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81900c4910 msgr2=0x7f81900c4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:42.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.016+0000 7f8195d89640 1 --2- 172.21.15.98:0/3569873247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81900c4910 0x7f81900c4da0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8188009e50 tx=0x7f818802bf60 comp rx=0 tx=0).stop 2023-11-28T17:41:42.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.016+0000 7f8195d89640 1 -- 172.21.15.98:0/3569873247 shutdown_connections 2023-11-28T17:41:42.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.016+0000 7f8195d89640 1 --2- 172.21.15.98:0/3569873247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81900c4910 0x7f81900c4da0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.016+0000 7f8195d89640 1 --2- 172.21.15.98:0/3569873247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8190157b60 0x7f819015bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.016+0000 7f8195d89640 1 -- 172.21.15.98:0/3569873247 >> 172.21.15.98:0/3569873247 conn(0x7f819006a4d0 msgr2=0x7f819006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:42.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.016+0000 7f8195d89640 1 -- 172.21.15.98:0/3569873247 shutdown_connections 2023-11-28T17:41:42.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.016+0000 7f8195d89640 1 -- 172.21.15.98:0/3569873247 wait complete. 2023-11-28T17:41:42.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.018+0000 7f8195d89640 1 Processor -- start 2023-11-28T17:41:42.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.018+0000 7f8195d89640 1 -- start start 2023-11-28T17:41:42.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.019+0000 7f8195d89640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81900c4910 0x7f819014dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.019+0000 7f8195d89640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8190151d80 0x7f8190155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.019+0000 7f8195d89640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8190157b60 0x7f81901f1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.019+0000 7f8195d89640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819014e910 con 0x7f8190151d80 2023-11-28T17:41:42.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.019+0000 7f8195d89640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819014afd0 con 0x7f8190157b60 2023-11-28T17:41:42.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.019+0000 7f8195d89640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819014e630 con 0x7f81900c4910 2023-11-28T17:41:42.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.019+0000 7f818effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8190151d80 0x7f8190155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.019+0000 7f818effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8190151d80 0x7f8190155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36640/0 (socket says 172.21.15.98:36640) 2023-11-28T17:41:42.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.019+0000 7f818effd640 1 -- 172.21.15.98:0/1943857702 learned_addr learned my addr 172.21.15.98:0/1943857702 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:42.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.019+0000 7f818ffff640 1 --2- 172.21.15.98:0/1943857702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8190157b60 0x7f81901f1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.019+0000 7f818effd640 1 -- 172.21.15.98:0/1943857702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81900c4910 msgr2=0x7f819014dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:42.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.019+0000 7f818effd640 1 --2- 172.21.15.98:0/1943857702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81900c4910 0x7f819014dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.019+0000 7f818effd640 1 -- 172.21.15.98:0/1943857702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8190157b60 msgr2=0x7f81901f1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:42.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.019+0000 7f818effd640 1 --2- 172.21.15.98:0/1943857702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8190157b60 0x7f81901f1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.020+0000 7f818effd640 1 -- 172.21.15.98:0/1943857702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8188009b30 con 0x7f8190151d80 2023-11-28T17:41:42.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.020+0000 7f818effd640 1 --2- 172.21.15.98:0/1943857702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8190151d80 0x7f8190155a10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f818400c7f0 tx=0x7f818400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:42.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.020+0000 7f8177fff640 1 -- 172.21.15.98:0/1943857702 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8184007a30 con 0x7f8190151d80 2023-11-28T17:41:42.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.020+0000 7f8177fff640 1 -- 172.21.15.98:0/1943857702 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8184007b90 con 0x7f8190151d80 2023-11-28T17:41:42.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.020+0000 7f8177fff640 1 -- 172.21.15.98:0/1943857702 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818400ce40 con 0x7f8190151d80 2023-11-28T17:41:42.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.020+0000 7f8195d89640 1 -- 172.21.15.98:0/1943857702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819014a6a0 con 0x7f8190151d80 2023-11-28T17:41:42.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.020+0000 7f8195d89640 1 -- 172.21.15.98:0/1943857702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819014ac60 con 0x7f8190151d80 2023-11-28T17:41:42.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.021+0000 7f8195d89640 1 -- 172.21.15.98:0/1943857702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8190067bf0 con 0x7f8190151d80 2023-11-28T17:41:42.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.022+0000 7f8177fff640 1 -- 172.21.15.98:0/1943857702 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f81840113f0 con 0x7f8190151d80 2023-11-28T17:41:42.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.023+0000 7f8177fff640 1 --2- 172.21.15.98:0/1943857702 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f817c06b2a0 0x7f817c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.023+0000 7f8177fff640 1 -- 172.21.15.98:0/1943857702 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8184089e40 con 0x7f8190151d80 2023-11-28T17:41:42.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.023+0000 7f818f7fe640 1 --2- 172.21.15.98:0/1943857702 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f817c06b2a0 0x7f817c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.024+0000 7f818f7fe640 1 --2- 172.21.15.98:0/1943857702 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f817c06b2a0 0x7f817c06d760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8180003cc0 tx=0x7f81800079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:42.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.027+0000 7f8177fff640 1 -- 172.21.15.98:0/1943857702 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8184054c10 con 0x7f8190151d80 2023-11-28T17:41:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.255+0000 7f8195d89640 1 -- 172.21.15.98:0/1943857702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f819006b2d0 con 0x7f8190151d80 2023-11-28T17:41:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.256+0000 7f8177fff640 1 -- 172.21.15.98:0/1943857702 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f818401f040 con 0x7f8190151d80 2023-11-28T17:41:42.258 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:42.258 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-28T17:41:42.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.259+0000 7f8195d89640 1 -- 172.21.15.98:0/1943857702 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f817c06b2a0 msgr2=0x7f817c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:42.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.259+0000 7f8195d89640 1 --2- 172.21.15.98:0/1943857702 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f817c06b2a0 0x7f817c06d760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8180003cc0 tx=0x7f81800079b0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.259+0000 7f8195d89640 1 -- 172.21.15.98:0/1943857702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8190151d80 msgr2=0x7f8190155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:42.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.259+0000 7f8195d89640 1 --2- 172.21.15.98:0/1943857702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8190151d80 0x7f8190155a10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f818400c7f0 tx=0x7f818400cbf0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.259+0000 7f8195d89640 1 -- 172.21.15.98:0/1943857702 shutdown_connections 2023-11-28T17:41:42.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.259+0000 7f8195d89640 1 --2- 172.21.15.98:0/1943857702 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f817c06b2a0 0x7f817c06d760 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.259+0000 7f8195d89640 1 --2- 172.21.15.98:0/1943857702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8190157b60 0x7f81901f1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.259+0000 7f8195d89640 1 --2- 172.21.15.98:0/1943857702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8190151d80 0x7f8190155a10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.259+0000 7f8195d89640 1 --2- 172.21.15.98:0/1943857702 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81900c4910 0x7f819014dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.259+0000 7f8195d89640 1 -- 172.21.15.98:0/1943857702 >> 172.21.15.98:0/1943857702 conn(0x7f819006a4d0 msgr2=0x7f819014f460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:42.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.259+0000 7f8195d89640 1 -- 172.21.15.98:0/1943857702 shutdown_connections 2023-11-28T17:41:42.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.259+0000 7f8195d89640 1 -- 172.21.15.98:0/1943857702 wait complete. 2023-11-28T17:41:42.279 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-28T17:41:42.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:41:42.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.474+0000 7fabd9c90640 1 Processor -- start 2023-11-28T17:41:42.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.474+0000 7fabd9c90640 1 -- start start 2023-11-28T17:41:42.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.476+0000 7fabd9c90640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabd40c7c50 0x7fabd415de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.476+0000 7fabd9c90640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fabd40c7140 0x7fabd4156f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.476+0000 7fabd9c90640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd415a230 con 0x7fabd40c7870 2023-11-28T17:41:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.476+0000 7fabd9c90640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd415a510 con 0x7fabd40c7140 2023-11-28T17:41:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.476+0000 7fabd9c90640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd415a7f0 con 0x7fabd40c7c50 2023-11-28T17:41:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.476+0000 7fabd3fff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fabd40c7140 0x7fabd4156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.477+0000 7fabd3fff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fabd40c7140 0x7fabd4156f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:39830/0 (socket says 172.21.15.98:39830) 2023-11-28T17:41:42.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.477+0000 7fabd3fff640 1 -- 172.21.15.98:0/38161712 learned_addr learned my addr 172.21.15.98:0/38161712 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.477+0000 7fabcbfff640 1 --2- 172.21.15.98:0/38161712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabd40c7c50 0x7fabd415de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.477+0000 7fabd3fff640 1 -- 172.21.15.98:0/38161712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabd40c7c50 msgr2=0x7fabd415de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.477+0000 7fabd3fff640 1 --2- 172.21.15.98:0/38161712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabd40c7c50 0x7fabd415de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.477+0000 7fabd3fff640 1 -- 172.21.15.98:0/38161712 >> v1:172.21.15.113:6789/0 conn(0x7fabd40c7870 legacy=0x7fabd4071700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:42.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.477+0000 7fabd3fff640 1 -- 172.21.15.98:0/38161712 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd415aad0 con 0x7fabd40c7140 2023-11-28T17:41:42.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.478+0000 7fabd3fff640 1 --2- 172.21.15.98:0/38161712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd40c7140 0x7fabd4156f70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fabcc009e50 tx=0x7fabcc02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d12070ac0d220a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.479+0000 7fabd2ffd640 1 -- 172.21.15.98:0/38161712 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc02ca70 con 0x7fabd40c7140 2023-11-28T17:41:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.479+0000 7fabd2ffd640 1 -- 172.21.15.98:0/38161712 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabcc02cbd0 con 0x7fabd40c7140 2023-11-28T17:41:42.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.479+0000 7fabd2ffd640 1 -- 172.21.15.98:0/38161712 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc0343f0 con 0x7fabd40c7140 2023-11-28T17:41:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.480+0000 7fabd9c90640 1 -- 172.21.15.98:0/38161712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd40c7140 msgr2=0x7fabd4156f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.480+0000 7fabd9c90640 1 --2- 172.21.15.98:0/38161712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd40c7140 0x7fabd4156f70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fabcc009e50 tx=0x7fabcc02bed0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.480+0000 7fabd9c90640 1 -- 172.21.15.98:0/38161712 shutdown_connections 2023-11-28T17:41:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.480+0000 7fabd9c90640 1 --2- 172.21.15.98:0/38161712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd40c7140 0x7fabd4156f70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.480+0000 7fabd9c90640 1 --2- 172.21.15.98:0/38161712 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabd40c7c50 0x7fabd415de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.480+0000 7fabd9c90640 1 -- 172.21.15.98:0/38161712 >> 172.21.15.98:0/38161712 conn(0x7fabd40c8d10 msgr2=0x7fabd40c90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.480+0000 7fabd9c90640 1 -- 172.21.15.98:0/38161712 shutdown_connections 2023-11-28T17:41:42.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.480+0000 7fabd9c90640 1 -- 172.21.15.98:0/38161712 wait complete. 2023-11-28T17:41:42.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.482+0000 7fabd9c90640 1 Processor -- start 2023-11-28T17:41:42.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.482+0000 7fabd9c90640 1 -- start start 2023-11-28T17:41:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.483+0000 7fabd9c90640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabd40c7140 0x7fabd406a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.483+0000 7fabd9c90640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabd40c7870 0x7fabd41f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.483+0000 7fabd9c90640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd40c7c50 0x7fabd40cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.483+0000 7fabd9c90640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd40c6520 con 0x7fabd40c7870 2023-11-28T17:41:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.483+0000 7fabd9c90640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd41efd70 con 0x7fabd40c7c50 2023-11-28T17:41:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.483+0000 7fabd9c90640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd40c4430 con 0x7fabd40c7140 2023-11-28T17:41:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.484+0000 7fabcbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabd40c7870 0x7fabd41f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.484+0000 7fabcbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabd40c7870 0x7fabd41f26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36656/0 (socket says 172.21.15.98:36656) 2023-11-28T17:41:42.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.484+0000 7fabcbfff640 1 -- 172.21.15.98:0/242235210 learned_addr learned my addr 172.21.15.98:0/242235210 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.484+0000 7fabd3fff640 1 --2- 172.21.15.98:0/242235210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd40c7c50 0x7fabd40cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.484+0000 7fabd37fe640 1 --2- 172.21.15.98:0/242235210 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabd40c7140 0x7fabd406a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.484+0000 7fabcbfff640 1 -- 172.21.15.98:0/242235210 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabd40c7140 msgr2=0x7fabd406a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.484+0000 7fabcbfff640 1 --2- 172.21.15.98:0/242235210 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabd40c7140 0x7fabd406a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.484+0000 7fabcbfff640 1 -- 172.21.15.98:0/242235210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd40c7c50 msgr2=0x7fabd40cc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.484+0000 7fabcbfff640 1 --2- 172.21.15.98:0/242235210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd40c7c50 0x7fabd40cc350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.484+0000 7fabcbfff640 1 -- 172.21.15.98:0/242235210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc009b30 con 0x7fabd40c7870 2023-11-28T17:41:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.485+0000 7fabcbfff640 1 --2- 172.21.15.98:0/242235210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabd40c7870 0x7fabd41f26e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00e9b0 tx=0x7fabbc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.485+0000 7fabd0ff9640 1 -- 172.21.15.98:0/242235210 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc009670 con 0x7fabd40c7870 2023-11-28T17:41:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.485+0000 7fabd0ff9640 1 -- 172.21.15.98:0/242235210 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabbc00c9a0 con 0x7fabd40c7870 2023-11-28T17:41:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.485+0000 7fabd0ff9640 1 -- 172.21.15.98:0/242235210 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc00cd50 con 0x7fabd40c7870 2023-11-28T17:41:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.485+0000 7fabd9c90640 1 -- 172.21.15.98:0/242235210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd40c4710 con 0x7fabd40c7870 2023-11-28T17:41:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.485+0000 7fabd9c90640 1 -- 172.21.15.98:0/242235210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd40c4cd0 con 0x7fabd40c7870 2023-11-28T17:41:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.486+0000 7fabd9c90640 1 -- 172.21.15.98:0/242235210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd4074f60 con 0x7fabd40c7870 2023-11-28T17:41:42.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.487+0000 7fabd0ff9640 1 -- 172.21.15.98:0/242235210 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fabbc013470 con 0x7fabd40c7870 2023-11-28T17:41:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.488+0000 7fabd0ff9640 1 --2- 172.21.15.98:0/242235210 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fabb406b2a0 0x7fabb406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.488+0000 7fabd0ff9640 1 -- 172.21.15.98:0/242235210 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fabbc08a190 con 0x7fabd40c7870 2023-11-28T17:41:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.488+0000 7fabd37fe640 1 --2- 172.21.15.98:0/242235210 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fabb406b2a0 0x7fabb406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.489+0000 7fabd37fe640 1 --2- 172.21.15.98:0/242235210 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fabb406b2a0 0x7fabb406d760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fabc4004ea0 tx=0x7fabc4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:42.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.493+0000 7fabd0ff9640 1 -- 172.21.15.98:0/242235210 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fabbc055000 con 0x7fabd40c7870 2023-11-28T17:41:42.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.681+0000 7fabd9c90640 1 -- 172.21.15.98:0/242235210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabd406e510 con 0x7fabd40c7870 2023-11-28T17:41:42.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.685+0000 7fabd0ff9640 1 -- 172.21.15.98:0/242235210 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16517 (secure 0 0 0) 0x7fabbc058620 con 0x7fabd40c7870 2023-11-28T17:41:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:42.687 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:41:18.195933+0000","last_up_change":"2023-11-28T17:41:09.556993+0000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:41:10.303280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:41:14.890871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:41:15.704103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6810","nonce":2188704385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6813","nonce":2188704385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6819","nonce":2188704385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6816","nonce":2188704385}]},"public_addr":"172.21.15.98:6810/2188704385","cluster_addr":"172.21.15.98:6813/2188704385","heartbeat_back_addr":"172.21.15.98:6819/2188704385","heartbeat_front_addr":"172.21.15.98:6816/2188704385","state":["exists","up"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6801","nonce":3918226860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6803","nonce":3918226860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6807","nonce":3918226860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6805","nonce":3918226860}]},"public_addr":"172.21.15.98:6801/3918226860","cluster_addr":"172.21.15.98:6803/3918226860","heartbeat_back_addr":"172.21.15.98:6807/3918226860","heartbeat_front_addr":"172.21.15.98:6805/3918226860","state":["exists","up"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6812","nonce":2691532400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6815","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6818","nonce":2691532400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6823","nonce":2691532400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6821","nonce":2691532400}]},"public_addr":"172.21.15.98:6812/2691532400","cluster_addr":"172.21.15.98:6818/2691532400","heartbeat_back_addr":"172.21.15.98:6823/2691532400","heartbeat_front_addr":"172.21.15.98:6821/2691532400","state":["exists","up"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6825","nonce":1905487008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6827","nonce":1905487008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6831","nonce":1905487008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6829","nonce":1905487008}]},"public_addr":"172.21.15.98:6825/1905487008","cluster_addr":"172.21.15.98:6827/1905487008","heartbeat_back_addr":"172.21.15.98:6831/1905487008","heartbeat_front_addr":"172.21.15.98:6829/1905487008","state":["exists","up"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6801","nonce":2974677215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6803","nonce":2974677215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6807","nonce":2974677215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6805","nonce":2974677215}]},"public_addr":"172.21.15.113:6801/2974677215","cluster_addr":"172.21.15.113:6803/2974677215","heartbeat_back_addr":"172.21.15.113:6807/2974677215","heartbeat_front_addr":"172.21.15.113:6805/2974677215","state":["exists","up"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6817","nonce":2685000355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6821","nonce":2685000355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6829","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6831","nonce":2685000355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6823","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6826","nonce":2685000355}]},"public_addr":"172.21.15.113:6817/2685000355","cluster_addr":"172.21.15.113:6821/2685000355","heartbeat_back_addr":"172.21.15.113:6831/2685000355","heartbeat_front_addr":"172.21.15.113:6826/2685000355","state":["exists","up"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6809","nonce":2764300831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6811","nonce":2764300831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6815","nonce":2764300831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6813","nonce":2764300831}]},"public_addr":"172.21.15.113:6809/2764300831","cluster_addr":"172.21.15.113:6811/2764300831","heartbeat_back_addr":"172.21.15.113:6815/2764300831","heartbeat_front_addr":"172.21.15.113:6813/2764300831","state":["exists","up"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6819","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6820","nonce":1656248400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6824","nonce":1656248400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6830","nonce":1656248400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6825","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6827","nonce":1656248400}]},"public_addr":"172.21.15.113:6820/1656248400","cluster_addr":"172.21.15.113:6824/1656248400","heartbeat_back_addr":"172.21.15.113:6830/1656248400","heartbeat_front_addr":"172.21.15.113:6827/1656248400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.379549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.312196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.117494+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.388128+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.360608+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.411101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.355995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.390362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:41:42.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.687+0000 7fabd9c90640 1 -- 172.21.15.98:0/242235210 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fabb406b2a0 msgr2=0x7fabb406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:42.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.687+0000 7fabd9c90640 1 --2- 172.21.15.98:0/242235210 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fabb406b2a0 0x7fabb406d760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fabc4004ea0 tx=0x7fabc4009040 comp rx=0 tx=0).stop 2023-11-28T17:41:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.687+0000 7fabd9c90640 1 -- 172.21.15.98:0/242235210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabd40c7870 msgr2=0x7fabd41f26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.687+0000 7fabd9c90640 1 --2- 172.21.15.98:0/242235210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabd40c7870 0x7fabd41f26e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00e9b0 tx=0x7fabbc00ee70 comp rx=0 tx=0).stop 2023-11-28T17:41:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.687+0000 7fabd9c90640 1 -- 172.21.15.98:0/242235210 shutdown_connections 2023-11-28T17:41:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.687+0000 7fabd9c90640 1 --2- 172.21.15.98:0/242235210 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fabb406b2a0 0x7fabb406d760 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.687+0000 7fabd9c90640 1 --2- 172.21.15.98:0/242235210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabd40c7c50 0x7fabd40cc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.687+0000 7fabd9c90640 1 --2- 172.21.15.98:0/242235210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fabd40c7870 0x7fabd41f26e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.687+0000 7fabd9c90640 1 --2- 172.21.15.98:0/242235210 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fabd40c7140 0x7fabd406a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.687+0000 7fabd9c90640 1 -- 172.21.15.98:0/242235210 >> 172.21.15.98:0/242235210 conn(0x7fabd40c8d10 msgr2=0x7fabd4159020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.687+0000 7fabd9c90640 1 -- 172.21.15.98:0/242235210 shutdown_connections 2023-11-28T17:41:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.687+0000 7fabd9c90640 1 -- 172.21.15.98:0/242235210 wait complete. 2023-11-28T17:41:42.707 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:41:10.303280+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:41:14.890871+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:41:15.704103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:41:42.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:41:42.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.923+0000 7f9183be5640 1 Processor -- start 2023-11-28T17:41:42.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.923+0000 7f9183be5640 1 -- start start 2023-11-28T17:41:42.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.924+0000 7f9183be5640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f917c151be0 0x7f917c0c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.926+0000 7f9183be5640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f917c0c5a60 0x7f917c1584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.926+0000 7f9183be5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f917c0c4730 0x7f917c0c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.926+0000 7f9183be5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f917c158a10 con 0x7f917c0c4730 2023-11-28T17:41:42.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.926+0000 7f9183be5640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f917c0c8150 con 0x7f917c0c5a60 2023-11-28T17:41:42.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.926+0000 7f9183be5640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f917c0c8850 con 0x7f917c151be0 2023-11-28T17:41:42.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.926+0000 7f9182182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f917c0c4730 0x7f917c0c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.926+0000 7f9182182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f917c0c4730 0x7f917c0c4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36662/0 (socket says 172.21.15.98:36662) 2023-11-28T17:41:42.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.926+0000 7f9182182640 1 -- 172.21.15.98:0/3361155225 learned_addr learned my addr 172.21.15.98:0/3361155225 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:42.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.926+0000 7f9181981640 1 --2- 172.21.15.98:0/3361155225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f917c151be0 0x7f917c0c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.926+0000 7f9181180640 1 --2- 172.21.15.98:0/3361155225 >> v2:172.21.15.113:3300/0 conn(0x7f917c0c5a60 0x7f917c1584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.926+0000 7f9182182640 1 -- 172.21.15.98:0/3361155225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f917c151be0 msgr2=0x7f917c0c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:42.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.926+0000 7f9182182640 1 --2- 172.21.15.98:0/3361155225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f917c151be0 0x7f917c0c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.926+0000 7f9182182640 1 -- 172.21.15.98:0/3361155225 >> v2:172.21.15.113:3300/0 conn(0x7f917c0c5a60 msgr2=0x7f917c1584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:42.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.926+0000 7f9182182640 1 --2- 172.21.15.98:0/3361155225 >> v2:172.21.15.113:3300/0 conn(0x7f917c0c5a60 0x7f917c1584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.926+0000 7f9182182640 1 -- 172.21.15.98:0/3361155225 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917c0c5f50 con 0x7f917c0c4730 2023-11-28T17:41:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.927+0000 7f9182182640 1 --2- 172.21.15.98:0/3361155225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f917c0c4730 0x7f917c0c4ba0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f917c0c5240 tx=0x7f917402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2fa47bd6f11cc1e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.927+0000 7f918097f640 1 -- 172.21.15.98:0/3361155225 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917402ca80 con 0x7f917c0c4730 2023-11-28T17:41:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.927+0000 7f918097f640 1 -- 172.21.15.98:0/3361155225 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917402cbe0 con 0x7f917c0c4730 2023-11-28T17:41:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.927+0000 7f918097f640 1 -- 172.21.15.98:0/3361155225 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91740343f0 con 0x7f917c0c4730 2023-11-28T17:41:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.927+0000 7f9183be5640 1 -- 172.21.15.98:0/3361155225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f917c0c4730 msgr2=0x7f917c0c4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.927+0000 7f9183be5640 1 --2- 172.21.15.98:0/3361155225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f917c0c4730 0x7f917c0c4ba0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f917c0c5240 tx=0x7f917402bee0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.928+0000 7f9183be5640 1 -- 172.21.15.98:0/3361155225 shutdown_connections 2023-11-28T17:41:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.928+0000 7f9183be5640 1 --2- 172.21.15.98:0/3361155225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f917c0c4730 0x7f917c0c4ba0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.928+0000 7f9183be5640 1 --2- 172.21.15.98:0/3361155225 >> v2:172.21.15.113:3300/0 conn(0x7f917c0c5a60 0x7f917c1584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.928+0000 7f9183be5640 1 --2- 172.21.15.98:0/3361155225 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f917c151be0 0x7f917c0c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.928+0000 7f9183be5640 1 -- 172.21.15.98:0/3361155225 >> 172.21.15.98:0/3361155225 conn(0x7f917c06c0f0 msgr2=0x7f917c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:42.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.928+0000 7f9183be5640 1 -- 172.21.15.98:0/3361155225 shutdown_connections 2023-11-28T17:41:42.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.928+0000 7f9183be5640 1 -- 172.21.15.98:0/3361155225 wait complete. 2023-11-28T17:41:42.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.929+0000 7f9183be5640 1 Processor -- start 2023-11-28T17:41:42.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.930+0000 7f9183be5640 1 -- start start 2023-11-28T17:41:42.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.930+0000 7f9183be5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f917c0c4730 0x7f917c14fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.930+0000 7f9183be5640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f917c0c5a60 0x7f917c1502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.930+0000 7f9183be5640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f917c151be0 0x7f917c15a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.930+0000 7f9183be5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917c159fe0 con 0x7f917c0c4730 2023-11-28T17:41:42.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.931+0000 7f9183be5640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917c159cb0 con 0x7f917c0c5a60 2023-11-28T17:41:42.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.931+0000 7f9183be5640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f917c14e5f0 con 0x7f917c151be0 2023-11-28T17:41:42.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.931+0000 7f9181981640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f917c0c4730 0x7f917c14fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.931+0000 7f9181981640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f917c0c4730 0x7f917c14fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36672/0 (socket says 172.21.15.98:36672) 2023-11-28T17:41:42.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.931+0000 7f9181981640 1 -- 172.21.15.98:0/858058388 learned_addr learned my addr 172.21.15.98:0/858058388 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:42.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.931+0000 7f9181180640 1 --2- 172.21.15.98:0/858058388 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f917c0c5a60 0x7f917c1502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.931+0000 7f9182182640 1 --2- 172.21.15.98:0/858058388 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f917c151be0 0x7f917c15a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.931+0000 7f9181981640 1 -- 172.21.15.98:0/858058388 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f917c151be0 msgr2=0x7f917c15a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:42.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.931+0000 7f9181981640 1 --2- 172.21.15.98:0/858058388 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f917c151be0 0x7f917c15a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.931+0000 7f9181981640 1 -- 172.21.15.98:0/858058388 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f917c0c5a60 msgr2=0x7f917c1502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:42.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.931+0000 7f9181981640 1 --2- 172.21.15.98:0/858058388 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f917c0c5a60 0x7f917c1502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:42.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.931+0000 7f9181981640 1 -- 172.21.15.98:0/858058388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9174009b30 con 0x7f917c0c4730 2023-11-28T17:41:42.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.931+0000 7f9181981640 1 --2- 172.21.15.98:0/858058388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f917c0c4730 0x7f917c14fd60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f917800d800 tx=0x7f917800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:42.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.932+0000 7f91727fc640 1 -- 172.21.15.98:0/858058388 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917800b6c0 con 0x7f917c0c4730 2023-11-28T17:41:42.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.932+0000 7f91727fc640 1 -- 172.21.15.98:0/858058388 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917800b820 con 0x7f917c0c4730 2023-11-28T17:41:42.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.932+0000 7f9183be5640 1 -- 172.21.15.98:0/858058388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f917c14e8d0 con 0x7f917c0c4730 2023-11-28T17:41:42.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.932+0000 7f91727fc640 1 -- 172.21.15.98:0/858058388 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917800bbb0 con 0x7f917c0c4730 2023-11-28T17:41:42.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.932+0000 7f9183be5640 1 -- 172.21.15.98:0/858058388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f917c14ee90 con 0x7f917c0c4730 2023-11-28T17:41:42.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.933+0000 7f9183be5640 1 -- 172.21.15.98:0/858058388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f917c159e20 con 0x7f917c0c4730 2023-11-28T17:41:42.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.934+0000 7f91727fc640 1 -- 172.21.15.98:0/858058388 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f917800bd10 con 0x7f917c0c4730 2023-11-28T17:41:42.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.935+0000 7f91727fc640 1 --2- 172.21.15.98:0/858058388 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f916006b2d0 0x7f916006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:42.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.935+0000 7f91727fc640 1 -- 172.21.15.98:0/858058388 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f917808a140 con 0x7f917c0c4730 2023-11-28T17:41:42.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.936+0000 7f9181180640 1 --2- 172.21.15.98:0/858058388 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f916006b2d0 0x7f916006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:42.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.936+0000 7f9181180640 1 --2- 172.21.15.98:0/858058388 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f916006b2d0 0x7f916006d790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f917c070c00 tx=0x7f916c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:42.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:42.940+0000 7f91727fc640 1 -- 172.21.15.98:0/858058388 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f917c159e20 con 0x7f917c0c4730 2023-11-28T17:41:43.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.130+0000 7f9183be5640 1 -- 172.21.15.98:0/858058388 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f917c159e20 con 0x7f917c0c4730 2023-11-28T17:41:43.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.131+0000 7f91727fc640 1 -- 172.21.15.98:0/858058388 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f917c159e20 con 0x7f917c0c4730 2023-11-28T17:41:43.133 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-11-28T17:41:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.134+0000 7f9183be5640 1 -- 172.21.15.98:0/858058388 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f916006b2d0 msgr2=0x7f916006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.134+0000 7f9183be5640 1 --2- 172.21.15.98:0/858058388 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f916006b2d0 0x7f916006d790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f917c070c00 tx=0x7f916c008040 comp rx=0 tx=0).stop 2023-11-28T17:41:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.134+0000 7f9183be5640 1 -- 172.21.15.98:0/858058388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f917c0c4730 msgr2=0x7f917c14fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.134+0000 7f9183be5640 1 --2- 172.21.15.98:0/858058388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f917c0c4730 0x7f917c14fd60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f917800d800 tx=0x7f917800dcc0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.134+0000 7f9183be5640 1 -- 172.21.15.98:0/858058388 shutdown_connections 2023-11-28T17:41:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.134+0000 7f9183be5640 1 --2- 172.21.15.98:0/858058388 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f917c151be0 0x7f917c15a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.134+0000 7f9183be5640 1 --2- 172.21.15.98:0/858058388 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f916006b2d0 0x7f916006d790 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.134+0000 7f9183be5640 1 --2- 172.21.15.98:0/858058388 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f917c0c5a60 0x7f917c1502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.134+0000 7f9183be5640 1 --2- 172.21.15.98:0/858058388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f917c0c4730 0x7f917c14fd60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.134+0000 7f9183be5640 1 -- 172.21.15.98:0/858058388 >> 172.21.15.98:0/858058388 conn(0x7f917c06c0f0 msgr2=0x7f917c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.134+0000 7f9183be5640 1 -- 172.21.15.98:0/858058388 shutdown_connections 2023-11-28T17:41:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.134+0000 7f9183be5640 1 -- 172.21.15.98:0/858058388 wait complete. 2023-11-28T17:41:43.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:41:43.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.372+0000 7fda40807640 1 Processor -- start 2023-11-28T17:41:43.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.372+0000 7fda40807640 1 -- start start 2023-11-28T17:41:43.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.373+0000 7fda40807640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda3c151be0 0x7fda3c0c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:43.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.373+0000 7fda40807640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fda3c0c5a60 0x7fda3c1584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:43.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.373+0000 7fda40807640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c158a10 con 0x7fda3c0c4730 2023-11-28T17:41:43.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.373+0000 7fda40807640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3c0c8150 con 0x7fda3c0c5a60 2023-11-28T17:41:43.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.373+0000 7fda40807640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c0c8850 con 0x7fda3c151be0 2023-11-28T17:41:43.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.373+0000 7fda33fff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fda3c0c5a60 0x7fda3c1584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:43.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.374+0000 7fda33fff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fda3c0c5a60 0x7fda3c1584d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:39882/0 (socket says 172.21.15.98:39882) 2023-11-28T17:41:43.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.374+0000 7fda33fff640 1 -- 172.21.15.98:0/1769535998 learned_addr learned my addr 172.21.15.98:0/1769535998 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:43.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.374+0000 7fda3a59c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda3c151be0 0x7fda3c0c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:43.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.374+0000 7fda33fff640 1 -- 172.21.15.98:0/1769535998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda3c151be0 msgr2=0x7fda3c0c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:43.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.374+0000 7fda33fff640 1 --2- 172.21.15.98:0/1769535998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda3c151be0 0x7fda3c0c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.374+0000 7fda33fff640 1 -- 172.21.15.98:0/1769535998 >> v1:172.21.15.113:6789/0 conn(0x7fda3c0c4730 legacy=0x7fda3c06dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:43.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.374+0000 7fda33fff640 1 -- 172.21.15.98:0/1769535998 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c0c5f50 con 0x7fda3c0c5a60 2023-11-28T17:41:43.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.375+0000 7fda33fff640 1 --2- 172.21.15.98:0/1769535998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda3c0c5a60 0x7fda3c1584d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fda24009870 tx=0x7fda2402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcb6fe2170f40d12 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:43.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.376+0000 7fda39d9b640 1 -- 172.21.15.98:0/1769535998 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda24030840 con 0x7fda3c0c5a60 2023-11-28T17:41:43.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.376+0000 7fda39d9b640 1 -- 172.21.15.98:0/1769535998 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda240309a0 con 0x7fda3c0c5a60 2023-11-28T17:41:43.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.376+0000 7fda39d9b640 1 -- 172.21.15.98:0/1769535998 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda24030d30 con 0x7fda3c0c5a60 2023-11-28T17:41:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.377+0000 7fda40807640 1 -- 172.21.15.98:0/1769535998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda3c0c5a60 msgr2=0x7fda3c1584d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.377+0000 7fda40807640 1 --2- 172.21.15.98:0/1769535998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda3c0c5a60 0x7fda3c1584d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fda24009870 tx=0x7fda2402b980 comp rx=0 tx=0).stop 2023-11-28T17:41:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.377+0000 7fda40807640 1 -- 172.21.15.98:0/1769535998 shutdown_connections 2023-11-28T17:41:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.377+0000 7fda40807640 1 --2- 172.21.15.98:0/1769535998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda3c0c5a60 0x7fda3c1584d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.377+0000 7fda40807640 1 --2- 172.21.15.98:0/1769535998 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda3c151be0 0x7fda3c0c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.377+0000 7fda40807640 1 -- 172.21.15.98:0/1769535998 >> 172.21.15.98:0/1769535998 conn(0x7fda3c06c0f0 msgr2=0x7fda3c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.377+0000 7fda40807640 1 -- 172.21.15.98:0/1769535998 shutdown_connections 2023-11-28T17:41:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.377+0000 7fda40807640 1 -- 172.21.15.98:0/1769535998 wait complete. 2023-11-28T17:41:43.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.379+0000 7fda40807640 1 Processor -- start 2023-11-28T17:41:43.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.379+0000 7fda40807640 1 -- start start 2023-11-28T17:41:43.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.380+0000 7fda40807640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda3c0c4730 0x7fda3c158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:43.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.380+0000 7fda40807640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda3c0c5a60 0x7fda3c15a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:43.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.380+0000 7fda40807640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda3c151be0 0x7fda3c15abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:43.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.380+0000 7fda40807640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1561d0 con 0x7fda3c151be0 2023-11-28T17:41:43.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.380+0000 7fda40807640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c157ac0 con 0x7fda3c0c4730 2023-11-28T17:41:43.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.380+0000 7fda40807640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c156420 con 0x7fda3c0c5a60 2023-11-28T17:41:43.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.380+0000 7fda3ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda3c151be0 0x7fda3c15abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:43.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.380+0000 7fda3ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda3c151be0 0x7fda3c15abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36674/0 (socket says 172.21.15.98:36674) 2023-11-28T17:41:43.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.380+0000 7fda3ad9d640 1 -- 172.21.15.98:0/844581757 learned_addr learned my addr 172.21.15.98:0/844581757 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:43.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.380+0000 7fda3a59c640 1 --2- 172.21.15.98:0/844581757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda3c0c4730 0x7fda3c158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:43.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.380+0000 7fda33fff640 1 --2- 172.21.15.98:0/844581757 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda3c0c5a60 0x7fda3c15a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:43.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.381+0000 7fda3ad9d640 1 -- 172.21.15.98:0/844581757 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda3c0c5a60 msgr2=0x7fda3c15a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:43.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.381+0000 7fda3ad9d640 1 --2- 172.21.15.98:0/844581757 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda3c0c5a60 0x7fda3c15a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.381+0000 7fda3ad9d640 1 -- 172.21.15.98:0/844581757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda3c0c4730 msgr2=0x7fda3c158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:43.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.381+0000 7fda3ad9d640 1 --2- 172.21.15.98:0/844581757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda3c0c4730 0x7fda3c158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.381+0000 7fda3ad9d640 1 -- 172.21.15.98:0/844581757 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda240094d0 con 0x7fda3c151be0 2023-11-28T17:41:43.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.381+0000 7fda3a59c640 1 --2- 172.21.15.98:0/844581757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda3c0c4730 0x7fda3c158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:41:43.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.381+0000 7fda3ad9d640 1 --2- 172.21.15.98:0/844581757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda3c151be0 0x7fda3c15abf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fda3400c7d0 tx=0x7fda3400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:43.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.381+0000 7fda32ffd640 1 -- 172.21.15.98:0/844581757 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda34003b70 con 0x7fda3c151be0 2023-11-28T17:41:43.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.381+0000 7fda32ffd640 1 -- 172.21.15.98:0/844581757 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda34003cf0 con 0x7fda3c151be0 2023-11-28T17:41:43.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.381+0000 7fda40807640 1 -- 172.21.15.98:0/844581757 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3c157190 con 0x7fda3c151be0 2023-11-28T17:41:43.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.381+0000 7fda32ffd640 1 -- 172.21.15.98:0/844581757 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3400aa90 con 0x7fda3c151be0 2023-11-28T17:41:43.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.381+0000 7fda40807640 1 -- 172.21.15.98:0/844581757 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c157750 con 0x7fda3c151be0 2023-11-28T17:41:43.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.382+0000 7fda40807640 1 -- 172.21.15.98:0/844581757 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3c15a3d0 con 0x7fda3c151be0 2023-11-28T17:41:43.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.383+0000 7fda32ffd640 1 -- 172.21.15.98:0/844581757 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fda3400abf0 con 0x7fda3c151be0 2023-11-28T17:41:43.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.384+0000 7fda32ffd640 1 --2- 172.21.15.98:0/844581757 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fda1c06b2a0 0x7fda1c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:43.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.384+0000 7fda32ffd640 1 -- 172.21.15.98:0/844581757 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda34089a60 con 0x7fda3c151be0 2023-11-28T17:41:43.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.384+0000 7fda3a59c640 1 --2- 172.21.15.98:0/844581757 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fda1c06b2a0 0x7fda1c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:43.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.385+0000 7fda3a59c640 1 --2- 172.21.15.98:0/844581757 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fda1c06b2a0 0x7fda1c06d760 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fda2c009af0 tx=0x7fda2c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:43.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.389+0000 7fda32ffd640 1 -- 172.21.15.98:0/844581757 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda3c15a3d0 con 0x7fda3c151be0 2023-11-28T17:41:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.578+0000 7fda40807640 1 -- 172.21.15.98:0/844581757 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fda3c158a10 con 0x7fda3c151be0 2023-11-28T17:41:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.579+0000 7fda32ffd640 1 -- 172.21.15.98:0/844581757 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fda3c158a10 con 0x7fda3c151be0 2023-11-28T17:41:43.581 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-11-28T17:41:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.581+0000 7fda40807640 1 -- 172.21.15.98:0/844581757 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fda1c06b2a0 msgr2=0x7fda1c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:43.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.581+0000 7fda40807640 1 --2- 172.21.15.98:0/844581757 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fda1c06b2a0 0x7fda1c06d760 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fda2c009af0 tx=0x7fda2c008040 comp rx=0 tx=0).stop 2023-11-28T17:41:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.581+0000 7fda40807640 1 -- 172.21.15.98:0/844581757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda3c151be0 msgr2=0x7fda3c15abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.581+0000 7fda40807640 1 --2- 172.21.15.98:0/844581757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda3c151be0 0x7fda3c15abf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fda3400c7d0 tx=0x7fda3400cc90 comp rx=0 tx=0).stop 2023-11-28T17:41:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.581+0000 7fda40807640 1 -- 172.21.15.98:0/844581757 shutdown_connections 2023-11-28T17:41:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.581+0000 7fda40807640 1 --2- 172.21.15.98:0/844581757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda3c151be0 0x7fda3c15abf0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.581+0000 7fda40807640 1 --2- 172.21.15.98:0/844581757 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fda3c0c5a60 0x7fda3c15a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.581+0000 7fda40807640 1 --2- 172.21.15.98:0/844581757 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fda1c06b2a0 0x7fda1c06d760 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.581+0000 7fda40807640 1 --2- 172.21.15.98:0/844581757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda3c0c4730 0x7fda3c158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.581+0000 7fda40807640 1 -- 172.21.15.98:0/844581757 >> 172.21.15.98:0/844581757 conn(0x7fda3c06c0f0 msgr2=0x7fda3c070290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.582+0000 7fda40807640 1 -- 172.21.15.98:0/844581757 shutdown_connections 2023-11-28T17:41:43.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.582+0000 7fda40807640 1 -- 172.21.15.98:0/844581757 wait complete. 2023-11-28T17:41:43.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:41:43.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.810+0000 7ff8e0e5c640 1 Processor -- start 2023-11-28T17:41:43.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.810+0000 7ff8e0e5c640 1 -- start start 2023-11-28T17:41:43.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.810+0000 7ff8e0e5c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8dc0c5a60 0x7ff8dc0c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.811+0000 7ff8e0e5c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8dc0c8150 con 0x7ff8dc151be0 2023-11-28T17:41:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.811+0000 7ff8e0e5c640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8dc0c8850 con 0x7ff8dc1584d0 2023-11-28T17:41:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.811+0000 7ff8e0e5c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc0c5f50 con 0x7ff8dc0c5a60 2023-11-28T17:41:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.811+0000 7ff8da59c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff8dc151be0 0x7ff8dc0c4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55536/0 (socket says 172.21.15.98:55536) 2023-11-28T17:41:43.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.811+0000 7ff8da59c640 1 -- 172.21.15.98:0/1937887744 learned_addr learned my addr 172.21.15.98:0/1937887744 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.811+0000 7ff8d9d9b640 1 --2- 172.21.15.98:0/1937887744 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8dc0c5a60 0x7ff8dc0c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.811+0000 7ff8d959a640 1 -- 172.21.15.98:0/1937887744 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875463712 0 0) 0x7ff8dc0c8150 con 0x7ff8dc151be0 2023-11-28T17:41:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.811+0000 7ff8d959a640 1 -- 172.21.15.98:0/1937887744 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8c4003520 con 0x7ff8dc151be0 2023-11-28T17:41:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.812+0000 7ff8d959a640 1 -- 172.21.15.98:0/1937887744 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 399270099 0 0) 0x7ff8c4003520 con 0x7ff8dc151be0 2023-11-28T17:41:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.812+0000 7ff8d959a640 1 -- 172.21.15.98:0/1937887744 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8dc0c5a60 msgr2=0x7ff8dc0c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.812+0000 7ff8d959a640 1 --2- 172.21.15.98:0/1937887744 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8dc0c5a60 0x7ff8dc0c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.812+0000 7ff8d959a640 1 -- 172.21.15.98:0/1937887744 >> v1:172.21.15.113:6789/0 conn(0x7ff8dc1584d0 legacy=0x7ff8dc06dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.812+0000 7ff8d959a640 1 -- 172.21.15.98:0/1937887744 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc0c7b90 con 0x7ff8dc151be0 2023-11-28T17:41:43.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.812+0000 7ff8d959a640 1 -- 172.21.15.98:0/1937887744 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7ff8cc001670 con 0x7ff8dc151be0 2023-11-28T17:41:43.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.812+0000 7ff8d959a640 1 -- 172.21.15.98:0/1937887744 >> v1:172.21.15.98:6789/0 conn(0x7ff8dc151be0 legacy=0x7ff8dc0c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:43.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.812+0000 7ff8d959a640 1 --2- 172.21.15.98:0/1937887744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8c40039a0 0x7ff8c4003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:43.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.812+0000 7ff8d959a640 1 --2- 172.21.15.98:0/1937887744 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8dc0c5a60 0x7ff8c40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:43.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.812+0000 7ff8d959a640 1 --2- 172.21.15.98:0/1937887744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8c4004d00 0x7ff8c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:43.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.812+0000 7ff8d959a640 1 -- 172.21.15.98:0/1937887744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc0c5f50 con 0x7ff8c4004d00 2023-11-28T17:41:43.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.812+0000 7ff8d959a640 1 -- 172.21.15.98:0/1937887744 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c400ba30 con 0x7ff8c40039a0 2023-11-28T17:41:43.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.812+0000 7ff8d959a640 1 -- 172.21.15.98:0/1937887744 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c400bba0 con 0x7ff8dc0c5a60 2023-11-28T17:41:43.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.812+0000 7ff8dad9d640 1 --2- 172.21.15.98:0/1937887744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8c4004d00 0x7ff8c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:43.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.813+0000 7ff8d9d9b640 1 --2- 172.21.15.98:0/1937887744 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8dc0c5a60 0x7ff8c40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:43.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.813+0000 7ff8da59c640 1 --2- 172.21.15.98:0/1937887744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8c40039a0 0x7ff8c4003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:43.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.813+0000 7ff8dad9d640 1 -- 172.21.15.98:0/1937887744 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8dc0c5a60 msgr2=0x7ff8c40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:43.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.813+0000 7ff8dad9d640 1 --2- 172.21.15.98:0/1937887744 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8dc0c5a60 0x7ff8c40047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.813+0000 7ff8dad9d640 1 -- 172.21.15.98:0/1937887744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8c40039a0 msgr2=0x7ff8c4003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:43.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.813+0000 7ff8dad9d640 1 --2- 172.21.15.98:0/1937887744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8c40039a0 0x7ff8c4003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.813+0000 7ff8dad9d640 1 -- 172.21.15.98:0/1937887744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c4003520 con 0x7ff8c4004d00 2023-11-28T17:41:43.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.813+0000 7ff8da59c640 1 --2- 172.21.15.98:0/1937887744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8c40039a0 0x7ff8c4003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:43.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.814+0000 7ff8dad9d640 1 --2- 172.21.15.98:0/1937887744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8c4004d00 0x7ff8c4005180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4001f60 tx=0x7ff8c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2023c45399abde server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:43.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.814+0000 7ff8d959a640 1 -- 172.21.15.98:0/1937887744 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d402d6f0 con 0x7ff8c4004d00 2023-11-28T17:41:43.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.814+0000 7ff8d959a640 1 -- 172.21.15.98:0/1937887744 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d402d870 con 0x7ff8c4004d00 2023-11-28T17:41:43.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.814+0000 7ff8d959a640 1 -- 172.21.15.98:0/1937887744 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d402dc20 con 0x7ff8c4004d00 2023-11-28T17:41:43.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.815+0000 7ff8e0e5c640 1 -- 172.21.15.98:0/1937887744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8c4004d00 msgr2=0x7ff8c4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:43.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.815+0000 7ff8e0e5c640 1 --2- 172.21.15.98:0/1937887744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8c4004d00 0x7ff8c4005180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4001f60 tx=0x7ff8c40057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.815+0000 7ff8dad9d640 1 -- 172.21.15.98:0/1937887744 reap_dead start 2023-11-28T17:41:43.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.815+0000 7ff8e0e5c640 1 -- 172.21.15.98:0/1937887744 shutdown_connections 2023-11-28T17:41:43.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.815+0000 7ff8e0e5c640 1 -- 172.21.15.98:0/1937887744 >> 172.21.15.98:0/1937887744 conn(0x7ff8dc06c0f0 msgr2=0x7ff8dc06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:43.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.815+0000 7ff8e0e5c640 1 -- 172.21.15.98:0/1937887744 shutdown_connections 2023-11-28T17:41:43.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.815+0000 7ff8e0e5c640 1 -- 172.21.15.98:0/1937887744 wait complete. 2023-11-28T17:41:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.817+0000 7ff8e0e5c640 1 Processor -- start 2023-11-28T17:41:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.817+0000 7ff8e0e5c640 1 -- start start 2023-11-28T17:41:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8e0e5c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8c4004d00 0x7ff8dc070c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8e0e5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8dc152390 0x7ff8dc0711c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8e0e5c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8dc15b0d0 0x7ff8dc14eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8e0e5c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc1eec90 con 0x7ff8dc152390 2023-11-28T17:41:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8e0e5c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc1f20f0 con 0x7ff8c4004d00 2023-11-28T17:41:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8e0e5c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc1f23b0 con 0x7ff8dc15b0d0 2023-11-28T17:41:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8d9d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8dc152390 0x7ff8dc0711c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8d9d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8dc152390 0x7ff8dc0711c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36688/0 (socket says 172.21.15.98:36688) 2023-11-28T17:41:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8d9d9b640 1 -- 172.21.15.98:0/1822883415 learned_addr learned my addr 172.21.15.98:0/1822883415 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8da59c640 1 --2- 172.21.15.98:0/1822883415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8c4004d00 0x7ff8dc070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8dad9d640 1 --2- 172.21.15.98:0/1822883415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8dc15b0d0 0x7ff8dc14eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8d9d9b640 1 -- 172.21.15.98:0/1822883415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8dc15b0d0 msgr2=0x7ff8dc14eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8d9d9b640 1 --2- 172.21.15.98:0/1822883415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8dc15b0d0 0x7ff8dc14eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8d9d9b640 1 -- 172.21.15.98:0/1822883415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8c4004d00 msgr2=0x7ff8dc070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8d9d9b640 1 --2- 172.21.15.98:0/1822883415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8c4004d00 0x7ff8dc070c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8d9d9b640 1 -- 172.21.15.98:0/1822883415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc1f2690 con 0x7ff8dc152390 2023-11-28T17:41:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.818+0000 7ff8dad9d640 1 --2- 172.21.15.98:0/1822883415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8dc15b0d0 0x7ff8dc14eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.819+0000 7ff8d9d9b640 1 --2- 172.21.15.98:0/1822883415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8dc152390 0x7ff8dc0711c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0010890 tx=0x7ff8d0010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.819+0000 7ff8caffd640 1 -- 172.21.15.98:0/1822883415 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d00095b0 con 0x7ff8dc152390 2023-11-28T17:41:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.819+0000 7ff8caffd640 1 -- 172.21.15.98:0/1822883415 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d000eb80 con 0x7ff8dc152390 2023-11-28T17:41:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.819+0000 7ff8caffd640 1 -- 172.21.15.98:0/1822883415 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d001c400 con 0x7ff8dc152390 2023-11-28T17:41:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.819+0000 7ff8e0e5c640 1 -- 172.21.15.98:0/1822883415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8dc1f2970 con 0x7ff8dc152390 2023-11-28T17:41:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.819+0000 7ff8e0e5c640 1 -- 172.21.15.98:0/1822883415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8dc0bc980 con 0x7ff8dc152390 2023-11-28T17:41:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.820+0000 7ff8e0e5c640 1 -- 172.21.15.98:0/1822883415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8dc074f60 con 0x7ff8dc152390 2023-11-28T17:41:43.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.821+0000 7ff8caffd640 1 -- 172.21.15.98:0/1822883415 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff8d00143f0 con 0x7ff8dc152390 2023-11-28T17:41:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.823+0000 7ff8caffd640 1 --2- 172.21.15.98:0/1822883415 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff8bc06b2d0 0x7ff8bc06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.823+0000 7ff8caffd640 1 -- 172.21.15.98:0/1822883415 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8d0089e30 con 0x7ff8dc152390 2023-11-28T17:41:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.823+0000 7ff8da59c640 1 --2- 172.21.15.98:0/1822883415 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff8bc06b2d0 0x7ff8bc06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.823+0000 7ff8da59c640 1 --2- 172.21.15.98:0/1822883415 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff8bc06b2d0 0x7ff8bc06d790 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc001eb0 tx=0x7ff8cc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:43.827+0000 7ff8caffd640 1 -- 172.21.15.98:0/1822883415 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff8d0054c10 con 0x7ff8dc152390 2023-11-28T17:41:44.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.018+0000 7ff8e0e5c640 1 -- 172.21.15.98:0/1822883415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff8dc1eedd0 con 0x7ff8dc152390 2023-11-28T17:41:44.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.021+0000 7ff8caffd640 1 -- 172.21.15.98:0/1822883415 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff8dc1eedd0 con 0x7ff8dc152390 2023-11-28T17:41:44.022 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-11-28T17:41:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.022+0000 7ff8e0e5c640 1 -- 172.21.15.98:0/1822883415 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff8bc06b2d0 msgr2=0x7ff8bc06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.022+0000 7ff8e0e5c640 1 --2- 172.21.15.98:0/1822883415 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff8bc06b2d0 0x7ff8bc06d790 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc001eb0 tx=0x7ff8cc009040 comp rx=0 tx=0).stop 2023-11-28T17:41:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.022+0000 7ff8e0e5c640 1 -- 172.21.15.98:0/1822883415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8dc152390 msgr2=0x7ff8dc0711c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:44.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.022+0000 7ff8e0e5c640 1 --2- 172.21.15.98:0/1822883415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8dc152390 0x7ff8dc0711c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0010890 tx=0x7ff8d0010d50 comp rx=0 tx=0).stop 2023-11-28T17:41:44.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.023+0000 7ff8e0e5c640 1 -- 172.21.15.98:0/1822883415 shutdown_connections 2023-11-28T17:41:44.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.023+0000 7ff8e0e5c640 1 --2- 172.21.15.98:0/1822883415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8dc15b0d0 0x7ff8dc14eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.023+0000 7ff8e0e5c640 1 --2- 172.21.15.98:0/1822883415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8dc152390 0x7ff8dc0711c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.023+0000 7ff8e0e5c640 1 --2- 172.21.15.98:0/1822883415 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff8bc06b2d0 0x7ff8bc06d790 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.023+0000 7ff8e0e5c640 1 --2- 172.21.15.98:0/1822883415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8c4004d00 0x7ff8dc070c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.023+0000 7ff8e0e5c640 1 -- 172.21.15.98:0/1822883415 >> 172.21.15.98:0/1822883415 conn(0x7ff8dc06c0f0 msgr2=0x7ff8dc06e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:44.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.023+0000 7ff8e0e5c640 1 -- 172.21.15.98:0/1822883415 shutdown_connections 2023-11-28T17:41:44.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.023+0000 7ff8e0e5c640 1 -- 172.21.15.98:0/1822883415 wait complete. 2023-11-28T17:41:44.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:41:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.258+0000 7f74892d3640 1 Processor -- start 2023-11-28T17:41:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.258+0000 7f74892d3640 1 -- start start 2023-11-28T17:41:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.260+0000 7f74892d3640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f748415dd70 0x7f74840c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.260+0000 7f74892d3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74840c8010 0x7f748415a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.260+0000 7f74892d3640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748415e190 con 0x7f7484154040 2023-11-28T17:41:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.260+0000 7f74892d3640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74840c2ad0 con 0x7f74840c8010 2023-11-28T17:41:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.260+0000 7f74892d3640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7484154420 con 0x7f748415dd70 2023-11-28T17:41:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.260+0000 7f74837fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74840c8010 0x7f748415a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.260+0000 7f74837fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74840c8010 0x7f748415a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36700/0 (socket says 172.21.15.98:36700) 2023-11-28T17:41:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.260+0000 7f74837fe640 1 -- 172.21.15.98:0/3526667077 learned_addr learned my addr 172.21.15.98:0/3526667077 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.260+0000 7f747bfff640 1 --2- 172.21.15.98:0/3526667077 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f748415dd70 0x7f74840c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.260+0000 7f74837fe640 1 -- 172.21.15.98:0/3526667077 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f748415dd70 msgr2=0x7f74840c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:44.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.260+0000 7f74837fe640 1 --2- 172.21.15.98:0/3526667077 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f748415dd70 0x7f74840c7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.260+0000 7f74837fe640 1 -- 172.21.15.98:0/3526667077 >> v1:172.21.15.113:6789/0 conn(0x7f7484154040 legacy=0x7f74840c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:44.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.260+0000 7f74837fe640 1 -- 172.21.15.98:0/3526667077 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7484159c50 con 0x7f74840c8010 2023-11-28T17:41:44.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.261+0000 7f74837fe640 1 --2- 172.21.15.98:0/3526667077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74840c8010 0x7f748415a150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f747c009e50 tx=0x7f747c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236b84e64216c41d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:44.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.261+0000 7f74827fc640 1 -- 172.21.15.98:0/3526667077 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c02cb00 con 0x7f74840c8010 2023-11-28T17:41:44.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.262+0000 7f74827fc640 1 -- 172.21.15.98:0/3526667077 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747c02cc60 con 0x7f74840c8010 2023-11-28T17:41:44.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.262+0000 7f74827fc640 1 -- 172.21.15.98:0/3526667077 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c0343f0 con 0x7f74840c8010 2023-11-28T17:41:44.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.262+0000 7f74892d3640 1 -- 172.21.15.98:0/3526667077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74840c8010 msgr2=0x7f748415a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:44.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.262+0000 7f74892d3640 1 --2- 172.21.15.98:0/3526667077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74840c8010 0x7f748415a150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f747c009e50 tx=0x7f747c02bf60 comp rx=0 tx=0).stop 2023-11-28T17:41:44.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.262+0000 7f74892d3640 1 -- 172.21.15.98:0/3526667077 shutdown_connections 2023-11-28T17:41:44.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.262+0000 7f74892d3640 1 --2- 172.21.15.98:0/3526667077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74840c8010 0x7f748415a150 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.262+0000 7f74892d3640 1 --2- 172.21.15.98:0/3526667077 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f748415dd70 0x7f74840c7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.262+0000 7f74892d3640 1 -- 172.21.15.98:0/3526667077 >> 172.21.15.98:0/3526667077 conn(0x7f748406a4d0 msgr2=0x7f748406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:44.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.262+0000 7f74892d3640 1 -- 172.21.15.98:0/3526667077 shutdown_connections 2023-11-28T17:41:44.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.262+0000 7f74892d3640 1 -- 172.21.15.98:0/3526667077 wait complete. 2023-11-28T17:41:44.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.264+0000 7f74892d3640 1 Processor -- start 2023-11-28T17:41:44.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.264+0000 7f74892d3640 1 -- start start 2023-11-28T17:41:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.265+0000 7f74892d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74840c8010 0x7f7484152930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.265+0000 7f74892d3640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7484154040 0x7f74841f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.265+0000 7f74892d3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f748415dd70 0x7f748415c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.265+0000 7f74892d3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7484152510 con 0x7f74840c8010 2023-11-28T17:41:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.265+0000 7f74892d3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74841556f0 con 0x7f748415dd70 2023-11-28T17:41:44.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.265+0000 7f74892d3640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f748414faf0 con 0x7f7484154040 2023-11-28T17:41:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.265+0000 7f7482ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74840c8010 0x7f7484152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.265+0000 7f7482ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74840c8010 0x7f7484152930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36710/0 (socket says 172.21.15.98:36710) 2023-11-28T17:41:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.265+0000 7f7482ffd640 1 -- 172.21.15.98:0/1411157090 learned_addr learned my addr 172.21.15.98:0/1411157090 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.266+0000 7f747bfff640 1 --2- 172.21.15.98:0/1411157090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7484154040 0x7f74841f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.266+0000 7f7482ffd640 1 -- 172.21.15.98:0/1411157090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7484154040 msgr2=0x7f74841f2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.266+0000 7f7482ffd640 1 --2- 172.21.15.98:0/1411157090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7484154040 0x7f74841f2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.266+0000 7f7482ffd640 1 -- 172.21.15.98:0/1411157090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f748415dd70 msgr2=0x7f748415c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.266+0000 7f74837fe640 1 --2- 172.21.15.98:0/1411157090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f748415dd70 0x7f748415c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.266+0000 7f7482ffd640 1 --2- 172.21.15.98:0/1411157090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f748415dd70 0x7f748415c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.266+0000 7f7482ffd640 1 -- 172.21.15.98:0/1411157090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c009b30 con 0x7f74840c8010 2023-11-28T17:41:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.266+0000 7f7482ffd640 1 --2- 172.21.15.98:0/1411157090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74840c8010 0x7f7484152930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f747400c460 tx=0x7f747400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.266+0000 7f747b7fe640 1 -- 172.21.15.98:0/1411157090 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474012830 con 0x7f74840c8010 2023-11-28T17:41:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.266+0000 7f747b7fe640 1 -- 172.21.15.98:0/1411157090 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74740129d0 con 0x7f74840c8010 2023-11-28T17:41:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.266+0000 7f747b7fe640 1 -- 172.21.15.98:0/1411157090 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474012d80 con 0x7f74840c8010 2023-11-28T17:41:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.266+0000 7f74892d3640 1 -- 172.21.15.98:0/1411157090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7484151640 con 0x7f74840c8010 2023-11-28T17:41:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.267+0000 7f74892d3640 1 -- 172.21.15.98:0/1411157090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7484150030 con 0x7f74840c8010 2023-11-28T17:41:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.268+0000 7f74892d3640 1 -- 172.21.15.98:0/1411157090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7484067bf0 con 0x7f74840c8010 2023-11-28T17:41:44.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.268+0000 7f747b7fe640 1 -- 172.21.15.98:0/1411157090 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f747401b400 con 0x7f74840c8010 2023-11-28T17:41:44.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.269+0000 7f747b7fe640 1 --2- 172.21.15.98:0/1411157090 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f746406b2d0 0x7f746406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:44.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.269+0000 7f747b7fe640 1 -- 172.21.15.98:0/1411157090 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7474089e70 con 0x7f74840c8010 2023-11-28T17:41:44.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.269+0000 7f747bfff640 1 --2- 172.21.15.98:0/1411157090 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f746406b2d0 0x7f746406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:44.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.270+0000 7f747bfff640 1 --2- 172.21.15.98:0/1411157090 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f746406b2d0 0x7f746406d790 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f746c006fd0 tx=0x7f746c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:44.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.275+0000 7f747b7fe640 1 -- 172.21.15.98:0/1411157090 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7474054c40 con 0x7f74840c8010 2023-11-28T17:41:44.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.509+0000 7f74892d3640 1 -- 172.21.15.98:0/1411157090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7484074f60 con 0x7f74840c8010 2023-11-28T17:41:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.510+0000 7f747b7fe640 1 -- 172.21.15.98:0/1411157090 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6560 (secure 0 0 0) 0x7f7474058260 con 0x7f74840c8010 2023-11-28T17:41:44.512 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:44.512 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2429440814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2429440814},{"type":"v1","addr":"172.21.15.98:6836","nonce":2429440814}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:41:16.691765+0000","modified":"2023-11-28T17:41:39.306297+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4273,"mds_2":4272,"mds_3":4270,"mds_4":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"d","rank":4,"incarnation":15,"state":"up:creating","state_seq":1,"addr":"172.21.15.113:6835/1057450253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1057450253},{"type":"v1","addr":"172.21.15.113:6835","nonce":1057450253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"c","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.98:6837/1925178583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1925178583},{"type":"v1","addr":"172.21.15.98:6837","nonce":1925178583}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4272":{"gid":4272,"name":"f","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.113:6837/1732613907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":1732613907},{"type":"v1","addr":"172.21.15.113:6837","nonce":1732613907}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4273":{"gid":4273,"name":"e","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6839/3585869153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3585869153},{"type":"v1","addr":"172.21.15.98:6839","nonce":3585869153}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.113:6834/2183585971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2183585971},{"type":"v1","addr":"172.21.15.113:6834","nonce":2183585971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T17:41:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 15 2023-11-28T17:41:44.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.514+0000 7f74892d3640 1 -- 172.21.15.98:0/1411157090 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f746406b2d0 msgr2=0x7f746406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:44.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.514+0000 7f74892d3640 1 --2- 172.21.15.98:0/1411157090 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f746406b2d0 0x7f746406d790 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f746c006fd0 tx=0x7f746c008040 comp rx=0 tx=0).stop 2023-11-28T17:41:44.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.514+0000 7f74892d3640 1 -- 172.21.15.98:0/1411157090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74840c8010 msgr2=0x7f7484152930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:44.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.514+0000 7f74892d3640 1 --2- 172.21.15.98:0/1411157090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74840c8010 0x7f7484152930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f747400c460 tx=0x7f747400c920 comp rx=0 tx=0).stop 2023-11-28T17:41:44.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.514+0000 7f74892d3640 1 -- 172.21.15.98:0/1411157090 shutdown_connections 2023-11-28T17:41:44.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.514+0000 7f74892d3640 1 --2- 172.21.15.98:0/1411157090 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f746406b2d0 0x7f746406d790 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.514+0000 7f74892d3640 1 --2- 172.21.15.98:0/1411157090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f748415dd70 0x7f748415c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.514+0000 7f74892d3640 1 --2- 172.21.15.98:0/1411157090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7484154040 0x7f74841f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.514+0000 7f74892d3640 1 --2- 172.21.15.98:0/1411157090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74840c8010 0x7f7484152930 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.514+0000 7f74892d3640 1 -- 172.21.15.98:0/1411157090 >> 172.21.15.98:0/1411157090 conn(0x7f748406a4d0 msgr2=0x7f74840c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:44.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.514+0000 7f74892d3640 1 -- 172.21.15.98:0/1411157090 shutdown_connections 2023-11-28T17:41:44.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.514+0000 7f74892d3640 1 -- 172.21.15.98:0/1411157090 wait complete. 2023-11-28T17:41:44.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:41:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.743+0000 7f3bab9c5640 1 Processor -- start 2023-11-28T17:41:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.743+0000 7f3bab9c5640 1 -- start start 2023-11-28T17:41:44.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.745+0000 7f3bab9c5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3ba4150c00 0x7f3ba40c2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:44.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.745+0000 7f3bab9c5640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3ba40c6600 0x7f3ba40c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:44.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.745+0000 7f3bab9c5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba4157850 con 0x7f3ba40c5c00 2023-11-28T17:41:44.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.745+0000 7f3bab9c5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba40c6fd0 con 0x7f3ba4150c00 2023-11-28T17:41:44.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.745+0000 7f3bab9c5640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba4156980 con 0x7f3ba40c6600 2023-11-28T17:41:44.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.745+0000 7f3ba8f60640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3ba40c5c00 0x7f3ba40c44e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55540/0 (socket says 172.21.15.98:55540) 2023-11-28T17:41:44.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.745+0000 7f3ba8f60640 1 -- 172.21.15.98:0/86669741 learned_addr learned my addr 172.21.15.98:0/86669741 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:44.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.745+0000 7f3ba9761640 1 --2- 172.21.15.98:0/86669741 >> v2:172.21.15.98:3300/0 conn(0x7f3ba4150c00 0x7f3ba40c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:44.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.746+0000 7f3ba9f62640 1 --2- 172.21.15.98:0/86669741 >> v2:172.21.15.113:3300/0 conn(0x7f3ba40c6600 0x7f3ba40c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.746+0000 7f3b9bfff640 1 -- 172.21.15.98:0/86669741 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604913143 0 0) 0x7f3ba4157850 con 0x7f3ba40c5c00 2023-11-28T17:41:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.746+0000 7f3b9bfff640 1 -- 172.21.15.98:0/86669741 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b90003520 con 0x7f3ba40c5c00 2023-11-28T17:41:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.746+0000 7f3ba9761640 1 -- 172.21.15.98:0/86669741 >> v2:172.21.15.113:3300/0 conn(0x7f3ba40c6600 msgr2=0x7f3ba40c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.746+0000 7f3ba9761640 1 --2- 172.21.15.98:0/86669741 >> v2:172.21.15.113:3300/0 conn(0x7f3ba40c6600 0x7f3ba40c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.746+0000 7f3ba9761640 1 -- 172.21.15.98:0/86669741 >> v1:172.21.15.98:6789/0 conn(0x7f3ba40c5c00 legacy=0x7f3ba40c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.746+0000 7f3ba9761640 1 -- 172.21.15.98:0/86669741 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba4156c40 con 0x7f3ba4150c00 2023-11-28T17:41:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.747+0000 7f3ba9761640 1 --2- 172.21.15.98:0/86669741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba4150c00 0x7f3ba40c2700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3ba00097a0 tx=0x7f3ba002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1656ab0ccec7f62 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.747+0000 7f3b9bfff640 1 -- 172.21.15.98:0/86669741 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0031830 con 0x7f3ba4150c00 2023-11-28T17:41:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.747+0000 7f3b9bfff640 1 -- 172.21.15.98:0/86669741 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba0031990 con 0x7f3ba4150c00 2023-11-28T17:41:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.747+0000 7f3b9bfff640 1 -- 172.21.15.98:0/86669741 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0031d00 con 0x7f3ba4150c00 2023-11-28T17:41:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.748+0000 7f3bab9c5640 1 -- 172.21.15.98:0/86669741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba4150c00 msgr2=0x7f3ba40c2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.748+0000 7f3bab9c5640 1 --2- 172.21.15.98:0/86669741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba4150c00 0x7f3ba40c2700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3ba00097a0 tx=0x7f3ba002b8d0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.748+0000 7f3bab9c5640 1 -- 172.21.15.98:0/86669741 shutdown_connections 2023-11-28T17:41:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.748+0000 7f3bab9c5640 1 --2- 172.21.15.98:0/86669741 >> v2:172.21.15.113:3300/0 conn(0x7f3ba40c6600 0x7f3ba40c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.748+0000 7f3bab9c5640 1 --2- 172.21.15.98:0/86669741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba4150c00 0x7f3ba40c2700 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.748+0000 7f3bab9c5640 1 -- 172.21.15.98:0/86669741 >> 172.21.15.98:0/86669741 conn(0x7f3ba406a4d0 msgr2=0x7f3ba406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.748+0000 7f3bab9c5640 1 -- 172.21.15.98:0/86669741 shutdown_connections 2023-11-28T17:41:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.748+0000 7f3bab9c5640 1 -- 172.21.15.98:0/86669741 wait complete. 2023-11-28T17:41:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.750+0000 7f3bab9c5640 1 Processor -- start 2023-11-28T17:41:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.750+0000 7f3bab9c5640 1 -- start start 2023-11-28T17:41:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3bab9c5640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ba40c5c00 0x7f3ba4148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3bab9c5640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ba40c6600 0x7f3ba40cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3bab9c5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba4150c00 0x7f3ba41ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3bab9c5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba41f3180 con 0x7f3ba4150c00 2023-11-28T17:41:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3bab9c5640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba41f35a0 con 0x7f3ba40c5c00 2023-11-28T17:41:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3bab9c5640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba4070ce0 con 0x7f3ba40c6600 2023-11-28T17:41:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3ba9f62640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba4150c00 0x7f3ba41ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3ba9f62640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba4150c00 0x7f3ba41ef0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36720/0 (socket says 172.21.15.98:36720) 2023-11-28T17:41:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3ba9f62640 1 -- 172.21.15.98:0/3806236916 learned_addr learned my addr 172.21.15.98:0/3806236916 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3ba9761640 1 --2- 172.21.15.98:0/3806236916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ba40c5c00 0x7f3ba4148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3ba8f60640 1 --2- 172.21.15.98:0/3806236916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ba40c6600 0x7f3ba40cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3ba9f62640 1 -- 172.21.15.98:0/3806236916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ba40c6600 msgr2=0x7f3ba40cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3ba9f62640 1 --2- 172.21.15.98:0/3806236916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ba40c6600 0x7f3ba40cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3ba9f62640 1 -- 172.21.15.98:0/3806236916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ba40c5c00 msgr2=0x7f3ba4148be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3ba9f62640 1 --2- 172.21.15.98:0/3806236916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ba40c5c00 0x7f3ba4148be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3ba9f62640 1 -- 172.21.15.98:0/3806236916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba0009400 con 0x7f3ba4150c00 2023-11-28T17:41:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.752+0000 7f3ba8f60640 1 --2- 172.21.15.98:0/3806236916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ba40c6600 0x7f3ba40cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.753+0000 7f3ba9f62640 1 --2- 172.21.15.98:0/3806236916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba4150c00 0x7f3ba41ef0f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c007e90 tx=0x7f3b9c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.753+0000 7f3b99ffb640 1 -- 172.21.15.98:0/3806236916 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c014420 con 0x7f3ba4150c00 2023-11-28T17:41:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.753+0000 7f3b99ffb640 1 -- 172.21.15.98:0/3806236916 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9c014580 con 0x7f3ba4150c00 2023-11-28T17:41:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.753+0000 7f3b99ffb640 1 -- 172.21.15.98:0/3806236916 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c014930 con 0x7f3ba4150c00 2023-11-28T17:41:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.753+0000 7f3bab9c5640 1 -- 172.21.15.98:0/3806236916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba41f27e0 con 0x7f3ba4150c00 2023-11-28T17:41:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.753+0000 7f3bab9c5640 1 -- 172.21.15.98:0/3806236916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba41f2da0 con 0x7f3ba4150c00 2023-11-28T17:41:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.755+0000 7f3bab9c5640 1 -- 172.21.15.98:0/3806236916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba4067bf0 con 0x7f3ba4150c00 2023-11-28T17:41:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.756+0000 7f3b99ffb640 1 -- 172.21.15.98:0/3806236916 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3b9c014a90 con 0x7f3ba4150c00 2023-11-28T17:41:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.757+0000 7f3b99ffb640 1 --2- 172.21.15.98:0/3806236916 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3b8806b2d0 0x7f3b8806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.757+0000 7f3b99ffb640 1 -- 172.21.15.98:0/3806236916 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b9c08a420 con 0x7f3ba4150c00 2023-11-28T17:41:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.757+0000 7f3ba9761640 1 --2- 172.21.15.98:0/3806236916 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3b8806b2d0 0x7f3b8806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.758+0000 7f3ba9761640 1 --2- 172.21.15.98:0/3806236916 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3b8806b2d0 0x7f3b8806d790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0036000 tx=0x7f3ba003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.761+0000 7f3b99ffb640 1 -- 172.21.15.98:0/3806236916 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b9c055320 con 0x7f3ba4150c00 2023-11-28T17:41:44.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.953+0000 7f3bab9c5640 1 -- 172.21.15.98:0/3806236916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ba41f32c0 con 0x7f3ba4150c00 2023-11-28T17:41:44.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.955+0000 7f3b99ffb640 1 -- 172.21.15.98:0/3806236916 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16517 (secure 0 0 0) 0x7f3ba41f32c0 con 0x7f3ba4150c00 2023-11-28T17:41:44.956 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:44.956 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:41:18.195933+0000","last_up_change":"2023-11-28T17:41:09.556993+0000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:41:10.303280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:41:14.890871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:41:15.704103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6810","nonce":2188704385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6813","nonce":2188704385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6819","nonce":2188704385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6816","nonce":2188704385}]},"public_addr":"172.21.15.98:6810/2188704385","cluster_addr":"172.21.15.98:6813/2188704385","heartbeat_back_addr":"172.21.15.98:6819/2188704385","heartbeat_front_addr":"172.21.15.98:6816/2188704385","state":["exists","up"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6801","nonce":3918226860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6803","nonce":3918226860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6807","nonce":3918226860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6805","nonce":3918226860}]},"public_addr":"172.21.15.98:6801/3918226860","cluster_addr":"172.21.15.98:6803/3918226860","heartbeat_back_addr":"172.21.15.98:6807/3918226860","heartbeat_front_addr":"172.21.15.98:6805/3918226860","state":["exists","up"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6812","nonce":2691532400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6815","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6818","nonce":2691532400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6823","nonce":2691532400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6821","nonce":2691532400}]},"public_addr":"172.21.15.98:6812/2691532400","cluster_addr":"172.21.15.98:6818/2691532400","heartbeat_back_addr":"172.21.15.98:6823/2691532400","heartbeat_front_addr":"172.21.15.98:6821/2691532400","state":["exists","up"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6825","nonce":1905487008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6827","nonce":1905487008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6831","nonce":1905487008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6829","nonce":1905487008}]},"public_addr":"172.21.15.98:6825/1905487008","cluster_addr":"172.21.15.98:6827/1905487008","heartbeat_back_addr":"172.21.15.98:6831/1905487008","heartbeat_front_addr":"172.21.15.98:6829/1905487008","state":["exists","up"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6801","nonce":2974677215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6803","nonce":2974677215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6807","nonce":2974677215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6805","nonce":2974677215}]},"public_addr":"172.21.15.113:6801/2974677215","cluster_addr":"172.21.15.113:6803/2974677215","heartbeat_back_addr":"172.21.15.113:6807/2974677215","heartbeat_front_addr":"172.21.15.113:6805/2974677215","state":["exists","up"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6817","nonce":2685000355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6821","nonce":2685000355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6829","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6831","nonce":2685000355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6823","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6826","nonce":2685000355}]},"public_addr":"172.21.15.113:6817/2685000355","cluster_addr":"172.21.15.113:6821/2685000355","heartbeat_back_addr":"172.21.15.113:6831/2685000355","heartbeat_front_addr":"172.21.15.113:6826/2685000355","state":["exists","up"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6809","nonce":2764300831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6811","nonce":2764300831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6815","nonce":2764300831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6813","nonce":2764300831}]},"public_addr":"172.21.15.113:6809/2764300831","cluster_addr":"172.21.15.113:6811/2764300831","heartbeat_back_addr":"172.21.15.113:6815/2764300831","heartbeat_front_addr":"172.21.15.113:6813/2764300831","state":["exists","up"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6819","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6820","nonce":1656248400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6824","nonce":1656248400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6830","nonce":1656248400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6825","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6827","nonce":1656248400}]},"public_addr":"172.21.15.113:6820/1656248400","cluster_addr":"172.21.15.113:6824/1656248400","heartbeat_back_addr":"172.21.15.113:6830/1656248400","heartbeat_front_addr":"172.21.15.113:6827/1656248400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.379549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.312196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.117494+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.388128+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.360608+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.411101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.355995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.390362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:41:44.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.957+0000 7f3bab9c5640 1 -- 172.21.15.98:0/3806236916 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3b8806b2d0 msgr2=0x7f3b8806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:44.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.957+0000 7f3bab9c5640 1 --2- 172.21.15.98:0/3806236916 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3b8806b2d0 0x7f3b8806d790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0036000 tx=0x7f3ba003c040 comp rx=0 tx=0).stop 2023-11-28T17:41:44.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.957+0000 7f3bab9c5640 1 -- 172.21.15.98:0/3806236916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba4150c00 msgr2=0x7f3ba41ef0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:44.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.957+0000 7f3bab9c5640 1 --2- 172.21.15.98:0/3806236916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba4150c00 0x7f3ba41ef0f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c007e90 tx=0x7f3b9c00e4b0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.958+0000 7f3bab9c5640 1 -- 172.21.15.98:0/3806236916 shutdown_connections 2023-11-28T17:41:44.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.958+0000 7f3bab9c5640 1 --2- 172.21.15.98:0/3806236916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ba4150c00 0x7f3ba41ef0f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.958+0000 7f3bab9c5640 1 --2- 172.21.15.98:0/3806236916 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ba40c6600 0x7f3ba40cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.958+0000 7f3bab9c5640 1 --2- 172.21.15.98:0/3806236916 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3b8806b2d0 0x7f3b8806d790 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.958+0000 7f3bab9c5640 1 --2- 172.21.15.98:0/3806236916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ba40c5c00 0x7f3ba4148be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:44.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.958+0000 7f3bab9c5640 1 -- 172.21.15.98:0/3806236916 >> 172.21.15.98:0/3806236916 conn(0x7f3ba406a4d0 msgr2=0x7f3ba40c71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:44.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.958+0000 7f3bab9c5640 1 -- 172.21.15.98:0/3806236916 shutdown_connections 2023-11-28T17:41:44.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:44.958+0000 7f3bab9c5640 1 -- 172.21.15.98:0/3806236916 wait complete. 2023-11-28T17:41:44.976 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:41:10.303280+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:41:14.890871+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:41:15.704103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:41:44.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:41:45.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.187+0000 7f4dfa783640 1 Processor -- start 2023-11-28T17:41:45.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.187+0000 7f4dfa783640 1 -- start start 2023-11-28T17:41:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.188+0000 7f4dfa783640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4df4151be0 0x7f4df40c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.188+0000 7f4dfa783640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4df41584d0 0x7f4df406dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.188+0000 7f4dfa783640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df40c8150 con 0x7f4df40c5a60 2023-11-28T17:41:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.188+0000 7f4dfa783640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df40c8850 con 0x7f4df4151be0 2023-11-28T17:41:45.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.188+0000 7f4dfa783640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df40c5f50 con 0x7f4df41584d0 2023-11-28T17:41:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.189+0000 7f4df3fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4df4151be0 0x7f4df40c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.189+0000 7f4debfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4df40c5a60 0x7f4df40c4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55550/0 (socket says 172.21.15.98:55550) 2023-11-28T17:41:45.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.189+0000 7f4debfff640 1 -- 172.21.15.98:0/1701946196 learned_addr learned my addr 172.21.15.98:0/1701946196 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.189+0000 7f4df8d20640 1 --2- 172.21.15.98:0/1701946196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4df41584d0 0x7f4df406dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.189+0000 7f4df3fff640 1 -- 172.21.15.98:0/1701946196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4df41584d0 msgr2=0x7f4df406dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.189+0000 7f4df3fff640 1 --2- 172.21.15.98:0/1701946196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4df41584d0 0x7f4df406dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.189+0000 7f4df3fff640 1 -- 172.21.15.98:0/1701946196 >> v1:172.21.15.98:6789/0 conn(0x7f4df40c5a60 legacy=0x7f4df40c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.189+0000 7f4df3fff640 1 -- 172.21.15.98:0/1701946196 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df40c7b90 con 0x7f4df4151be0 2023-11-28T17:41:45.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.189+0000 7f4df8d20640 1 --2- 172.21.15.98:0/1701946196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4df41584d0 0x7f4df406dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.190+0000 7f4df3fff640 1 --2- 172.21.15.98:0/1701946196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df4151be0 0x7f4df40c75a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4de40097a0 tx=0x7f4de402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b3f771a7a4cb36a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.190+0000 7f4df37fe640 1 -- 172.21.15.98:0/1701946196 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de4031830 con 0x7f4df4151be0 2023-11-28T17:41:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.191+0000 7f4df37fe640 1 -- 172.21.15.98:0/1701946196 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de4031990 con 0x7f4df4151be0 2023-11-28T17:41:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.191+0000 7f4df37fe640 1 -- 172.21.15.98:0/1701946196 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de4031d20 con 0x7f4df4151be0 2023-11-28T17:41:45.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.191+0000 7f4dfa783640 1 -- 172.21.15.98:0/1701946196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df4151be0 msgr2=0x7f4df40c75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.191+0000 7f4dfa783640 1 --2- 172.21.15.98:0/1701946196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df4151be0 0x7f4df40c75a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4de40097a0 tx=0x7f4de402b8d0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.191+0000 7f4dfa783640 1 -- 172.21.15.98:0/1701946196 shutdown_connections 2023-11-28T17:41:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.191+0000 7f4dfa783640 1 --2- 172.21.15.98:0/1701946196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4df41584d0 0x7f4df406dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.191+0000 7f4dfa783640 1 --2- 172.21.15.98:0/1701946196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df4151be0 0x7f4df40c75a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.191+0000 7f4dfa783640 1 -- 172.21.15.98:0/1701946196 >> 172.21.15.98:0/1701946196 conn(0x7f4df406c0f0 msgr2=0x7f4df406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.191+0000 7f4dfa783640 1 -- 172.21.15.98:0/1701946196 shutdown_connections 2023-11-28T17:41:45.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.191+0000 7f4dfa783640 1 -- 172.21.15.98:0/1701946196 wait complete. 2023-11-28T17:41:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.193+0000 7f4dfa783640 1 Processor -- start 2023-11-28T17:41:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.193+0000 7f4dfa783640 1 -- start start 2023-11-28T17:41:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.194+0000 7f4dfa783640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df40c5a60 0x7f4df415a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.194+0000 7f4dfa783640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4df4151be0 0x7f4df41ed590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.194+0000 7f4dfa783640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4df41584d0 0x7f4df4159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.194+0000 7f4dfa783640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df415ac60 con 0x7f4df40c5a60 2023-11-28T17:41:45.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.194+0000 7f4dfa783640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df41edad0 con 0x7f4df41584d0 2023-11-28T17:41:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.194+0000 7f4dfa783640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df415a1e0 con 0x7f4df4151be0 2023-11-28T17:41:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.194+0000 7f4df3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df40c5a60 0x7f4df415a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.194+0000 7f4df3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df40c5a60 0x7f4df415a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36736/0 (socket says 172.21.15.98:36736) 2023-11-28T17:41:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.194+0000 7f4df3fff640 1 -- 172.21.15.98:0/1069347900 learned_addr learned my addr 172.21.15.98:0/1069347900 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.195+0000 7f4df3fff640 1 -- 172.21.15.98:0/1069347900 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4df4151be0 msgr2=0x7f4df41ed590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.195+0000 7f4df3fff640 1 --2- 172.21.15.98:0/1069347900 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4df4151be0 0x7f4df41ed590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.195+0000 7f4df3fff640 1 -- 172.21.15.98:0/1069347900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4df41584d0 msgr2=0x7f4df4159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.195+0000 7f4df3fff640 1 --2- 172.21.15.98:0/1069347900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4df41584d0 0x7f4df4159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.195+0000 7f4df3fff640 1 -- 172.21.15.98:0/1069347900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de4009400 con 0x7f4df40c5a60 2023-11-28T17:41:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.195+0000 7f4df3fff640 1 --2- 172.21.15.98:0/1069347900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df40c5a60 0x7f4df415a720 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4de402bee0 tx=0x7f4de40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.195+0000 7f4df17fa640 1 -- 172.21.15.98:0/1069347900 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de403ba70 con 0x7f4df40c5a60 2023-11-28T17:41:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.195+0000 7f4df17fa640 1 -- 172.21.15.98:0/1069347900 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de403bbf0 con 0x7f4df40c5a60 2023-11-28T17:41:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.195+0000 7f4df17fa640 1 -- 172.21.15.98:0/1069347900 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de40453f0 con 0x7f4df40c5a60 2023-11-28T17:41:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.195+0000 7f4dfa783640 1 -- 172.21.15.98:0/1069347900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df415a4c0 con 0x7f4df40c5a60 2023-11-28T17:41:45.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.195+0000 7f4dfa783640 1 -- 172.21.15.98:0/1069347900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df41499b0 con 0x7f4df40c5a60 2023-11-28T17:41:45.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.197+0000 7f4dfa783640 1 -- 172.21.15.98:0/1069347900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df4074f60 con 0x7f4df40c5a60 2023-11-28T17:41:45.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.197+0000 7f4df17fa640 1 -- 172.21.15.98:0/1069347900 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4de404e450 con 0x7f4df40c5a60 2023-11-28T17:41:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.198+0000 7f4df17fa640 1 --2- 172.21.15.98:0/1069347900 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4dd406b2a0 0x7f4dd406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.198+0000 7f4df17fa640 1 -- 172.21.15.98:0/1069347900 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4de40abef0 con 0x7f4df40c5a60 2023-11-28T17:41:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.198+0000 7f4debfff640 1 --2- 172.21.15.98:0/1069347900 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4dd406b2a0 0x7f4dd406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.199+0000 7f4debfff640 1 --2- 172.21.15.98:0/1069347900 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4dd406b2a0 0x7f4dd406d760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4df406c920 tx=0x7f4ddc007a40 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.202+0000 7f4df17fa640 1 -- 172.21.15.98:0/1069347900 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4de4076cc0 con 0x7f4df40c5a60 2023-11-28T17:41:45.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.398+0000 7f4dfa783640 1 -- 172.21.15.98:0/1069347900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4df40cbcf0 con 0x7f4df40c5a60 2023-11-28T17:41:45.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.398+0000 7f4df17fa640 1 -- 172.21.15.98:0/1069347900 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4de40350a0 con 0x7f4df40c5a60 2023-11-28T17:41:45.400 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-11-28T17:41:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.401+0000 7f4dfa783640 1 -- 172.21.15.98:0/1069347900 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4dd406b2a0 msgr2=0x7f4dd406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.401+0000 7f4dfa783640 1 --2- 172.21.15.98:0/1069347900 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4dd406b2a0 0x7f4dd406d760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4df406c920 tx=0x7f4ddc007a40 comp rx=0 tx=0).stop 2023-11-28T17:41:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.401+0000 7f4dfa783640 1 -- 172.21.15.98:0/1069347900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df40c5a60 msgr2=0x7f4df415a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.401+0000 7f4dfa783640 1 --2- 172.21.15.98:0/1069347900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df40c5a60 0x7f4df415a720 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4de402bee0 tx=0x7f4de40046c0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.401+0000 7f4dfa783640 1 -- 172.21.15.98:0/1069347900 shutdown_connections 2023-11-28T17:41:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.401+0000 7f4dfa783640 1 --2- 172.21.15.98:0/1069347900 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4dd406b2a0 0x7f4dd406d760 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.401+0000 7f4dfa783640 1 --2- 172.21.15.98:0/1069347900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4df41584d0 0x7f4df4159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.401+0000 7f4dfa783640 1 --2- 172.21.15.98:0/1069347900 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4df4151be0 0x7f4df41ed590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.401+0000 7f4dfa783640 1 --2- 172.21.15.98:0/1069347900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4df40c5a60 0x7f4df415a720 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.401+0000 7f4dfa783640 1 -- 172.21.15.98:0/1069347900 >> 172.21.15.98:0/1069347900 conn(0x7f4df406c0f0 msgr2=0x7f4df4070270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.401+0000 7f4dfa783640 1 -- 172.21.15.98:0/1069347900 shutdown_connections 2023-11-28T17:41:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.401+0000 7f4dfa783640 1 -- 172.21.15.98:0/1069347900 wait complete. 2023-11-28T17:41:45.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:41:45.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.636+0000 7fba89735640 1 Processor -- start 2023-11-28T17:41:45.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.636+0000 7fba89735640 1 -- start start 2023-11-28T17:41:45.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.637+0000 7fba89735640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fba840c69b0 0x7fba840c7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:45.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.637+0000 7fba89735640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840c80d0 con 0x7fba840c7c60 2023-11-28T17:41:45.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.638+0000 7fba89735640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8415d230 con 0x7fba84153c80 2023-11-28T17:41:45.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.638+0000 7fba89735640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8415e070 con 0x7fba840c69b0 2023-11-28T17:41:45.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.638+0000 7fba88f34640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fba840c7c60 0x7fba84164440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55554/0 (socket says 172.21.15.98:55554) 2023-11-28T17:41:45.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.638+0000 7fba88f34640 1 -- 172.21.15.98:0/1738596180 learned_addr learned my addr 172.21.15.98:0/1738596180 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:45.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.638+0000 7fba837fe640 1 --2- 172.21.15.98:0/1738596180 >> v2:172.21.15.113:3300/0 conn(0x7fba840c69b0 0x7fba840c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:45.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.638+0000 7fba82ffd640 1 -- 172.21.15.98:0/1738596180 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814378089 0 0) 0x7fba840c80d0 con 0x7fba840c7c60 2023-11-28T17:41:45.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.638+0000 7fba82ffd640 1 -- 172.21.15.98:0/1738596180 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba6c003520 con 0x7fba840c7c60 2023-11-28T17:41:45.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba82ffd640 1 -- 172.21.15.98:0/1738596180 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 298668757 0 0) 0x7fba6c003520 con 0x7fba840c7c60 2023-11-28T17:41:45.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba82ffd640 1 -- 172.21.15.98:0/1738596180 >> v2:172.21.15.113:3300/0 conn(0x7fba840c69b0 msgr2=0x7fba840c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:45.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba82ffd640 1 --2- 172.21.15.98:0/1738596180 >> v2:172.21.15.113:3300/0 conn(0x7fba840c69b0 0x7fba840c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba82ffd640 1 -- 172.21.15.98:0/1738596180 >> v1:172.21.15.113:6789/0 conn(0x7fba84153c80 legacy=0x7fba8406b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:45.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba82ffd640 1 -- 172.21.15.98:0/1738596180 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8415d4f0 con 0x7fba840c7c60 2023-11-28T17:41:45.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba82ffd640 1 -- 172.21.15.98:0/1738596180 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7fba7c0019f0 con 0x7fba840c7c60 2023-11-28T17:41:45.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba82ffd640 1 -- 172.21.15.98:0/1738596180 >> v1:172.21.15.98:6789/0 conn(0x7fba840c7c60 legacy=0x7fba84164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:45.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba82ffd640 1 --2- 172.21.15.98:0/1738596180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0039c0 0x7fba6c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:45.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba82ffd640 1 --2- 172.21.15.98:0/1738596180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba6c0047c0 0x7fba6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba82ffd640 1 --2- 172.21.15.98:0/1738596180 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba6c005180 0x7fba6c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba82ffd640 1 -- 172.21.15.98:0/1738596180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8415e070 con 0x7fba6c0047c0 2023-11-28T17:41:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba82ffd640 1 -- 172.21.15.98:0/1738596180 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c00c060 con 0x7fba6c0039c0 2023-11-28T17:41:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba82ffd640 1 -- 172.21.15.98:0/1738596180 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c00c220 con 0x7fba6c005180 2023-11-28T17:41:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba837fe640 1 --2- 172.21.15.98:0/1738596180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba6c0047c0 0x7fba6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba83fff640 1 --2- 172.21.15.98:0/1738596180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0039c0 0x7fba6c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.639+0000 7fba88f34640 1 --2- 172.21.15.98:0/1738596180 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba6c005180 0x7fba6c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.640+0000 7fba837fe640 1 -- 172.21.15.98:0/1738596180 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba6c005180 msgr2=0x7fba6c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.640+0000 7fba837fe640 1 --2- 172.21.15.98:0/1738596180 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba6c005180 0x7fba6c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.640+0000 7fba837fe640 1 -- 172.21.15.98:0/1738596180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0039c0 msgr2=0x7fba6c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.640+0000 7fba837fe640 1 --2- 172.21.15.98:0/1738596180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0039c0 0x7fba6c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.640+0000 7fba837fe640 1 -- 172.21.15.98:0/1738596180 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c003520 con 0x7fba6c0047c0 2023-11-28T17:41:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.640+0000 7fba83fff640 1 --2- 172.21.15.98:0/1738596180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0039c0 0x7fba6c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:45.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.640+0000 7fba88f34640 1 -- 172.21.15.98:0/1738596180 reap_dead start 2023-11-28T17:41:45.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.641+0000 7fba837fe640 1 --2- 172.21.15.98:0/1738596180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba6c0047c0 0x7fba6c004c40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fba78009ac0 tx=0x7fba6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c658bd703e9a6e51 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:45.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.641+0000 7fba82ffd640 1 -- 172.21.15.98:0/1738596180 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7802c6b0 con 0x7fba6c0047c0 2023-11-28T17:41:45.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.641+0000 7fba82ffd640 1 -- 172.21.15.98:0/1738596180 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7802c810 con 0x7fba6c0047c0 2023-11-28T17:41:45.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.641+0000 7fba82ffd640 1 -- 172.21.15.98:0/1738596180 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7802cba0 con 0x7fba6c0047c0 2023-11-28T17:41:45.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.641+0000 7fba89735640 1 -- 172.21.15.98:0/1738596180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba6c0047c0 msgr2=0x7fba6c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:45.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.641+0000 7fba89735640 1 --2- 172.21.15.98:0/1738596180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba6c0047c0 0x7fba6c004c40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fba78009ac0 tx=0x7fba6c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.642+0000 7fba89735640 1 -- 172.21.15.98:0/1738596180 shutdown_connections 2023-11-28T17:41:45.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.642+0000 7fba89735640 1 --2- 172.21.15.98:0/1738596180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba6c0047c0 0x7fba6c004c40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.642+0000 7fba89735640 1 -- 172.21.15.98:0/1738596180 >> 172.21.15.98:0/1738596180 conn(0x7fba840cf300 msgr2=0x7fba840cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:45.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.642+0000 7fba89735640 1 -- 172.21.15.98:0/1738596180 shutdown_connections 2023-11-28T17:41:45.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.642+0000 7fba89735640 1 -- 172.21.15.98:0/1738596180 wait complete. 2023-11-28T17:41:45.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.643+0000 7fba89735640 1 Processor -- start 2023-11-28T17:41:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.643+0000 7fba89735640 1 -- start start 2023-11-28T17:41:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.644+0000 7fba89735640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0047c0 0x7fba841f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.644+0000 7fba89735640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba841efd50 0x7fba840c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.644+0000 7fba89735640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba84150b50 0x7fba8414d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.644+0000 7fba89735640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba841ef060 con 0x7fba841efd50 2023-11-28T17:41:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.644+0000 7fba89735640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba841f34d0 con 0x7fba6c0047c0 2023-11-28T17:41:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.644+0000 7fba89735640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba84157b10 con 0x7fba84150b50 2023-11-28T17:41:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.645+0000 7fba837fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba841efd50 0x7fba840c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:45.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.645+0000 7fba837fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba841efd50 0x7fba840c45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36760/0 (socket says 172.21.15.98:36760) 2023-11-28T17:41:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.645+0000 7fba837fe640 1 -- 172.21.15.98:0/3197899551 learned_addr learned my addr 172.21.15.98:0/3197899551 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.645+0000 7fba88f34640 1 --2- 172.21.15.98:0/3197899551 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba84150b50 0x7fba8414d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.645+0000 7fba837fe640 1 -- 172.21.15.98:0/3197899551 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba84150b50 msgr2=0x7fba8414d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.645+0000 7fba83fff640 1 --2- 172.21.15.98:0/3197899551 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0047c0 0x7fba841f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.645+0000 7fba837fe640 1 --2- 172.21.15.98:0/3197899551 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba84150b50 0x7fba8414d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.645+0000 7fba837fe640 1 -- 172.21.15.98:0/3197899551 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0047c0 msgr2=0x7fba841f2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.645+0000 7fba837fe640 1 --2- 172.21.15.98:0/3197899551 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0047c0 0x7fba841f2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.645+0000 7fba837fe640 1 -- 172.21.15.98:0/3197899551 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba84157db0 con 0x7fba841efd50 2023-11-28T17:41:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.645+0000 7fba837fe640 1 --2- 172.21.15.98:0/3197899551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba841efd50 0x7fba840c45f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fba7802c250 tx=0x7fba78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.645+0000 7fba80ff9640 1 -- 172.21.15.98:0/3197899551 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78003810 con 0x7fba841efd50 2023-11-28T17:41:45.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.646+0000 7fba80ff9640 1 -- 172.21.15.98:0/3197899551 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba78003970 con 0x7fba841efd50 2023-11-28T17:41:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.646+0000 7fba80ff9640 1 -- 172.21.15.98:0/3197899551 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78003d00 con 0x7fba841efd50 2023-11-28T17:41:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.646+0000 7fba89735640 1 -- 172.21.15.98:0/3197899551 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba841ef270 con 0x7fba841efd50 2023-11-28T17:41:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.646+0000 7fba89735640 1 -- 172.21.15.98:0/3197899551 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba841ef830 con 0x7fba841efd50 2023-11-28T17:41:45.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.647+0000 7fba89735640 1 -- 172.21.15.98:0/3197899551 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba841ef9c0 con 0x7fba841efd50 2023-11-28T17:41:45.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.648+0000 7fba80ff9640 1 -- 172.21.15.98:0/3197899551 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fba78034440 con 0x7fba841efd50 2023-11-28T17:41:45.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.649+0000 7fba80ff9640 1 --2- 172.21.15.98:0/3197899551 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fba7006b2a0 0x7fba7006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:45.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.649+0000 7fba80ff9640 1 -- 172.21.15.98:0/3197899551 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba780abff0 con 0x7fba841efd50 2023-11-28T17:41:45.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.649+0000 7fba83fff640 1 --2- 172.21.15.98:0/3197899551 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fba7006b2a0 0x7fba7006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:45.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.650+0000 7fba83fff640 1 --2- 172.21.15.98:0/3197899551 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fba7006b2a0 0x7fba7006d760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fba840699e0 tx=0x7fba74009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:45.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.653+0000 7fba80ff9640 1 -- 172.21.15.98:0/3197899551 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba780dee90 con 0x7fba841efd50 2023-11-28T17:41:45.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.843+0000 7fba89735640 1 -- 172.21.15.98:0/3197899551 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fba841ef060 con 0x7fba841efd50 2023-11-28T17:41:45.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.844+0000 7fba80ff9640 1 -- 172.21.15.98:0/3197899551 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fba7804c0a0 con 0x7fba841efd50 2023-11-28T17:41:45.845 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-11-28T17:41:45.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.846+0000 7fba89735640 1 -- 172.21.15.98:0/3197899551 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fba7006b2a0 msgr2=0x7fba7006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:45.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.846+0000 7fba89735640 1 --2- 172.21.15.98:0/3197899551 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fba7006b2a0 0x7fba7006d760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fba840699e0 tx=0x7fba74009040 comp rx=0 tx=0).stop 2023-11-28T17:41:45.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.846+0000 7fba89735640 1 -- 172.21.15.98:0/3197899551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba841efd50 msgr2=0x7fba840c45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:45.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.846+0000 7fba89735640 1 --2- 172.21.15.98:0/3197899551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba841efd50 0x7fba840c45f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fba7802c250 tx=0x7fba78004730 comp rx=0 tx=0).stop 2023-11-28T17:41:45.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.847+0000 7fba89735640 1 -- 172.21.15.98:0/3197899551 shutdown_connections 2023-11-28T17:41:45.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.847+0000 7fba89735640 1 --2- 172.21.15.98:0/3197899551 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba84150b50 0x7fba8414d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.847+0000 7fba89735640 1 --2- 172.21.15.98:0/3197899551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba841efd50 0x7fba840c45f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.847+0000 7fba89735640 1 --2- 172.21.15.98:0/3197899551 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fba7006b2a0 0x7fba7006d760 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.847+0000 7fba89735640 1 --2- 172.21.15.98:0/3197899551 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba6c0047c0 0x7fba841f2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:45.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.847+0000 7fba89735640 1 -- 172.21.15.98:0/3197899551 >> 172.21.15.98:0/3197899551 conn(0x7fba840cf300 msgr2=0x7fba8406ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:45.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.847+0000 7fba89735640 1 -- 172.21.15.98:0/3197899551 shutdown_connections 2023-11-28T17:41:45.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:45.847+0000 7fba89735640 1 -- 172.21.15.98:0/3197899551 wait complete. 2023-11-28T17:41:45.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:41:45.880 INFO:tasks.ceph.mds.b.smithi113.stderr:2023-11-28T17:41:45.879+0000 7f6aad4d5640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-11-28T17:41:46.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.079+0000 7f58b335a640 1 Processor -- start 2023-11-28T17:41:46.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.080+0000 7f58b335a640 1 -- start start 2023-11-28T17:41:46.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.081+0000 7f58b335a640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58ac0caf60 0x7f58ac0cb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:46.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.081+0000 7f58b335a640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f58ac0c76d0 0x7f58ac1623d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:46.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.081+0000 7f58b335a640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ac151eb0 con 0x7f58ac06d170 2023-11-28T17:41:46.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.081+0000 7f58b335a640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ac0cb880 con 0x7f58ac0c76d0 2023-11-28T17:41:46.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.081+0000 7f58b335a640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac157d20 con 0x7f58ac0caf60 2023-11-28T17:41:46.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.082+0000 7f58b10f6640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58ac0caf60 0x7f58ac0cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:46.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.082+0000 7f58b10f6640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58ac0caf60 0x7f58ac0cb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:56576/0 (socket says 172.21.15.98:56576) 2023-11-28T17:41:46.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.082+0000 7f58b10f6640 1 -- 172.21.15.98:0/50719102 learned_addr learned my addr 172.21.15.98:0/50719102 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:46.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.082+0000 7f58b18f7640 1 --2- 172.21.15.98:0/50719102 >> v2:172.21.15.113:3300/0 conn(0x7f58ac0c76d0 0x7f58ac1623d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:46.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.082+0000 7f58b10f6640 1 -- 172.21.15.98:0/50719102 >> v2:172.21.15.113:3300/0 conn(0x7f58ac0c76d0 msgr2=0x7f58ac1623d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:46.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.082+0000 7f58b10f6640 1 --2- 172.21.15.98:0/50719102 >> v2:172.21.15.113:3300/0 conn(0x7f58ac0c76d0 0x7f58ac1623d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.082+0000 7f58b10f6640 1 -- 172.21.15.98:0/50719102 >> v1:172.21.15.113:6789/0 conn(0x7f58ac06d170 legacy=0x7f58ac06baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:46.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.082+0000 7f58b10f6640 1 -- 172.21.15.98:0/50719102 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac157a50 con 0x7f58ac0caf60 2023-11-28T17:41:46.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.084+0000 7f58b10f6640 1 --2- 172.21.15.98:0/50719102 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58ac0caf60 0x7f58ac0cb340 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f58a8009870 tx=0x7f58a802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df2c8f23d02441d1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:46.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.084+0000 7f58a3fff640 1 -- 172.21.15.98:0/50719102 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8030840 con 0x7f58ac0caf60 2023-11-28T17:41:46.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.084+0000 7f58a3fff640 1 -- 172.21.15.98:0/50719102 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a80309a0 con 0x7f58ac0caf60 2023-11-28T17:41:46.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.084+0000 7f58a3fff640 1 -- 172.21.15.98:0/50719102 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8030d30 con 0x7f58ac0caf60 2023-11-28T17:41:46.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.085+0000 7f58b335a640 1 -- 172.21.15.98:0/50719102 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58ac0caf60 msgr2=0x7f58ac0cb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:46.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.085+0000 7f58b335a640 1 --2- 172.21.15.98:0/50719102 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58ac0caf60 0x7f58ac0cb340 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f58a8009870 tx=0x7f58a802b980 comp rx=0 tx=0).stop 2023-11-28T17:41:46.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.085+0000 7f58b335a640 1 -- 172.21.15.98:0/50719102 shutdown_connections 2023-11-28T17:41:46.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.085+0000 7f58b335a640 1 --2- 172.21.15.98:0/50719102 >> v2:172.21.15.113:3300/0 conn(0x7f58ac0c76d0 0x7f58ac1623d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.085+0000 7f58b335a640 1 --2- 172.21.15.98:0/50719102 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58ac0caf60 0x7f58ac0cb340 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.085+0000 7f58b335a640 1 -- 172.21.15.98:0/50719102 >> 172.21.15.98:0/50719102 conn(0x7f58ac06a4d0 msgr2=0x7f58ac06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:46.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.085+0000 7f58b335a640 1 -- 172.21.15.98:0/50719102 shutdown_connections 2023-11-28T17:41:46.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.085+0000 7f58b335a640 1 -- 172.21.15.98:0/50719102 wait complete. 2023-11-28T17:41:46.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.087+0000 7f58b335a640 1 Processor -- start 2023-11-28T17:41:46.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.087+0000 7f58b335a640 1 -- start start 2023-11-28T17:41:46.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.088+0000 7f58b335a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58ac06d170 0x7f58ac145570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:46.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.088+0000 7f58b335a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58ac0c76d0 0x7f58ac140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:46.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.088+0000 7f58b335a640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58ac0caf60 0x7f58ac140dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:46.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.088+0000 7f58b335a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac1403f0 con 0x7f58ac06d170 2023-11-28T17:41:46.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.088+0000 7f58b335a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac0c2ec0 con 0x7f58ac0c76d0 2023-11-28T17:41:46.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.088+0000 7f58b335a640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac0c4610 con 0x7f58ac0caf60 2023-11-28T17:41:46.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.088+0000 7f58b10f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58ac06d170 0x7f58ac145570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.088+0000 7f58b10f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58ac06d170 0x7f58ac145570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36776/0 (socket says 172.21.15.98:36776) 2023-11-28T17:41:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.088+0000 7f58b10f6640 1 -- 172.21.15.98:0/568936786 learned_addr learned my addr 172.21.15.98:0/568936786 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.088+0000 7f58b08f5640 1 --2- 172.21.15.98:0/568936786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58ac0c76d0 0x7f58ac140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.088+0000 7f58b18f7640 1 --2- 172.21.15.98:0/568936786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58ac0caf60 0x7f58ac140dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.089+0000 7f58b10f6640 1 -- 172.21.15.98:0/568936786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58ac0caf60 msgr2=0x7f58ac140dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.089+0000 7f58b10f6640 1 --2- 172.21.15.98:0/568936786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58ac0caf60 0x7f58ac140dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.089+0000 7f58b10f6640 1 -- 172.21.15.98:0/568936786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58ac0c76d0 msgr2=0x7f58ac140880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.089+0000 7f58b10f6640 1 --2- 172.21.15.98:0/568936786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58ac0c76d0 0x7f58ac140880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.089+0000 7f58b10f6640 1 -- 172.21.15.98:0/568936786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a80094d0 con 0x7f58ac06d170 2023-11-28T17:41:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.089+0000 7f58b10f6640 1 --2- 172.21.15.98:0/568936786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58ac06d170 0x7f58ac145570 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f58a80090b0 tx=0x7f58a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.089+0000 7f58a1ffb640 1 -- 172.21.15.98:0/568936786 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a803e070 con 0x7f58ac06d170 2023-11-28T17:41:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.089+0000 7f58a1ffb640 1 -- 172.21.15.98:0/568936786 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a8039800 con 0x7f58ac06d170 2023-11-28T17:41:46.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.089+0000 7f58b335a640 1 -- 172.21.15.98:0/568936786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ac0c3160 con 0x7f58ac06d170 2023-11-28T17:41:46.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.089+0000 7f58a1ffb640 1 -- 172.21.15.98:0/568936786 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8039bb0 con 0x7f58ac06d170 2023-11-28T17:41:46.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.089+0000 7f58b335a640 1 -- 172.21.15.98:0/568936786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ac0c8f50 con 0x7f58ac06d170 2023-11-28T17:41:46.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.090+0000 7f58b335a640 1 -- 172.21.15.98:0/568936786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ac148810 con 0x7f58ac06d170 2023-11-28T17:41:46.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.091+0000 7f58a1ffb640 1 -- 172.21.15.98:0/568936786 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f58a8042070 con 0x7f58ac06d170 2023-11-28T17:41:46.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.092+0000 7f58a1ffb640 1 --2- 172.21.15.98:0/568936786 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f589006b2a0 0x7f589006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:46.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.093+0000 7f58a1ffb640 1 -- 172.21.15.98:0/568936786 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58a80ab9c0 con 0x7f58ac06d170 2023-11-28T17:41:46.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.093+0000 7f58b08f5640 1 --2- 172.21.15.98:0/568936786 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f589006b2a0 0x7f589006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:46.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.093+0000 7f58b08f5640 1 --2- 172.21.15.98:0/568936786 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f589006b2a0 0x7f589006d760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f58ac14f490 tx=0x7f589c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:46.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.097+0000 7f58a1ffb640 1 -- 172.21.15.98:0/568936786 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f58a8076700 con 0x7f58ac06d170 2023-11-28T17:41:46.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.289+0000 7f58b335a640 1 -- 172.21.15.98:0/568936786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f58ac13fee0 con 0x7f58ac06d170 2023-11-28T17:41:46.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.290+0000 7f58a1ffb640 1 -- 172.21.15.98:0/568936786 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f58ac13fee0 con 0x7f58ac06d170 2023-11-28T17:41:46.291 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-11-28T17:41:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.293+0000 7f58b335a640 1 -- 172.21.15.98:0/568936786 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f589006b2a0 msgr2=0x7f589006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.293+0000 7f58b335a640 1 --2- 172.21.15.98:0/568936786 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f589006b2a0 0x7f589006d760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f58ac14f490 tx=0x7f589c009040 comp rx=0 tx=0).stop 2023-11-28T17:41:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.293+0000 7f58b335a640 1 -- 172.21.15.98:0/568936786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58ac06d170 msgr2=0x7f58ac145570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.293+0000 7f58b335a640 1 --2- 172.21.15.98:0/568936786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58ac06d170 0x7f58ac145570 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f58a80090b0 tx=0x7f58a80046c0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.293+0000 7f58b335a640 1 -- 172.21.15.98:0/568936786 shutdown_connections 2023-11-28T17:41:46.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.293+0000 7f58b335a640 1 --2- 172.21.15.98:0/568936786 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f58ac0caf60 0x7f58ac140dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.293+0000 7f58b335a640 1 --2- 172.21.15.98:0/568936786 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f589006b2a0 0x7f589006d760 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.293+0000 7f58b335a640 1 --2- 172.21.15.98:0/568936786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58ac0c76d0 0x7f58ac140880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.293+0000 7f58b335a640 1 --2- 172.21.15.98:0/568936786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58ac06d170 0x7f58ac145570 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.293+0000 7f58b335a640 1 -- 172.21.15.98:0/568936786 >> 172.21.15.98:0/568936786 conn(0x7f58ac06a4d0 msgr2=0x7f58ac0ca7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.293+0000 7f58b335a640 1 -- 172.21.15.98:0/568936786 shutdown_connections 2023-11-28T17:41:46.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.293+0000 7f58b335a640 1 -- 172.21.15.98:0/568936786 wait complete. 2023-11-28T17:41:46.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:41:46.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.526+0000 7fdf130ad640 1 Processor -- start 2023-11-28T17:41:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.526+0000 7fdf130ad640 1 -- start start 2023-11-28T17:41:46.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.527+0000 7fdf130ad640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf0c154040 0x7fdf0c15dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.527+0000 7fdf130ad640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdf0c0c9420 0x7fdf0c15a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.527+0000 7fdf130ad640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0c159c50 con 0x7fdf0c0c9000 2023-11-28T17:41:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.527+0000 7fdf130ad640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf0c0c2ad0 con 0x7fdf0c0c9420 2023-11-28T17:41:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.527+0000 7fdf130ad640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c154420 con 0x7fdf0c154040 2023-11-28T17:41:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.528+0000 7fdf1164a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdf0c0c9420 0x7fdf0c15a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.528+0000 7fdf1164a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdf0c0c9420 0x7fdf0c15a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36792/0 (socket says 172.21.15.98:36792) 2023-11-28T17:41:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.528+0000 7fdf1164a640 1 -- 172.21.15.98:0/2556536090 learned_addr learned my addr 172.21.15.98:0/2556536090 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.528+0000 7fdf10e49640 1 --2- 172.21.15.98:0/2556536090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf0c154040 0x7fdf0c15dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:46.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.528+0000 7fdf1164a640 1 -- 172.21.15.98:0/2556536090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf0c154040 msgr2=0x7fdf0c15dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:46.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.528+0000 7fdf1164a640 1 --2- 172.21.15.98:0/2556536090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf0c154040 0x7fdf0c15dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.528+0000 7fdf1164a640 1 -- 172.21.15.98:0/2556536090 >> v1:172.21.15.113:6789/0 conn(0x7fdf0c0c9000 legacy=0x7fdf0c0c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:46.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.528+0000 7fdf1164a640 1 -- 172.21.15.98:0/2556536090 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c0cab90 con 0x7fdf0c0c9420 2023-11-28T17:41:46.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.530+0000 7fdf1164a640 1 --2- 172.21.15.98:0/2556536090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf0c0c9420 0x7fdf0c15a150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdf04009e50 tx=0x7fdf0402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a91f248a52b5e88e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:46.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.530+0000 7fdf0b7fe640 1 -- 172.21.15.98:0/2556536090 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0402cb90 con 0x7fdf0c0c9420 2023-11-28T17:41:46.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.530+0000 7fdf0b7fe640 1 -- 172.21.15.98:0/2556536090 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0402ccf0 con 0x7fdf0c0c9420 2023-11-28T17:41:46.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.530+0000 7fdf0b7fe640 1 -- 172.21.15.98:0/2556536090 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf04034610 con 0x7fdf0c0c9420 2023-11-28T17:41:46.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.530+0000 7fdf130ad640 1 -- 172.21.15.98:0/2556536090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf0c0c9420 msgr2=0x7fdf0c15a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:46.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.530+0000 7fdf130ad640 1 --2- 172.21.15.98:0/2556536090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf0c0c9420 0x7fdf0c15a150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdf04009e50 tx=0x7fdf0402bff0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.530+0000 7fdf130ad640 1 -- 172.21.15.98:0/2556536090 shutdown_connections 2023-11-28T17:41:46.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.530+0000 7fdf130ad640 1 --2- 172.21.15.98:0/2556536090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf0c0c9420 0x7fdf0c15a150 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.531+0000 7fdf130ad640 1 --2- 172.21.15.98:0/2556536090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf0c154040 0x7fdf0c15dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.531+0000 7fdf130ad640 1 -- 172.21.15.98:0/2556536090 >> 172.21.15.98:0/2556536090 conn(0x7fdf0c06a4d0 msgr2=0x7fdf0c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:46.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.531+0000 7fdf130ad640 1 -- 172.21.15.98:0/2556536090 shutdown_connections 2023-11-28T17:41:46.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.531+0000 7fdf130ad640 1 -- 172.21.15.98:0/2556536090 wait complete. 2023-11-28T17:41:46.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.532+0000 7fdf130ad640 1 Processor -- start 2023-11-28T17:41:46.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.532+0000 7fdf130ad640 1 -- start start 2023-11-28T17:41:46.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.533+0000 7fdf130ad640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf0c0c9000 0x7fdf0c1f02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:46.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.533+0000 7fdf130ad640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf0c0c9420 0x7fdf0c14bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:46.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.533+0000 7fdf130ad640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf0c154040 0x7fdf0c072050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:46.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.533+0000 7fdf130ad640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c157290 con 0x7fdf0c154040 2023-11-28T17:41:46.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.533+0000 7fdf130ad640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c1576c0 con 0x7fdf0c0c9420 2023-11-28T17:41:46.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.533+0000 7fdf130ad640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c071020 con 0x7fdf0c0c9000 2023-11-28T17:41:46.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.534+0000 7fdf1164a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf0c154040 0x7fdf0c072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:46.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.534+0000 7fdf1164a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf0c154040 0x7fdf0c072050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36808/0 (socket says 172.21.15.98:36808) 2023-11-28T17:41:46.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.534+0000 7fdf1164a640 1 -- 172.21.15.98:0/1403596680 learned_addr learned my addr 172.21.15.98:0/1403596680 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:46.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.534+0000 7fdf10e49640 1 --2- 172.21.15.98:0/1403596680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf0c0c9000 0x7fdf0c1f02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:46.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.534+0000 7fdf0bfff640 1 --2- 172.21.15.98:0/1403596680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf0c0c9420 0x7fdf0c14bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:46.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.534+0000 7fdf1164a640 1 -- 172.21.15.98:0/1403596680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf0c0c9000 msgr2=0x7fdf0c1f02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:46.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.534+0000 7fdf1164a640 1 --2- 172.21.15.98:0/1403596680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf0c0c9000 0x7fdf0c1f02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.534+0000 7fdf1164a640 1 -- 172.21.15.98:0/1403596680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf0c0c9420 msgr2=0x7fdf0c14bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:46.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.534+0000 7fdf1164a640 1 --2- 172.21.15.98:0/1403596680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf0c0c9420 0x7fdf0c14bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.534+0000 7fdf1164a640 1 -- 172.21.15.98:0/1403596680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf04009b30 con 0x7fdf0c154040 2023-11-28T17:41:46.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.535+0000 7fdf1164a640 1 --2- 172.21.15.98:0/1403596680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf0c154040 0x7fdf0c072050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdf04036040 tx=0x7fdf04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:46.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.535+0000 7fdf097fa640 1 -- 172.21.15.98:0/1403596680 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf04009510 con 0x7fdf0c154040 2023-11-28T17:41:46.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.535+0000 7fdf097fa640 1 -- 172.21.15.98:0/1403596680 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0403e450 con 0x7fdf0c154040 2023-11-28T17:41:46.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.535+0000 7fdf097fa640 1 -- 172.21.15.98:0/1403596680 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0403e800 con 0x7fdf0c154040 2023-11-28T17:41:46.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.535+0000 7fdf130ad640 1 -- 172.21.15.98:0/1403596680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0c1f07f0 con 0x7fdf0c154040 2023-11-28T17:41:46.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.535+0000 7fdf130ad640 1 -- 172.21.15.98:0/1403596680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0c1e6740 con 0x7fdf0c154040 2023-11-28T17:41:46.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.536+0000 7fdf130ad640 1 -- 172.21.15.98:0/1403596680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0c067bf0 con 0x7fdf0c154040 2023-11-28T17:41:46.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.537+0000 7fdf097fa640 1 -- 172.21.15.98:0/1403596680 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fdf0403e960 con 0x7fdf0c154040 2023-11-28T17:41:46.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.538+0000 7fdf097fa640 1 --2- 172.21.15.98:0/1403596680 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdef006b2a0 0x7fdef006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:46.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.538+0000 7fdf097fa640 1 -- 172.21.15.98:0/1403596680 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf040ac750 con 0x7fdf0c154040 2023-11-28T17:41:46.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.538+0000 7fdf10e49640 1 --2- 172.21.15.98:0/1403596680 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdef006b2a0 0x7fdef006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:46.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.539+0000 7fdf10e49640 1 --2- 172.21.15.98:0/1403596680 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdef006b2a0 0x7fdef006d760 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdf00006fd0 tx=0x7fdf00008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:46.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.543+0000 7fdf097fa640 1 -- 172.21.15.98:0/1403596680 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf04077490 con 0x7fdf0c154040 2023-11-28T17:41:46.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.771+0000 7fdf130ad640 1 -- 172.21.15.98:0/1403596680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf0c1f07f0 con 0x7fdf0c154040 2023-11-28T17:41:46.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.773+0000 7fdf097fa640 1 -- 172.21.15.98:0/1403596680 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6559 (secure 0 0 0) 0x7fdf0c1f07f0 con 0x7fdf0c154040 2023-11-28T17:41:46.774 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:46.774 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2429440814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2429440814},{"type":"v1","addr":"172.21.15.98:6836","nonce":2429440814}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:41:16.691765+0000","modified":"2023-11-28T17:41:44.341058+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4273,"mds_2":4272,"mds_3":4270,"mds_4":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"d","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.113:6835/1057450253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1057450253},{"type":"v1","addr":"172.21.15.113:6835","nonce":1057450253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"c","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.98:6837/1925178583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1925178583},{"type":"v1","addr":"172.21.15.98:6837","nonce":1925178583}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4272":{"gid":4272,"name":"f","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.113:6837/1732613907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":1732613907},{"type":"v1","addr":"172.21.15.113:6837","nonce":1732613907}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4273":{"gid":4273,"name":"e","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6839/3585869153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3585869153},{"type":"v1","addr":"172.21.15.98:6839","nonce":3585869153}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.113:6834/2183585971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2183585971},{"type":"v1","addr":"172.21.15.113:6834","nonce":2183585971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T17:41:46.774 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 16 2023-11-28T17:41:46.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.775+0000 7fdf130ad640 1 -- 172.21.15.98:0/1403596680 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdef006b2a0 msgr2=0x7fdef006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:46.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.775+0000 7fdf130ad640 1 --2- 172.21.15.98:0/1403596680 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdef006b2a0 0x7fdef006d760 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdf00006fd0 tx=0x7fdf00008040 comp rx=0 tx=0).stop 2023-11-28T17:41:46.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.775+0000 7fdf130ad640 1 -- 172.21.15.98:0/1403596680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf0c154040 msgr2=0x7fdf0c072050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:46.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.775+0000 7fdf130ad640 1 --2- 172.21.15.98:0/1403596680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf0c154040 0x7fdf0c072050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdf04036040 tx=0x7fdf04004a40 comp rx=0 tx=0).stop 2023-11-28T17:41:46.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.775+0000 7fdf130ad640 1 -- 172.21.15.98:0/1403596680 shutdown_connections 2023-11-28T17:41:46.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.775+0000 7fdf130ad640 1 --2- 172.21.15.98:0/1403596680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf0c154040 0x7fdf0c072050 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.775+0000 7fdf130ad640 1 --2- 172.21.15.98:0/1403596680 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdef006b2a0 0x7fdef006d760 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.775+0000 7fdf130ad640 1 --2- 172.21.15.98:0/1403596680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf0c0c9420 0x7fdf0c14bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.775+0000 7fdf130ad640 1 --2- 172.21.15.98:0/1403596680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdf0c0c9000 0x7fdf0c1f02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:46.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.775+0000 7fdf130ad640 1 -- 172.21.15.98:0/1403596680 >> 172.21.15.98:0/1403596680 conn(0x7fdf0c06a4d0 msgr2=0x7fdf0c15a150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:46.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.775+0000 7fdf130ad640 1 -- 172.21.15.98:0/1403596680 shutdown_connections 2023-11-28T17:41:46.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:46.775+0000 7fdf130ad640 1 -- 172.21.15.98:0/1403596680 wait complete. 2023-11-28T17:41:46.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:41:47.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.005+0000 7f60b3a15640 1 Processor -- start 2023-11-28T17:41:47.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.005+0000 7f60b3a15640 1 -- start start 2023-11-28T17:41:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.006+0000 7f60b3a15640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f60ac151d80 0x7f60ac157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.006+0000 7f60b3a15640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60ac15bc10 0x7f60ac0c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.006+0000 7f60b3a15640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60ac15b150 con 0x7f60ac0c6140 2023-11-28T17:41:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.006+0000 7f60b3a15640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60ac06e1f0 con 0x7f60ac151d80 2023-11-28T17:41:47.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.006+0000 7f60b3a15640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac06e4d0 con 0x7f60ac15bc10 2023-11-28T17:41:47.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.007+0000 7f60b17b1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f60ac151d80 0x7f60ac157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.007+0000 7f60b0fb0640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60ac15bc10 0x7f60ac0c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.007+0000 7f60b17b1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f60ac151d80 0x7f60ac157b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36818/0 (socket says 172.21.15.98:36818) 2023-11-28T17:41:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.007+0000 7f60b17b1640 1 -- 172.21.15.98:0/2338029251 learned_addr learned my addr 172.21.15.98:0/2338029251 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.007+0000 7f60b17b1640 1 -- 172.21.15.98:0/2338029251 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60ac15bc10 msgr2=0x7f60ac0c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.007+0000 7f60b17b1640 1 --2- 172.21.15.98:0/2338029251 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60ac15bc10 0x7f60ac0c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.007+0000 7f60b17b1640 1 -- 172.21.15.98:0/2338029251 >> v1:172.21.15.98:6789/0 conn(0x7f60ac0c6140 legacy=0x7f60ac0c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.008+0000 7f60b17b1640 1 -- 172.21.15.98:0/2338029251 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac06e7b0 con 0x7f60ac151d80 2023-11-28T17:41:47.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.009+0000 7f60b17b1640 1 --2- 172.21.15.98:0/2338029251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac151d80 0x7f60ac157b60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f60a8009870 tx=0x7f60a802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7133e2d048853f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.009+0000 7f60a3fff640 1 -- 172.21.15.98:0/2338029251 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a802f840 con 0x7f60ac151d80 2023-11-28T17:41:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.009+0000 7f60a3fff640 1 -- 172.21.15.98:0/2338029251 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a802f9a0 con 0x7f60ac151d80 2023-11-28T17:41:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.009+0000 7f60a3fff640 1 -- 172.21.15.98:0/2338029251 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a802fd30 con 0x7f60ac151d80 2023-11-28T17:41:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.009+0000 7f60b3a15640 1 -- 172.21.15.98:0/2338029251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac151d80 msgr2=0x7f60ac157b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.009+0000 7f60b3a15640 1 --2- 172.21.15.98:0/2338029251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac151d80 0x7f60ac157b60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f60a8009870 tx=0x7f60a802ba20 comp rx=0 tx=0).stop 2023-11-28T17:41:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.010+0000 7f60b3a15640 1 -- 172.21.15.98:0/2338029251 shutdown_connections 2023-11-28T17:41:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.010+0000 7f60b3a15640 1 --2- 172.21.15.98:0/2338029251 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60ac15bc10 0x7f60ac0c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.010+0000 7f60b3a15640 1 --2- 172.21.15.98:0/2338029251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac151d80 0x7f60ac157b60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.010+0000 7f60b3a15640 1 -- 172.21.15.98:0/2338029251 >> 172.21.15.98:0/2338029251 conn(0x7f60ac06a4d0 msgr2=0x7f60ac06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.010+0000 7f60b3a15640 1 -- 172.21.15.98:0/2338029251 shutdown_connections 2023-11-28T17:41:47.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.010+0000 7f60b3a15640 1 -- 172.21.15.98:0/2338029251 wait complete. 2023-11-28T17:41:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.011+0000 7f60b3a15640 1 Processor -- start 2023-11-28T17:41:47.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.011+0000 7f60b3a15640 1 -- start start 2023-11-28T17:41:47.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.012+0000 7f60b3a15640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60ac0c6140 0x7f60ac1e3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.012+0000 7f60b3a15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac151d80 0x7f60ac1557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.012+0000 7f60b3a15640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60ac15bc10 0x7f60ac139a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.012+0000 7f60b3a15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac156550 con 0x7f60ac151d80 2023-11-28T17:41:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.012+0000 7f60b3a15640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac1ed8f0 con 0x7f60ac15bc10 2023-11-28T17:41:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.012+0000 7f60b3a15640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac1edb90 con 0x7f60ac0c6140 2023-11-28T17:41:47.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.013+0000 7f60b0fb0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac151d80 0x7f60ac1557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.013+0000 7f60b0fb0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac151d80 0x7f60ac1557a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36822/0 (socket says 172.21.15.98:36822) 2023-11-28T17:41:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.013+0000 7f60b0fb0640 1 -- 172.21.15.98:0/1257076504 learned_addr learned my addr 172.21.15.98:0/1257076504 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.013+0000 7f60b0fb0640 1 -- 172.21.15.98:0/1257076504 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60ac0c6140 msgr2=0x7f60ac1e3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.013+0000 7f60b0fb0640 1 --2- 172.21.15.98:0/1257076504 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60ac0c6140 0x7f60ac1e3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.013+0000 7f60b0fb0640 1 -- 172.21.15.98:0/1257076504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60ac15bc10 msgr2=0x7f60ac139a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.013+0000 7f60b0fb0640 1 --2- 172.21.15.98:0/1257076504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60ac15bc10 0x7f60ac139a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.013+0000 7f60b0fb0640 1 -- 172.21.15.98:0/1257076504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a80094d0 con 0x7f60ac151d80 2023-11-28T17:41:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.013+0000 7f60b0fb0640 1 --2- 172.21.15.98:0/1257076504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac151d80 0x7f60ac1557a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f609c00e870 tx=0x7f609c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.013+0000 7f60a1ffb640 1 -- 172.21.15.98:0/1257076504 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c0094f0 con 0x7f60ac151d80 2023-11-28T17:41:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.013+0000 7f60a1ffb640 1 -- 172.21.15.98:0/1257076504 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c00c940 con 0x7f60ac151d80 2023-11-28T17:41:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.014+0000 7f60a1ffb640 1 -- 172.21.15.98:0/1257076504 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c00ccf0 con 0x7f60ac151d80 2023-11-28T17:41:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.014+0000 7f60b3a15640 1 -- 172.21.15.98:0/1257076504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ac1ede50 con 0x7f60ac151d80 2023-11-28T17:41:47.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.014+0000 7f60b3a15640 1 -- 172.21.15.98:0/1257076504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ac1ee410 con 0x7f60ac151d80 2023-11-28T17:41:47.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.015+0000 7f60b3a15640 1 -- 172.21.15.98:0/1257076504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60ac067bf0 con 0x7f60ac151d80 2023-11-28T17:41:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.016+0000 7f60a1ffb640 1 -- 172.21.15.98:0/1257076504 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f609c013470 con 0x7f60ac151d80 2023-11-28T17:41:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.017+0000 7f60a1ffb640 1 --2- 172.21.15.98:0/1257076504 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f609006b2a0 0x7f609006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.017+0000 7f60a1ffb640 1 -- 172.21.15.98:0/1257076504 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f609c089d10 con 0x7f60ac151d80 2023-11-28T17:41:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.017+0000 7f60b17b1640 1 --2- 172.21.15.98:0/1257076504 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f609006b2a0 0x7f609006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:47.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.017+0000 7f60b17b1640 1 --2- 172.21.15.98:0/1257076504 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f609006b2a0 0x7f609006d760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f60a8036000 tx=0x7f60a8039040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:47.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.021+0000 7f60a1ffb640 1 -- 172.21.15.98:0/1257076504 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f609c054af0 con 0x7f60ac151d80 2023-11-28T17:41:47.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.213+0000 7f60b3a15640 1 -- 172.21.15.98:0/1257076504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60ac06b2d0 con 0x7f60ac151d80 2023-11-28T17:41:47.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.215+0000 7f60a1ffb640 1 -- 172.21.15.98:0/1257076504 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16517 (secure 0 0 0) 0x7f609c021090 con 0x7f60ac151d80 2023-11-28T17:41:47.216 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:47.216 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:41:18.195933+0000","last_up_change":"2023-11-28T17:41:09.556993+0000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:41:10.303280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:41:14.890871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:41:15.704103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6810","nonce":2188704385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6813","nonce":2188704385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6819","nonce":2188704385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6816","nonce":2188704385}]},"public_addr":"172.21.15.98:6810/2188704385","cluster_addr":"172.21.15.98:6813/2188704385","heartbeat_back_addr":"172.21.15.98:6819/2188704385","heartbeat_front_addr":"172.21.15.98:6816/2188704385","state":["exists","up"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6801","nonce":3918226860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6803","nonce":3918226860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6807","nonce":3918226860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6805","nonce":3918226860}]},"public_addr":"172.21.15.98:6801/3918226860","cluster_addr":"172.21.15.98:6803/3918226860","heartbeat_back_addr":"172.21.15.98:6807/3918226860","heartbeat_front_addr":"172.21.15.98:6805/3918226860","state":["exists","up"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6812","nonce":2691532400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6815","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6818","nonce":2691532400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6823","nonce":2691532400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6821","nonce":2691532400}]},"public_addr":"172.21.15.98:6812/2691532400","cluster_addr":"172.21.15.98:6818/2691532400","heartbeat_back_addr":"172.21.15.98:6823/2691532400","heartbeat_front_addr":"172.21.15.98:6821/2691532400","state":["exists","up"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6825","nonce":1905487008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6827","nonce":1905487008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6831","nonce":1905487008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6829","nonce":1905487008}]},"public_addr":"172.21.15.98:6825/1905487008","cluster_addr":"172.21.15.98:6827/1905487008","heartbeat_back_addr":"172.21.15.98:6831/1905487008","heartbeat_front_addr":"172.21.15.98:6829/1905487008","state":["exists","up"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6801","nonce":2974677215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6803","nonce":2974677215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6807","nonce":2974677215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6805","nonce":2974677215}]},"public_addr":"172.21.15.113:6801/2974677215","cluster_addr":"172.21.15.113:6803/2974677215","heartbeat_back_addr":"172.21.15.113:6807/2974677215","heartbeat_front_addr":"172.21.15.113:6805/2974677215","state":["exists","up"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6817","nonce":2685000355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6821","nonce":2685000355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6829","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6831","nonce":2685000355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6823","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6826","nonce":2685000355}]},"public_addr":"172.21.15.113:6817/2685000355","cluster_addr":"172.21.15.113:6821/2685000355","heartbeat_back_addr":"172.21.15.113:6831/2685000355","heartbeat_front_addr":"172.21.15.113:6826/2685000355","state":["exists","up"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6809","nonce":2764300831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6811","nonce":2764300831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6815","nonce":2764300831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6813","nonce":2764300831}]},"public_addr":"172.21.15.113:6809/2764300831","cluster_addr":"172.21.15.113:6811/2764300831","heartbeat_back_addr":"172.21.15.113:6815/2764300831","heartbeat_front_addr":"172.21.15.113:6813/2764300831","state":["exists","up"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6819","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6820","nonce":1656248400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6824","nonce":1656248400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6830","nonce":1656248400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6825","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6827","nonce":1656248400}]},"public_addr":"172.21.15.113:6820/1656248400","cluster_addr":"172.21.15.113:6824/1656248400","heartbeat_back_addr":"172.21.15.113:6830/1656248400","heartbeat_front_addr":"172.21.15.113:6827/1656248400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.379549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.312196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.117494+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.388128+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.360608+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.411101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.355995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.390362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:41:47.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.217+0000 7f60b3a15640 1 -- 172.21.15.98:0/1257076504 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f609006b2a0 msgr2=0x7f609006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.217+0000 7f60b3a15640 1 --2- 172.21.15.98:0/1257076504 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f609006b2a0 0x7f609006d760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f60a8036000 tx=0x7f60a8039040 comp rx=0 tx=0).stop 2023-11-28T17:41:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.217+0000 7f60b3a15640 1 -- 172.21.15.98:0/1257076504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac151d80 msgr2=0x7f60ac1557a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.217+0000 7f60b3a15640 1 --2- 172.21.15.98:0/1257076504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac151d80 0x7f60ac1557a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f609c00e870 tx=0x7f609c00ed30 comp rx=0 tx=0).stop 2023-11-28T17:41:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.217+0000 7f60b3a15640 1 -- 172.21.15.98:0/1257076504 shutdown_connections 2023-11-28T17:41:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.217+0000 7f60b3a15640 1 --2- 172.21.15.98:0/1257076504 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f609006b2a0 0x7f609006d760 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.217+0000 7f60b3a15640 1 --2- 172.21.15.98:0/1257076504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60ac15bc10 0x7f60ac139a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.217+0000 7f60b3a15640 1 --2- 172.21.15.98:0/1257076504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60ac151d80 0x7f60ac1557a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.217+0000 7f60b3a15640 1 --2- 172.21.15.98:0/1257076504 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f60ac0c6140 0x7f60ac1e3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.217+0000 7f60b3a15640 1 -- 172.21.15.98:0/1257076504 >> 172.21.15.98:0/1257076504 conn(0x7f60ac06a4d0 msgr2=0x7f60ac14dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.218+0000 7f60b3a15640 1 -- 172.21.15.98:0/1257076504 shutdown_connections 2023-11-28T17:41:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.218+0000 7f60b3a15640 1 -- 172.21.15.98:0/1257076504 wait complete. 2023-11-28T17:41:47.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:41:47.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.445+0000 7f29d80c2640 1 Processor -- start 2023-11-28T17:41:47.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.445+0000 7f29d80c2640 1 -- start start 2023-11-28T17:41:47.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d80c2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29d0150c00 0x7f29d00c2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:47.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d80c2640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f29d00c5c00 0x7f29d00c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:47.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d80c2640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29d00c4a20 0x7f29d00c6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:47.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d80c2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d0157850 con 0x7f29d0150c00 2023-11-28T17:41:47.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d80c2640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d0156980 con 0x7f29d00c5c00 2023-11-28T17:41:47.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d80c2640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d0156c40 con 0x7f29d00c4a20 2023-11-28T17:41:47.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d5e5e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29d0150c00 0x7f29d00c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:47.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d5e5e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29d0150c00 0x7f29d00c2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36838/0 (socket says 172.21.15.98:36838) 2023-11-28T17:41:47.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d5e5e640 1 -- 172.21.15.98:0/4247389277 learned_addr learned my addr 172.21.15.98:0/4247389277 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:47.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d5e5e640 1 -- 172.21.15.98:0/4247389277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29d00c4a20 msgr2=0x7f29d00c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:47.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d665f640 1 --2- 172.21.15.98:0/4247389277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29d00c4a20 0x7f29d00c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:47.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d565d640 1 --2- 172.21.15.98:0/4247389277 >> v2:172.21.15.113:3300/0 conn(0x7f29d00c5c00 0x7f29d00c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:47.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d5e5e640 1 --2- 172.21.15.98:0/4247389277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29d00c4a20 0x7f29d00c6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d5e5e640 1 -- 172.21.15.98:0/4247389277 >> v2:172.21.15.113:3300/0 conn(0x7f29d00c5c00 msgr2=0x7f29d00c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:47.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d5e5e640 1 --2- 172.21.15.98:0/4247389277 >> v2:172.21.15.113:3300/0 conn(0x7f29d00c5c00 0x7f29d00c44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d5e5e640 1 -- 172.21.15.98:0/4247389277 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d0156f20 con 0x7f29d0150c00 2023-11-28T17:41:47.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d665f640 1 --2- 172.21.15.98:0/4247389277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29d00c4a20 0x7f29d00c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:47.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.447+0000 7f29d565d640 1 --2- 172.21.15.98:0/4247389277 >> v2:172.21.15.113:3300/0 conn(0x7f29d00c5c00 0x7f29d00c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:47.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.449+0000 7f29d5e5e640 1 --2- 172.21.15.98:0/4247389277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29d0150c00 0x7f29d00c2700 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f29d0070410 tx=0x7f29cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3706ea1971cb739d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:47.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.449+0000 7f29d4e5c640 1 -- 172.21.15.98:0/4247389277 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc031870 con 0x7f29d0150c00 2023-11-28T17:41:47.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.449+0000 7f29d4e5c640 1 -- 172.21.15.98:0/4247389277 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29cc0319d0 con 0x7f29d0150c00 2023-11-28T17:41:47.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.449+0000 7f29d4e5c640 1 -- 172.21.15.98:0/4247389277 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc031d60 con 0x7f29d0150c00 2023-11-28T17:41:47.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.450+0000 7f29d80c2640 1 -- 172.21.15.98:0/4247389277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29d0150c00 msgr2=0x7f29d00c2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:47.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.450+0000 7f29d80c2640 1 --2- 172.21.15.98:0/4247389277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29d0150c00 0x7f29d00c2700 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f29d0070410 tx=0x7f29cc02b910 comp rx=0 tx=0).stop 2023-11-28T17:41:47.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.450+0000 7f29d80c2640 1 -- 172.21.15.98:0/4247389277 shutdown_connections 2023-11-28T17:41:47.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.450+0000 7f29d80c2640 1 --2- 172.21.15.98:0/4247389277 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29d00c4a20 0x7f29d00c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.450+0000 7f29d80c2640 1 --2- 172.21.15.98:0/4247389277 >> v2:172.21.15.113:3300/0 conn(0x7f29d00c5c00 0x7f29d00c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.450+0000 7f29d80c2640 1 --2- 172.21.15.98:0/4247389277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29d0150c00 0x7f29d00c2700 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.450+0000 7f29d80c2640 1 -- 172.21.15.98:0/4247389277 >> 172.21.15.98:0/4247389277 conn(0x7f29d006a4d0 msgr2=0x7f29d006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.450+0000 7f29d80c2640 1 -- 172.21.15.98:0/4247389277 shutdown_connections 2023-11-28T17:41:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.450+0000 7f29d80c2640 1 -- 172.21.15.98:0/4247389277 wait complete. 2023-11-28T17:41:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.451+0000 7f29d80c2640 1 Processor -- start 2023-11-28T17:41:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.452+0000 7f29d80c2640 1 -- start start 2023-11-28T17:41:47.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.452+0000 7f29d80c2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29d00c4a20 0x7f29d0147ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:47.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.452+0000 7f29d80c2640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29d00c5c00 0x7f29d01edd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:47.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.453+0000 7f29d80c2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29d0150c00 0x7f29d013da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:47.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.453+0000 7f29d80c2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d0140420 con 0x7f29d0150c00 2023-11-28T17:41:47.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.453+0000 7f29d80c2640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d01487a0 con 0x7f29d00c4a20 2023-11-28T17:41:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.453+0000 7f29d80c2640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d01ed590 con 0x7f29d00c5c00 2023-11-28T17:41:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.453+0000 7f29d665f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29d0150c00 0x7f29d013da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.453+0000 7f29d665f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29d0150c00 0x7f29d013da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36854/0 (socket says 172.21.15.98:36854) 2023-11-28T17:41:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.453+0000 7f29d665f640 1 -- 172.21.15.98:0/1768787127 learned_addr learned my addr 172.21.15.98:0/1768787127 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.453+0000 7f29d565d640 1 --2- 172.21.15.98:0/1768787127 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29d00c5c00 0x7f29d01edd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.453+0000 7f29d5e5e640 1 --2- 172.21.15.98:0/1768787127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29d00c4a20 0x7f29d0147ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.453+0000 7f29d665f640 1 -- 172.21.15.98:0/1768787127 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29d00c5c00 msgr2=0x7f29d01edd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.453+0000 7f29d665f640 1 --2- 172.21.15.98:0/1768787127 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29d00c5c00 0x7f29d01edd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.453+0000 7f29d665f640 1 -- 172.21.15.98:0/1768787127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29d00c4a20 msgr2=0x7f29d0147ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.453+0000 7f29d665f640 1 --2- 172.21.15.98:0/1768787127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29d00c4a20 0x7f29d0147ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.453+0000 7f29d665f640 1 -- 172.21.15.98:0/1768787127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29cc0094d0 con 0x7f29d0150c00 2023-11-28T17:41:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.454+0000 7f29d665f640 1 --2- 172.21.15.98:0/1768787127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29d0150c00 0x7f29d013da00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f29c800cd40 tx=0x7f29c800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.454+0000 7f29c67fc640 1 -- 172.21.15.98:0/1768787127 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c800f680 con 0x7f29d0150c00 2023-11-28T17:41:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.454+0000 7f29c67fc640 1 -- 172.21.15.98:0/1768787127 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c800f800 con 0x7f29d0150c00 2023-11-28T17:41:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.454+0000 7f29d80c2640 1 -- 172.21.15.98:0/1768787127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d0140160 con 0x7f29d0150c00 2023-11-28T17:41:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.454+0000 7f29c67fc640 1 -- 172.21.15.98:0/1768787127 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c800fbb0 con 0x7f29d0150c00 2023-11-28T17:41:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.454+0000 7f29d80c2640 1 -- 172.21.15.98:0/1768787127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d01ed830 con 0x7f29d0150c00 2023-11-28T17:41:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.455+0000 7f29d80c2640 1 -- 172.21.15.98:0/1768787127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d0067bf0 con 0x7f29d0150c00 2023-11-28T17:41:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.456+0000 7f29c67fc640 1 -- 172.21.15.98:0/1768787127 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f29c800b040 con 0x7f29d0150c00 2023-11-28T17:41:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.457+0000 7f29c67fc640 1 --2- 172.21.15.98:0/1768787127 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f29b406b2d0 0x7f29b406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.457+0000 7f29c67fc640 1 -- 172.21.15.98:0/1768787127 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29c808a5c0 con 0x7f29d0150c00 2023-11-28T17:41:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.457+0000 7f29d5e5e640 1 --2- 172.21.15.98:0/1768787127 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f29b406b2d0 0x7f29b406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.458+0000 7f29d5e5e640 1 --2- 172.21.15.98:0/1768787127 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f29b406b2d0 0x7f29b406d790 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f29cc036000 tx=0x7f29cc03c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.462+0000 7f29c67fc640 1 -- 172.21.15.98:0/1768787127 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29c80553a0 con 0x7f29d0150c00 2023-11-28T17:41:47.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.687+0000 7f29d80c2640 1 -- 172.21.15.98:0/1768787127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29d01eda20 con 0x7f29d0150c00 2023-11-28T17:41:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.688+0000 7f29c67fc640 1 -- 172.21.15.98:0/1768787127 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6559 (secure 0 0 0) 0x7f29c801d090 con 0x7f29d0150c00 2023-11-28T17:41:47.688 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:47.689 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2429440814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2429440814},{"type":"v1","addr":"172.21.15.98:6836","nonce":2429440814}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:41:16.691765+0000","modified":"2023-11-28T17:41:44.341058+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4273,"mds_2":4272,"mds_3":4270,"mds_4":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"d","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.113:6835/1057450253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1057450253},{"type":"v1","addr":"172.21.15.113:6835","nonce":1057450253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"c","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.98:6837/1925178583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1925178583},{"type":"v1","addr":"172.21.15.98:6837","nonce":1925178583}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4272":{"gid":4272,"name":"f","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.113:6837/1732613907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":1732613907},{"type":"v1","addr":"172.21.15.113:6837","nonce":1732613907}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4273":{"gid":4273,"name":"e","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6839/3585869153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3585869153},{"type":"v1","addr":"172.21.15.98:6839","nonce":3585869153}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.113:6834/2183585971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2183585971},{"type":"v1","addr":"172.21.15.113:6834","nonce":2183585971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T17:41:47.689 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 16 2023-11-28T17:41:47.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.690+0000 7f29d80c2640 1 -- 172.21.15.98:0/1768787127 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f29b406b2d0 msgr2=0x7f29b406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:47.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.690+0000 7f29d80c2640 1 --2- 172.21.15.98:0/1768787127 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f29b406b2d0 0x7f29b406d790 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f29cc036000 tx=0x7f29cc03c040 comp rx=0 tx=0).stop 2023-11-28T17:41:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.690+0000 7f29d80c2640 1 -- 172.21.15.98:0/1768787127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29d0150c00 msgr2=0x7f29d013da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.690+0000 7f29d80c2640 1 --2- 172.21.15.98:0/1768787127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29d0150c00 0x7f29d013da00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f29c800cd40 tx=0x7f29c800a650 comp rx=0 tx=0).stop 2023-11-28T17:41:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.690+0000 7f29d80c2640 1 -- 172.21.15.98:0/1768787127 shutdown_connections 2023-11-28T17:41:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.690+0000 7f29d80c2640 1 --2- 172.21.15.98:0/1768787127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29d0150c00 0x7f29d013da00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.690+0000 7f29d80c2640 1 --2- 172.21.15.98:0/1768787127 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29d00c5c00 0x7f29d01edd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.690+0000 7f29d80c2640 1 --2- 172.21.15.98:0/1768787127 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f29b406b2d0 0x7f29b406d790 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.690+0000 7f29d80c2640 1 --2- 172.21.15.98:0/1768787127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29d00c4a20 0x7f29d0147ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.690+0000 7f29d80c2640 1 -- 172.21.15.98:0/1768787127 >> 172.21.15.98:0/1768787127 conn(0x7f29d006a4d0 msgr2=0x7f29d00c3790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.690+0000 7f29d80c2640 1 -- 172.21.15.98:0/1768787127 shutdown_connections 2023-11-28T17:41:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:47.690+0000 7f29d80c2640 1 -- 172.21.15.98:0/1768787127 wait complete. 2023-11-28T17:41:48.315 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T17:41:16.691765+0000', 'modified': '2023-11-28T17:41:44.341058+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4291, 'mds_1': 4273, 'mds_2': 4272, 'mds_3': 4270, 'mds_4': 4269}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4269': {'gid': 4269, 'name': 'd', 'rank': 4, 'incarnation': 15, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.113:6835/1057450253', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6833', 'nonce': 1057450253}, {'type': 'v1', 'addr': '172.21.15.113:6835', 'nonce': 1057450253}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4270': {'gid': 4270, 'name': 'c', 'rank': 3, 'incarnation': 13, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.98:6837/1925178583', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 1925178583}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 1925178583}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4272': {'gid': 4272, 'name': 'f', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.113:6837/1732613907', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6836', 'nonce': 1732613907}, {'type': 'v1', 'addr': '172.21.15.113:6837', 'nonce': 1732613907}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4273': {'gid': 4273, 'name': 'e', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.98:6839/3585869153', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 3585869153}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 3585869153}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4291': {'gid': 4291, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.113:6834/2183585971', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6832', 'nonce': 2183585971}, {'type': 'v1', 'addr': '172.21.15.113:6834', 'nonce': 2183585971}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T17:41:48.315 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2023-11-28T17:41:48.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-11-28T17:41:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.533+0000 7fb2863ec640 1 Processor -- start 2023-11-28T17:41:48.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.533+0000 7fb2863ec640 1 -- start start 2023-11-28T17:41:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.535+0000 7fb2863ec640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb280154030 0x7fb280070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.535+0000 7fb2863ec640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb28006dcf0 0x7fb28006e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.535+0000 7fb2863ec640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb280154410 con 0x7fb2800c8600 2023-11-28T17:41:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.535+0000 7fb2863ec640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2800715a0 con 0x7fb28006dcf0 2023-11-28T17:41:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.535+0000 7fb2863ec640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2800c5da0 con 0x7fb280154030 2023-11-28T17:41:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.535+0000 7fb277fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb2800c8600 0x7fb280070e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55568/0 (socket says 172.21.15.98:55568) 2023-11-28T17:41:48.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.536+0000 7fb277fff640 1 -- 172.21.15.98:0/4052608047 learned_addr learned my addr 172.21.15.98:0/4052608047 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.536+0000 7fb284989640 1 --2- 172.21.15.98:0/4052608047 >> v2:172.21.15.98:3300/0 conn(0x7fb28006dcf0 0x7fb28006e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.536+0000 7fb27ffff640 1 --2- 172.21.15.98:0/4052608047 >> v2:172.21.15.113:3300/0 conn(0x7fb280154030 0x7fb280070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.536+0000 7fb284989640 1 -- 172.21.15.98:0/4052608047 >> v2:172.21.15.113:3300/0 conn(0x7fb280154030 msgr2=0x7fb280070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.536+0000 7fb284989640 1 --2- 172.21.15.98:0/4052608047 >> v2:172.21.15.113:3300/0 conn(0x7fb280154030 0x7fb280070660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:48.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.536+0000 7fb284989640 1 -- 172.21.15.98:0/4052608047 >> v1:172.21.15.98:6789/0 conn(0x7fb2800c8600 legacy=0x7fb280070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.536+0000 7fb284989640 1 -- 172.21.15.98:0/4052608047 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28015d370 con 0x7fb28006dcf0 2023-11-28T17:41:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.537+0000 7fb284989640 1 --2- 172.21.15.98:0/4052608047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb28006dcf0 0x7fb28006e120 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb28006eba0 tx=0x7fb27802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1d661775bde8cf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.537+0000 7fb27f7fe640 1 -- 172.21.15.98:0/4052608047 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27802cd10 con 0x7fb28006dcf0 2023-11-28T17:41:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.538+0000 7fb27f7fe640 1 -- 172.21.15.98:0/4052608047 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27802ce70 con 0x7fb28006dcf0 2023-11-28T17:41:48.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.538+0000 7fb27f7fe640 1 -- 172.21.15.98:0/4052608047 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb278034640 con 0x7fb28006dcf0 2023-11-28T17:41:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.538+0000 7fb2863ec640 1 -- 172.21.15.98:0/4052608047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb28006dcf0 msgr2=0x7fb28006e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.538+0000 7fb2863ec640 1 --2- 172.21.15.98:0/4052608047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb28006dcf0 0x7fb28006e120 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb28006eba0 tx=0x7fb27802bf60 comp rx=0 tx=0).stop 2023-11-28T17:41:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.538+0000 7fb2863ec640 1 -- 172.21.15.98:0/4052608047 shutdown_connections 2023-11-28T17:41:48.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.538+0000 7fb2863ec640 1 --2- 172.21.15.98:0/4052608047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb28006dcf0 0x7fb28006e120 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.538+0000 7fb2863ec640 1 --2- 172.21.15.98:0/4052608047 >> v2:172.21.15.113:3300/0 conn(0x7fb280154030 0x7fb280070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.538+0000 7fb2863ec640 1 -- 172.21.15.98:0/4052608047 >> 172.21.15.98:0/4052608047 conn(0x7fb28006c0f0 msgr2=0x7fb28006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.538+0000 7fb2863ec640 1 -- 172.21.15.98:0/4052608047 shutdown_connections 2023-11-28T17:41:48.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.538+0000 7fb2863ec640 1 -- 172.21.15.98:0/4052608047 wait complete. 2023-11-28T17:41:48.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.540+0000 7fb2863ec640 1 Processor -- start 2023-11-28T17:41:48.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.540+0000 7fb2863ec640 1 -- start start 2023-11-28T17:41:48.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.541+0000 7fb2863ec640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb28006dcf0 0x7fb280152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:48.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.541+0000 7fb2863ec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2800c8600 0x7fb28014e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:48.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.541+0000 7fb2863ec640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb280154030 0x7fb280158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:48.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.541+0000 7fb2863ec640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2801482c0 con 0x7fb2800c8600 2023-11-28T17:41:48.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.541+0000 7fb2863ec640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb280072040 con 0x7fb28006dcf0 2023-11-28T17:41:48.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.541+0000 7fb2863ec640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb280148b20 con 0x7fb280154030 2023-11-28T17:41:48.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.541+0000 7fb277fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2800c8600 0x7fb28014e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.541+0000 7fb277fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2800c8600 0x7fb28014e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36882/0 (socket says 172.21.15.98:36882) 2023-11-28T17:41:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.541+0000 7fb277fff640 1 -- 172.21.15.98:0/2134457464 learned_addr learned my addr 172.21.15.98:0/2134457464 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.542+0000 7fb284989640 1 --2- 172.21.15.98:0/2134457464 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb280154030 0x7fb280158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.542+0000 7fb27ffff640 1 --2- 172.21.15.98:0/2134457464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb28006dcf0 0x7fb280152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.542+0000 7fb277fff640 1 -- 172.21.15.98:0/2134457464 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb280154030 msgr2=0x7fb280158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.542+0000 7fb277fff640 1 --2- 172.21.15.98:0/2134457464 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb280154030 0x7fb280158b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.542+0000 7fb277fff640 1 -- 172.21.15.98:0/2134457464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb28006dcf0 msgr2=0x7fb280152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.542+0000 7fb277fff640 1 --2- 172.21.15.98:0/2134457464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb28006dcf0 0x7fb280152260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.542+0000 7fb277fff640 1 -- 172.21.15.98:0/2134457464 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb278009a70 con 0x7fb2800c8600 2023-11-28T17:41:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.542+0000 7fb277fff640 1 --2- 172.21.15.98:0/2134457464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2800c8600 0x7fb28014e9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb26800c480 tx=0x7fb26800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.542+0000 7fb27d7fa640 1 -- 172.21.15.98:0/2134457464 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb268005ce0 con 0x7fb2800c8600 2023-11-28T17:41:48.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.542+0000 7fb27d7fa640 1 -- 172.21.15.98:0/2134457464 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb268005e40 con 0x7fb2800c8600 2023-11-28T17:41:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.542+0000 7fb27d7fa640 1 -- 172.21.15.98:0/2134457464 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2680072b0 con 0x7fb2800c8600 2023-11-28T17:41:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.542+0000 7fb2863ec640 1 -- 172.21.15.98:0/2134457464 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb280149d50 con 0x7fb2800c8600 2023-11-28T17:41:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.542+0000 7fb2863ec640 1 -- 172.21.15.98:0/2134457464 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2801484d0 con 0x7fb2800c8600 2023-11-28T17:41:48.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.544+0000 7fb27d7fa640 1 -- 172.21.15.98:0/2134457464 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb268007410 con 0x7fb2800c8600 2023-11-28T17:41:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.545+0000 7fb27d7fa640 1 --2- 172.21.15.98:0/2134457464 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb26006b2a0 0x7fb26006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.545+0000 7fb27d7fa640 1 -- 172.21.15.98:0/2134457464 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb268089190 con 0x7fb2800c8600 2023-11-28T17:41:48.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.546+0000 7fb27ffff640 1 --2- 172.21.15.98:0/2134457464 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb26006b2a0 0x7fb26006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.546+0000 7fb288f85740 1 Processor -- start 2023-11-28T17:41:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.546+0000 7fb288f85740 1 -- start start 2023-11-28T17:41:48.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.546+0000 7fb27ffff640 1 --2- 172.21.15.98:0/2134457464 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb26006b2a0 0x7fb26006d760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb270009a10 tx=0x7fb270008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.547+0000 7fb288f85740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb145890 0x563beb145c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.547+0000 7fb288f85740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb18f710 0x563beb1461b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.547+0000 7fb288f85740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb1442f0 0x563beb185210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.547+0000 7fb288f85740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x563beb0a4880 con 0x563beb145890 2023-11-28T17:41:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.547+0000 7fb288f85740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x563beb0b3ae0 con 0x563beb1442f0 2023-11-28T17:41:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.547+0000 7fb288f85740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x563beb0b11d0 con 0x563beb18f710 2023-11-28T17:41:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.547+0000 7fb27ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb145890 0x563beb145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.547+0000 7fb27ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb145890 0x563beb145c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36890/0 (socket says 172.21.15.98:36890) 2023-11-28T17:41:48.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.547+0000 7fb27ffff640 1 -- 172.21.15.98:0/3714353560 learned_addr learned my addr 172.21.15.98:0/3714353560 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.548+0000 7fb27ffff640 1 -- 172.21.15.98:0/3714353560 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb18f710 msgr2=0x563beb1461b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.548+0000 7fb27ffff640 1 --2- 172.21.15.98:0/3714353560 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb18f710 0x563beb1461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.548+0000 7fb27ffff640 1 -- 172.21.15.98:0/3714353560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb1442f0 msgr2=0x563beb185210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.548+0000 7fb27ffff640 1 --2- 172.21.15.98:0/3714353560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb1442f0 0x563beb185210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.548+0000 7fb27ffff640 1 -- 172.21.15.98:0/3714353560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563beb074860 con 0x563beb145890 2023-11-28T17:41:48.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.548+0000 7fb27ffff640 1 --2- 172.21.15.98:0/3714353560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb145890 0x563beb145c70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb270010b80 tx=0x7fb270013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b648f40c05fb77a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.548+0000 7fb275ffb640 1 -- 172.21.15.98:0/3714353560 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb270021440 con 0x563beb145890 2023-11-28T17:41:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.548+0000 7fb275ffb640 1 -- 172.21.15.98:0/3714353560 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27001a070 con 0x563beb145890 2023-11-28T17:41:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.548+0000 7fb275ffb640 1 -- 172.21.15.98:0/3714353560 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2700217a0 con 0x563beb145890 2023-11-28T17:41:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.549+0000 7fb288f85740 1 -- 172.21.15.98:0/3714353560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb145890 msgr2=0x563beb145c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.549+0000 7fb288f85740 1 --2- 172.21.15.98:0/3714353560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb145890 0x563beb145c70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb270010b80 tx=0x7fb270013000 comp rx=0 tx=0).stop 2023-11-28T17:41:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.549+0000 7fb288f85740 1 -- 172.21.15.98:0/3714353560 shutdown_connections 2023-11-28T17:41:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.549+0000 7fb288f85740 1 --2- 172.21.15.98:0/3714353560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb1442f0 0x563beb185210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.549+0000 7fb288f85740 1 --2- 172.21.15.98:0/3714353560 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb18f710 0x563beb1461b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.549+0000 7fb288f85740 1 --2- 172.21.15.98:0/3714353560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb145890 0x563beb145c70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.549+0000 7fb288f85740 1 -- 172.21.15.98:0/3714353560 >> 172.21.15.98:0/3714353560 conn(0x563beb142290 msgr2=0x563beb144940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.549+0000 7fb288f85740 1 -- 172.21.15.98:0/3714353560 shutdown_connections 2023-11-28T17:41:48.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.549+0000 7fb288f85740 1 -- 172.21.15.98:0/3714353560 wait complete. 2023-11-28T17:41:48.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.550+0000 7fb288f85740 1 -- start start 2023-11-28T17:41:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.550+0000 7fb288f85740 1 Processor -- start 2023-11-28T17:41:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.551+0000 7fb288f85740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb145890 0x563beb1838c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.551+0000 7fb288f85740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb18f710 0x563beb1817b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.551+0000 7fb288f85740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb186660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.551+0000 7fb288f85740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x563beb0acac0 con 0x563beb196030 2023-11-28T17:41:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.551+0000 7fb288f85740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x563beb0492f0 con 0x563beb145890 2023-11-28T17:41:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.551+0000 7fb288f85740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x563beb066c70 con 0x563beb18f710 2023-11-28T17:41:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.551+0000 7fb288f85740 10 client.0 _ll_register_callbacks cb 0x563beadbeac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:41:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.551+0000 7fb284989640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb186660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:48.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.551+0000 7fb284989640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb186660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36894/0 (socket says 172.21.15.98:36894) 2023-11-28T17:41:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.551+0000 7fb284989640 1 -- 172.21.15.98:0/2396567817 learned_addr learned my addr 172.21.15.98:0/2396567817 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.552+0000 7fb277fff640 1 --2- 172.21.15.98:0/2396567817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb18f710 0x563beb1817b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.552+0000 7fb284989640 1 -- 172.21.15.98:0/2396567817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb18f710 msgr2=0x563beb1817b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.552+0000 7fb27ffff640 1 --2- 172.21.15.98:0/2396567817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb145890 0x563beb1838c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.552+0000 7fb284989640 1 --2- 172.21.15.98:0/2396567817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb18f710 0x563beb1817b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.552+0000 7fb284989640 1 -- 172.21.15.98:0/2396567817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb145890 msgr2=0x563beb1838c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.552+0000 7fb284989640 1 --2- 172.21.15.98:0/2396567817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb145890 0x563beb1838c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.552+0000 7fb284989640 1 -- 172.21.15.98:0/2396567817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27000e050 con 0x563beb196030 2023-11-28T17:41:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.552+0000 7fb27ffff640 1 --2- 172.21.15.98:0/2396567817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb145890 0x563beb1838c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.552+0000 7fb284989640 1 --2- 172.21.15.98:0/2396567817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb186660 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb278009e50 tx=0x7fb278004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d83039132b7075 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.552+0000 7fb25effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-11-28T17:41:48.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.552+0000 7fb25effd640 1 -- 172.21.15.98:0/2396567817 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb278034bb0 con 0x563beb196030 2023-11-28T17:41:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.552+0000 7fb25effd640 1 -- 172.21.15.98:0/2396567817 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb278034d10 con 0x563beb196030 2023-11-28T17:41:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.552+0000 7fb288f85740 1 -- 172.21.15.98:0/2396567817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563beb0a5aa0 con 0x563beb196030 2023-11-28T17:41:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.553+0000 7fb25effd640 1 -- 172.21.15.98:0/2396567817 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb27803e470 con 0x563beb196030 2023-11-28T17:41:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.553+0000 7fb288f85740 10 client.4644 fetch_fsmap learned FSMap version 16 2023-11-28T17:41:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.553+0000 7fb288f85740 1 -- 172.21.15.98:0/2396567817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x563beb0ab140 con 0x563beb196030 2023-11-28T17:41:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.553+0000 7fb25effd640 1 -- 172.21.15.98:0/2396567817 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563beb0a5aa0 con 0x563beb196030 2023-11-28T17:41:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.553+0000 7fb25effd640 1 -- 172.21.15.98:0/2396567817 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fb27803e5d0 con 0x563beb196030 2023-11-28T17:41:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.553+0000 7fb288f85740 10 client.4644 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T17:41:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.553+0000 7fb288f85740 10 client.4644 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-28T17:41:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.553+0000 7fb288f85740 20 client.4644 populate_metadata read hostname 'smithi098' 2023-11-28T17:41:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.553+0000 7fb288f85740 1 --2- 172.21.15.98:0/2396567817 >> [v2:172.21.15.113:6833/1057450253,v1:172.21.15.113:6835/1057450253] conn(0x563beb199ec0 0x563beb1a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.553+0000 7fb288f85740 4 client.4644 mds_command: new command op to 4269 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T17:41:48.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.553+0000 7fb288f85740 1 -- 172.21.15.98:0/2396567817 --> [v2:172.21.15.113:6833/1057450253,v1:172.21.15.113:6835/1057450253] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563beb0a57c0 con 0x563beb199ec0 2023-11-28T17:41:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.554+0000 7fb27ffff640 1 --2- 172.21.15.98:0/2396567817 >> [v2:172.21.15.113:6833/1057450253,v1:172.21.15.113:6835/1057450253] conn(0x563beb199ec0 0x563beb1a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.555+0000 7fb27ffff640 1 --2- 172.21.15.98:0/2396567817 >> [v2:172.21.15.113:6833/1057450253,v1:172.21.15.113:6835/1057450253] conn(0x563beb199ec0 0x563beb1a4be0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4fb0ef0993784d96 server_cookie=fbd3466a681eaebb in_seq=0 out_seq=0 2023-11-28T17:41:48.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:48.555+0000 7fb25effd640 10 client.4644 ms_handle_connect on v2:172.21.15.113:6833/1057450253 2023-11-28T17:41:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.231+0000 7fb25effd640 1 -- 172.21.15.98:0/2396567817 <== mds.4 v2:172.21.15.113:6833/1057450253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563beb0a57c0 con 0x563beb199ec0 2023-11-28T17:41:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.231+0000 7fb25effd640 10 client.4644 handle_command_reply: tid=0 2023-11-28T17:41:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.231+0000 7fb288f85740 1 client.4644 shutdown 2023-11-28T17:41:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.232+0000 7fb288f85740 1 -- 172.21.15.98:0/2396567817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 msgr2=0x563beb186660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.232+0000 7fb288f85740 1 --2- 172.21.15.98:0/2396567817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb186660 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb278009e50 tx=0x7fb278004a40 comp rx=0 tx=0).stop 2023-11-28T17:41:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.232+0000 7fb288f85740 1 -- 172.21.15.98:0/2396567817 shutdown_connections 2023-11-28T17:41:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.232+0000 7fb288f85740 1 --2- 172.21.15.98:0/2396567817 >> [v2:172.21.15.113:6833/1057450253,v1:172.21.15.113:6835/1057450253] conn(0x563beb199ec0 0x563beb1a4be0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.232+0000 7fb288f85740 1 --2- 172.21.15.98:0/2396567817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb186660 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.232+0000 7fb288f85740 1 --2- 172.21.15.98:0/2396567817 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb18f710 0x563beb1817b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.232+0000 7fb288f85740 1 --2- 172.21.15.98:0/2396567817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb145890 0x563beb1838c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.232+0000 7fb25effd640 0 client.4644 ms_handle_reset on v2:172.21.15.113:6833/1057450253 2023-11-28T17:41:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.232+0000 7fb288f85740 1 -- 172.21.15.98:0/2396567817 >> 172.21.15.98:0/2396567817 conn(0x563beb1442f0 msgr2=0x563beb13f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.232+0000 7fb288f85740 1 -- 172.21.15.98:0/2396567817 shutdown_connections 2023-11-28T17:41:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.232+0000 7fb288f85740 1 -- 172.21.15.98:0/2396567817 wait complete. 2023-11-28T17:41:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.233+0000 7fb288f85740 20 client.4644 trim_cache size 0 max 16384 2023-11-28T17:41:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.269+0000 7fb288f85740 1 Processor -- start 2023-11-28T17:41:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.269+0000 7fb288f85740 1 -- start start 2023-11-28T17:41:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.271+0000 7fb288f85740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb145890 0x563beb145c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.271+0000 7fb288f85740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb18f710 0x563beb1461b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.271+0000 7fb288f85740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb16e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.271+0000 7fb288f85740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x563beae0ff20 con 0x563beb196030 2023-11-28T17:41:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.271+0000 7fb288f85740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x563beb16bfc0 con 0x563beb18f710 2023-11-28T17:41:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.271+0000 7fb288f85740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x563beb144180 con 0x563beb145890 2023-11-28T17:41:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.271+0000 7fb284989640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb16e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.271+0000 7fb284989640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb16e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56760/0 (socket says 172.21.15.98:56760) 2023-11-28T17:41:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.271+0000 7fb284989640 1 -- 172.21.15.98:0/2035938888 learned_addr learned my addr 172.21.15.98:0/2035938888 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.272+0000 7fb27ffff640 1 --2- 172.21.15.98:0/2035938888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb145890 0x563beb145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.272+0000 7fb277fff640 1 --2- 172.21.15.98:0/2035938888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb18f710 0x563beb1461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.272+0000 7fb284989640 1 -- 172.21.15.98:0/2035938888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb145890 msgr2=0x563beb145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.272+0000 7fb284989640 1 --2- 172.21.15.98:0/2035938888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb145890 0x563beb145c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.272+0000 7fb284989640 1 -- 172.21.15.98:0/2035938888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb18f710 msgr2=0x563beb1461b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.272+0000 7fb284989640 1 --2- 172.21.15.98:0/2035938888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb18f710 0x563beb1461b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.272+0000 7fb284989640 1 -- 172.21.15.98:0/2035938888 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563beb1452e0 con 0x563beb196030 2023-11-28T17:41:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.272+0000 7fb284989640 1 --2- 172.21.15.98:0/2035938888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb16e340 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb27802c790 tx=0x7fb278042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16ff0ca32964f4c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.273+0000 7fb2767fc640 1 -- 172.21.15.98:0/2035938888 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb278034440 con 0x563beb196030 2023-11-28T17:41:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.273+0000 7fb2767fc640 1 -- 172.21.15.98:0/2035938888 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2780345a0 con 0x563beb196030 2023-11-28T17:41:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.273+0000 7fb2767fc640 1 -- 172.21.15.98:0/2035938888 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27803e6b0 con 0x563beb196030 2023-11-28T17:41:50.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.274+0000 7fb288f85740 1 -- 172.21.15.98:0/2035938888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 msgr2=0x563beb16e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:50.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.274+0000 7fb288f85740 1 --2- 172.21.15.98:0/2035938888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb16e340 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb27802c790 tx=0x7fb278042000 comp rx=0 tx=0).stop 2023-11-28T17:41:50.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.274+0000 7fb288f85740 1 -- 172.21.15.98:0/2035938888 shutdown_connections 2023-11-28T17:41:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.274+0000 7fb288f85740 1 --2- 172.21.15.98:0/2035938888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb16e340 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.274+0000 7fb288f85740 1 --2- 172.21.15.98:0/2035938888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb18f710 0x563beb1461b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.274+0000 7fb288f85740 1 --2- 172.21.15.98:0/2035938888 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb145890 0x563beb145c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.274+0000 7fb288f85740 1 -- 172.21.15.98:0/2035938888 >> 172.21.15.98:0/2035938888 conn(0x563beb1442f0 msgr2=0x563beb18ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.274+0000 7fb288f85740 1 -- 172.21.15.98:0/2035938888 shutdown_connections 2023-11-28T17:41:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.274+0000 7fb288f85740 1 -- 172.21.15.98:0/2035938888 wait complete. 2023-11-28T17:41:50.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.276+0000 7fb288f85740 1 -- start start 2023-11-28T17:41:50.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.276+0000 7fb288f85740 1 Processor -- start 2023-11-28T17:41:50.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.278+0000 7fb288f85740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb145890 0x563beb180450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.278+0000 7fb288f85740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb18f710 0x563beb180ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.279+0000 7fb288f85740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb1eae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.279+0000 7fb288f85740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x563beb1427b0 con 0x563beb196030 2023-11-28T17:41:50.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.279+0000 7fb288f85740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x563beb181400 con 0x563beb145890 2023-11-28T17:41:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.279+0000 7fb288f85740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x563beb1f4a40 con 0x563beb18f710 2023-11-28T17:41:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.279+0000 7fb288f85740 10 client.0 _ll_register_callbacks cb 0x7fb278037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:41:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.279+0000 7fb284989640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb1eae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.279+0000 7fb284989640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb1eae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56774/0 (socket says 172.21.15.98:56774) 2023-11-28T17:41:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.279+0000 7fb284989640 1 -- 172.21.15.98:0/2415874274 learned_addr learned my addr 172.21.15.98:0/2415874274 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.279+0000 7fb27ffff640 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb145890 0x563beb180450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.279+0000 7fb277fff640 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb18f710 0x563beb180ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.279+0000 7fb284989640 1 -- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb18f710 msgr2=0x563beb180ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:50.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.279+0000 7fb284989640 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb18f710 0x563beb180ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.279+0000 7fb284989640 1 -- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb145890 msgr2=0x563beb180450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:50.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.279+0000 7fb284989640 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb145890 0x563beb180450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.279+0000 7fb284989640 1 -- 172.21.15.98:0/2415874274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563beb0ab140 con 0x563beb196030 2023-11-28T17:41:50.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.280+0000 7fb284989640 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb1eae30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb27804c040 tx=0x7fb278030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93048b87d2b94fa9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:50.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.280+0000 7fb25effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-11-28T17:41:50.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.280+0000 7fb25effd640 1 -- 172.21.15.98:0/2415874274 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb278034440 con 0x563beb196030 2023-11-28T17:41:50.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.280+0000 7fb25effd640 1 -- 172.21.15.98:0/2415874274 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb278034dc0 con 0x563beb196030 2023-11-28T17:41:50.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.280+0000 7fb288f85740 1 -- 172.21.15.98:0/2415874274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb27001a070 con 0x563beb196030 2023-11-28T17:41:50.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.281+0000 7fb25effd640 1 -- 172.21.15.98:0/2415874274 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb27803ee90 con 0x563beb196030 2023-11-28T17:41:50.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.281+0000 7fb288f85740 10 client.4650 fetch_fsmap learned FSMap version 16 2023-11-28T17:41:50.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.281+0000 7fb288f85740 1 -- 172.21.15.98:0/2415874274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x563beb048b80 con 0x563beb196030 2023-11-28T17:41:50.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.281+0000 7fb25effd640 1 -- 172.21.15.98:0/2415874274 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27001a070 con 0x563beb196030 2023-11-28T17:41:50.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.281+0000 7fb25effd640 1 -- 172.21.15.98:0/2415874274 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fb278003990 con 0x563beb196030 2023-11-28T17:41:50.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.281+0000 7fb288f85740 10 client.4650 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T17:41:50.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.281+0000 7fb288f85740 10 client.4650 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-28T17:41:50.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.281+0000 7fb288f85740 20 client.4650 populate_metadata read hostname 'smithi098' 2023-11-28T17:41:50.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.281+0000 7fb288f85740 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:6833/1057450253,v1:172.21.15.113:6835/1057450253] conn(0x563beb161f60 0x563beb162340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.282+0000 7fb288f85740 4 client.4650 mds_command: new command op to 4269 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T17:41:50.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.282+0000 7fb288f85740 1 -- 172.21.15.98:0/2415874274 --> [v2:172.21.15.113:6833/1057450253,v1:172.21.15.113:6835/1057450253] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563beadad6c0 con 0x563beb161f60 2023-11-28T17:41:50.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.282+0000 7fb27ffff640 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:6833/1057450253,v1:172.21.15.113:6835/1057450253] conn(0x563beb161f60 0x563beb162340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.282+0000 7fb27ffff640 1 -- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:6833/1057450253,v1:172.21.15.113:6835/1057450253] conn(0x563beb161f60 msgr2=0x563beb162340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2023-11-28T17:41:50.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.282+0000 7fb27ffff640 1 -- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:6833/1057450253,v1:172.21.15.113:6835/1057450253] conn(0x563beb161f60 msgr2=0x563beb162340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:41:50.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.282+0000 7fb27ffff640 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:6833/1057450253,v1:172.21.15.113:6835/1057450253] conn(0x563beb161f60 0x563beb162340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:41:50.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.282+0000 7fb27ffff640 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:6833/1057450253,v1:172.21.15.113:6835/1057450253] conn(0x563beb161f60 0x563beb162340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:41:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.483+0000 7fb27ffff640 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:6833/1057450253,v1:172.21.15.113:6835/1057450253] conn(0x563beb161f60 0x563beb162340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.485+0000 7fb27ffff640 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:6833/1057450253,v1:172.21.15.113:6835/1057450253] conn(0x563beb161f60 0x563beb162340 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=db8284c0bdac9471 server_cookie=c1c1c5dcaa465e79 in_seq=0 out_seq=0 2023-11-28T17:41:50.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.485+0000 7fb25effd640 10 client.4650 ms_handle_connect on v2:172.21.15.113:6833/1057450253 2023-11-28T17:41:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.486+0000 7fb25effd640 1 -- 172.21.15.98:0/2415874274 <== mds.4 v2:172.21.15.113:6833/1057450253 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x563beadad6c0 con 0x563beb161f60 2023-11-28T17:41:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.486+0000 7fb25effd640 10 client.4650 handle_command_reply: tid=0 2023-11-28T17:41:50.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.486+0000 7fb288f85740 1 client.4650 shutdown 2023-11-28T17:41:50.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.486+0000 7fb288f85740 1 -- 172.21.15.98:0/2415874274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 msgr2=0x563beb1eae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:50.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.486+0000 7fb288f85740 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb1eae30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb27804c040 tx=0x7fb278030000 comp rx=0 tx=0).stop 2023-11-28T17:41:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.486+0000 7fb288f85740 1 -- 172.21.15.98:0/2415874274 shutdown_connections 2023-11-28T17:41:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.486+0000 7fb288f85740 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:6833/1057450253,v1:172.21.15.113:6835/1057450253] conn(0x563beb161f60 0x563beb162340 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.486+0000 7fb288f85740 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563beb196030 0x563beb1eae30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.486+0000 7fb288f85740 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563beb18f710 0x563beb180ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.486+0000 7fb25effd640 0 client.4650 ms_handle_reset on v2:172.21.15.113:6833/1057450253 2023-11-28T17:41:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.486+0000 7fb288f85740 1 --2- 172.21.15.98:0/2415874274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563beb145890 0x563beb180450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.487+0000 7fb288f85740 1 -- 172.21.15.98:0/2415874274 >> 172.21.15.98:0/2415874274 conn(0x563beb1442f0 msgr2=0x563beb1437e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:50.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.487+0000 7fb288f85740 1 -- 172.21.15.98:0/2415874274 shutdown_connections 2023-11-28T17:41:50.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.487+0000 7fb288f85740 1 -- 172.21.15.98:0/2415874274 wait complete. 2023-11-28T17:41:50.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.487+0000 7fb288f85740 20 client.4650 trim_cache size 0 max 16384 2023-11-28T17:41:50.489 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:41:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:41:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 4, 2023-11-28T17:41:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4269, 2023-11-28T17:41:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-11-28T17:41:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-11-28T17:41:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-11-28T17:41:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 11.173903921999999, 2023-11-28T17:41:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 16, 2023-11-28T17:41:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 22, 2023-11-28T17:41:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 22, 2023-11-28T17:41:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 40.601302496000002 2023-11-28T17:41:50.491 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:41:50.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.488+0000 7fb2863ec640 1 -- 172.21.15.98:0/2134457464 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb26006b2a0 msgr2=0x7fb26006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:50.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.488+0000 7fb2863ec640 1 --2- 172.21.15.98:0/2134457464 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb26006b2a0 0x7fb26006d760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb270009a10 tx=0x7fb270008040 comp rx=0 tx=0).stop 2023-11-28T17:41:50.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.488+0000 7fb2863ec640 1 -- 172.21.15.98:0/2134457464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2800c8600 msgr2=0x7fb28014e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:50.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.488+0000 7fb2863ec640 1 --2- 172.21.15.98:0/2134457464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2800c8600 0x7fb28014e9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb26800c480 tx=0x7fb26800c940 comp rx=0 tx=0).stop 2023-11-28T17:41:50.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.488+0000 7fb2863ec640 1 -- 172.21.15.98:0/2134457464 shutdown_connections 2023-11-28T17:41:50.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.488+0000 7fb2863ec640 1 --2- 172.21.15.98:0/2134457464 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb280154030 0x7fb280158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.488+0000 7fb2863ec640 1 --2- 172.21.15.98:0/2134457464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2800c8600 0x7fb28014e9a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.488+0000 7fb2863ec640 1 --2- 172.21.15.98:0/2134457464 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb26006b2a0 0x7fb26006d760 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.488+0000 7fb2863ec640 1 --2- 172.21.15.98:0/2134457464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb28006dcf0 0x7fb280152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.488+0000 7fb2863ec640 1 -- 172.21.15.98:0/2134457464 >> 172.21.15.98:0/2134457464 conn(0x7fb28006c0f0 msgr2=0x7fb28006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:50.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.489+0000 7fb2863ec640 1 -- 172.21.15.98:0/2134457464 shutdown_connections 2023-11-28T17:41:50.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.489+0000 7fb2863ec640 1 -- 172.21.15.98:0/2134457464 wait complete. 2023-11-28T17:41:50.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T17:41:50.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.730+0000 7f187a0b4640 1 Processor -- start 2023-11-28T17:41:50.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.730+0000 7f187a0b4640 1 -- start start 2023-11-28T17:41:50.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.732+0000 7f187a0b4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1874154030 0x7f1874070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.732+0000 7f187a0b4640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f187406dcf0 0x7f187406e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.732+0000 7f187a0b4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1874154410 con 0x7f18740c8600 2023-11-28T17:41:50.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.732+0000 7f187a0b4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18740715a0 con 0x7f1874154030 2023-11-28T17:41:50.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.732+0000 7f187a0b4640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18740c5da0 con 0x7f187406dcf0 2023-11-28T17:41:50.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.732+0000 7f18737fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1874154030 0x7f1874070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.732+0000 7f18737fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1874154030 0x7f1874070660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56788/0 (socket says 172.21.15.98:56788) 2023-11-28T17:41:50.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.732+0000 7f18737fe640 1 -- 172.21.15.98:0/2578713765 learned_addr learned my addr 172.21.15.98:0/2578713765 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:50.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.732+0000 7f186bfff640 1 --1- 172.21.15.98:0/2578713765 >> v1:172.21.15.98:6789/0 conn(0x7f18740c8600 0x7f1874070e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46846/0 (socket says 172.21.15.98:46846) 2023-11-28T17:41:50.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.733+0000 7f1873fff640 1 --2- 172.21.15.98:0/2578713765 >> v2:172.21.15.113:3300/0 conn(0x7f187406dcf0 0x7f187406e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.733+0000 7f18737fe640 1 -- 172.21.15.98:0/2578713765 >> v2:172.21.15.113:3300/0 conn(0x7f187406dcf0 msgr2=0x7f187406e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:50.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.733+0000 7f18737fe640 1 --2- 172.21.15.98:0/2578713765 >> v2:172.21.15.113:3300/0 conn(0x7f187406dcf0 0x7f187406e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.733+0000 7f18737fe640 1 -- 172.21.15.98:0/2578713765 >> v1:172.21.15.98:6789/0 conn(0x7f18740c8600 legacy=0x7f1874070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:50.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.733+0000 7f18737fe640 1 -- 172.21.15.98:0/2578713765 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187415d370 con 0x7f1874154030 2023-11-28T17:41:50.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.734+0000 7f18737fe640 1 --2- 172.21.15.98:0/2578713765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874154030 0x7f1874070660 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f187406ea60 tx=0x7f186402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa2f526f88e0b57e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:50.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.735+0000 7f1872ffd640 1 -- 172.21.15.98:0/2578713765 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186402f850 con 0x7f1874154030 2023-11-28T17:41:50.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.735+0000 7f1872ffd640 1 -- 172.21.15.98:0/2578713765 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186402f9b0 con 0x7f1874154030 2023-11-28T17:41:50.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.735+0000 7f1872ffd640 1 -- 172.21.15.98:0/2578713765 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186402fd40 con 0x7f1874154030 2023-11-28T17:41:50.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.735+0000 7f187a0b4640 1 -- 172.21.15.98:0/2578713765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874154030 msgr2=0x7f1874070660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:50.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.735+0000 7f187a0b4640 1 --2- 172.21.15.98:0/2578713765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874154030 0x7f1874070660 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f187406ea60 tx=0x7f186402ba30 comp rx=0 tx=0).stop 2023-11-28T17:41:50.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.735+0000 7f187a0b4640 1 -- 172.21.15.98:0/2578713765 shutdown_connections 2023-11-28T17:41:50.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.735+0000 7f187a0b4640 1 --2- 172.21.15.98:0/2578713765 >> v2:172.21.15.113:3300/0 conn(0x7f187406dcf0 0x7f187406e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.735+0000 7f187a0b4640 1 --2- 172.21.15.98:0/2578713765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874154030 0x7f1874070660 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.735+0000 7f187a0b4640 1 -- 172.21.15.98:0/2578713765 >> 172.21.15.98:0/2578713765 conn(0x7f187406c0f0 msgr2=0x7f187406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:50.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.735+0000 7f187a0b4640 1 -- 172.21.15.98:0/2578713765 shutdown_connections 2023-11-28T17:41:50.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.736+0000 7f187a0b4640 1 -- 172.21.15.98:0/2578713765 wait complete. 2023-11-28T17:41:50.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.737+0000 7f187a0b4640 1 Processor -- start 2023-11-28T17:41:50.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.737+0000 7f187a0b4640 1 -- start start 2023-11-28T17:41:50.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.738+0000 7f187a0b4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187406dcf0 0x7f1874158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.738+0000 7f187a0b4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18740c8600 0x7f18741466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.738+0000 7f187a0b4640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1874154030 0x7f1874146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.738+0000 7f187a0b4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1874147160 con 0x7f187406dcf0 2023-11-28T17:41:50.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.738+0000 7f187a0b4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1874145ca0 con 0x7f18740c8600 2023-11-28T17:41:50.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.738+0000 7f187a0b4640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18741456e0 con 0x7f1874154030 2023-11-28T17:41:50.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.739+0000 7f18737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187406dcf0 0x7f1874158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.739+0000 7f18737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187406dcf0 0x7f1874158ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56792/0 (socket says 172.21.15.98:56792) 2023-11-28T17:41:50.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.739+0000 7f18737fe640 1 -- 172.21.15.98:0/2066641507 learned_addr learned my addr 172.21.15.98:0/2066641507 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:50.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.739+0000 7f186bfff640 1 --2- 172.21.15.98:0/2066641507 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18740c8600 0x7f18741466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.739+0000 7f1873fff640 1 --2- 172.21.15.98:0/2066641507 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1874154030 0x7f1874146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.739+0000 7f18737fe640 1 -- 172.21.15.98:0/2066641507 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1874154030 msgr2=0x7f1874146c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.739+0000 7f18737fe640 1 --2- 172.21.15.98:0/2066641507 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1874154030 0x7f1874146c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.739+0000 7f18737fe640 1 -- 172.21.15.98:0/2066641507 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18740c8600 msgr2=0x7f18741466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.739+0000 7f18737fe640 1 --2- 172.21.15.98:0/2066641507 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18740c8600 0x7f18741466e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.739+0000 7f18737fe640 1 -- 172.21.15.98:0/2066641507 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18640094d0 con 0x7f187406dcf0 2023-11-28T17:41:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.740+0000 7f18737fe640 1 --2- 172.21.15.98:0/2066641507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187406dcf0 0x7f1874158ed0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f186402ba70 tx=0x7f1864004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.740+0000 7f1870ff9640 1 -- 172.21.15.98:0/2066641507 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864037d10 con 0x7f187406dcf0 2023-11-28T17:41:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.740+0000 7f1870ff9640 1 -- 172.21.15.98:0/2066641507 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1864037e90 con 0x7f187406dcf0 2023-11-28T17:41:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.740+0000 7f1870ff9640 1 -- 172.21.15.98:0/2066641507 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864041640 con 0x7f187406dcf0 2023-11-28T17:41:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.740+0000 7f187a0b4640 1 -- 172.21.15.98:0/2066641507 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18741459c0 con 0x7f187406dcf0 2023-11-28T17:41:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.740+0000 7f187a0b4640 1 -- 172.21.15.98:0/2066641507 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1874151c80 con 0x7f187406dcf0 2023-11-28T17:41:50.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.742+0000 7f1870ff9640 1 -- 172.21.15.98:0/2066641507 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f18640417a0 con 0x7f187406dcf0 2023-11-28T17:41:50.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.743+0000 7f1870ff9640 1 --2- 172.21.15.98:0/2066641507 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f185406b2a0 0x7f185406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.743+0000 7f1870ff9640 1 -- 172.21.15.98:0/2066641507 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f186403c070 con 0x7f187406dcf0 2023-11-28T17:41:50.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.743+0000 7f186bfff640 1 --2- 172.21.15.98:0/2066641507 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f185406b2a0 0x7f185406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.744+0000 7f186bfff640 1 --2- 172.21.15.98:0/2066641507 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f185406b2a0 0x7f185406d760 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f18740c54c0 tx=0x7f185c007e00 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:50.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.744+0000 7f187cc46740 1 Processor -- start 2023-11-28T17:41:50.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.744+0000 7f187cc46740 1 -- start start 2023-11-28T17:41:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.745+0000 7f187cc46740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e8477c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.745+0000 7f187cc46740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84c1710 0x5627e84781b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.745+0000 7f187cc46740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84762f0 0x5627e84b7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.745+0000 7f187cc46740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e83d6880 con 0x5627e8477890 2023-11-28T17:41:50.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.745+0000 7f187cc46740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e83e5ae0 con 0x5627e84762f0 2023-11-28T17:41:50.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.745+0000 7f187cc46740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x5627e83e31d0 con 0x5627e84c1710 2023-11-28T17:41:50.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.745+0000 7f18737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e8477c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.745+0000 7f18737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e8477c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56804/0 (socket says 172.21.15.98:56804) 2023-11-28T17:41:50.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.745+0000 7f18737fe640 1 -- 172.21.15.98:0/2713231660 learned_addr learned my addr 172.21.15.98:0/2713231660 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:50.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.746+0000 7f18737fe640 1 -- 172.21.15.98:0/2713231660 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84c1710 msgr2=0x5627e84781b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:50.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.746+0000 7f18737fe640 1 --2- 172.21.15.98:0/2713231660 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84c1710 0x5627e84781b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.746+0000 7f18737fe640 1 -- 172.21.15.98:0/2713231660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84762f0 msgr2=0x5627e84b7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:50.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.746+0000 7f18737fe640 1 --2- 172.21.15.98:0/2713231660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84762f0 0x5627e84b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.746+0000 7f18737fe640 1 -- 172.21.15.98:0/2713231660 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627e83a6860 con 0x5627e8477890 2023-11-28T17:41:50.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.746+0000 7f18737fe640 1 --2- 172.21.15.98:0/2713231660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e8477c70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1864006da0 tx=0x7f186407b3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7882d389c83b4de server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:50.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.746+0000 7f18697fa640 1 -- 172.21.15.98:0/2713231660 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864083850 con 0x5627e8477890 2023-11-28T17:41:50.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.746+0000 7f18697fa640 1 -- 172.21.15.98:0/2713231660 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1864079860 con 0x5627e8477890 2023-11-28T17:41:50.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.746+0000 7f18697fa640 1 -- 172.21.15.98:0/2713231660 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18640833c0 con 0x5627e8477890 2023-11-28T17:41:50.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.747+0000 7f187cc46740 1 -- 172.21.15.98:0/2713231660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 msgr2=0x5627e8477c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:50.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.747+0000 7f187cc46740 1 --2- 172.21.15.98:0/2713231660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e8477c70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1864006da0 tx=0x7f186407b3c0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.747+0000 7f187cc46740 1 -- 172.21.15.98:0/2713231660 shutdown_connections 2023-11-28T17:41:50.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.747+0000 7f187cc46740 1 --2- 172.21.15.98:0/2713231660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84762f0 0x5627e84b7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.747+0000 7f187cc46740 1 --2- 172.21.15.98:0/2713231660 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84c1710 0x5627e84781b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.747+0000 7f187cc46740 1 --2- 172.21.15.98:0/2713231660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e8477c70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.747+0000 7f187cc46740 1 -- 172.21.15.98:0/2713231660 >> 172.21.15.98:0/2713231660 conn(0x5627e8474290 msgr2=0x5627e8476940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:50.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.747+0000 7f187cc46740 1 -- 172.21.15.98:0/2713231660 shutdown_connections 2023-11-28T17:41:50.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.747+0000 7f187cc46740 1 -- 172.21.15.98:0/2713231660 wait complete. 2023-11-28T17:41:50.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.748+0000 7f187cc46740 1 -- start start 2023-11-28T17:41:50.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.748+0000 7f187cc46740 1 Processor -- start 2023-11-28T17:41:50.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f187cc46740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e84b86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f187cc46740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84c1710 0x5627e84b2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f187cc46740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84b4330 0x5627e84b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f187cc46740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e83deac0 con 0x5627e8477890 2023-11-28T17:41:50.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f187cc46740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e837b2f0 con 0x5627e84c1710 2023-11-28T17:41:50.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f187cc46740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x5627e8398c70 con 0x5627e84b4330 2023-11-28T17:41:50.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f187cc46740 10 client.0 _ll_register_callbacks cb 0x5627e80f0ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:41:50.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f18737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e84b86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f18737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e84b86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56818/0 (socket says 172.21.15.98:56818) 2023-11-28T17:41:50.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f18737fe640 1 -- 172.21.15.98:0/2909470103 learned_addr learned my addr 172.21.15.98:0/2909470103 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:50.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f186bfff640 1 --2- 172.21.15.98:0/2909470103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84c1710 0x5627e84b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f18737fe640 1 -- 172.21.15.98:0/2909470103 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84b4330 msgr2=0x5627e84b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:50.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f18737fe640 1 --2- 172.21.15.98:0/2909470103 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84b4330 0x5627e84b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f18737fe640 1 -- 172.21.15.98:0/2909470103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84c1710 msgr2=0x5627e84b2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:50.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f18737fe640 1 --2- 172.21.15.98:0/2909470103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84c1710 0x5627e84b2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f18737fe640 1 -- 172.21.15.98:0/2909470103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1874151c80 con 0x5627e8477890 2023-11-28T17:41:50.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.749+0000 7f186bfff640 1 --2- 172.21.15.98:0/2909470103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84c1710 0x5627e84b2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:50.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.750+0000 7f18737fe640 1 --2- 172.21.15.98:0/2909470103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e84b86e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1864083320 tx=0x7f18640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b83d3592943ab50 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:50.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.750+0000 7f18527fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-11-28T17:41:50.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.750+0000 7f18527fc640 1 -- 172.21.15.98:0/2909470103 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864083850 con 0x5627e8477890 2023-11-28T17:41:50.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.750+0000 7f18527fc640 1 -- 172.21.15.98:0/2909470103 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18640772b0 con 0x5627e8477890 2023-11-28T17:41:50.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.750+0000 7f187cc46740 1 -- 172.21.15.98:0/2909470103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5627e83d7aa0 con 0x5627e8477890 2023-11-28T17:41:50.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.750+0000 7f18527fc640 1 -- 172.21.15.98:0/2909470103 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1864077470 con 0x5627e8477890 2023-11-28T17:41:50.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.750+0000 7f187cc46740 10 client.4665 fetch_fsmap learned FSMap version 16 2023-11-28T17:41:50.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.750+0000 7f187cc46740 1 -- 172.21.15.98:0/2909470103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x5627e83dd140 con 0x5627e8477890 2023-11-28T17:41:50.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.750+0000 7f18527fc640 1 -- 172.21.15.98:0/2909470103 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5627e83d7aa0 con 0x5627e8477890 2023-11-28T17:41:50.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.750+0000 7f18527fc640 1 -- 172.21.15.98:0/2909470103 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f186409d020 con 0x5627e8477890 2023-11-28T17:41:50.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.750+0000 7f187cc46740 10 client.4665 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T17:41:50.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.750+0000 7f187cc46740 10 client.4665 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T17:41:50.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.751+0000 7f187cc46740 20 client.4665 populate_metadata read hostname 'smithi098' 2023-11-28T17:41:50.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.751+0000 7f187cc46740 1 --2- 172.21.15.98:0/2909470103 >> [v2:172.21.15.98:6835/1925178583,v1:172.21.15.98:6837/1925178583] conn(0x5627e84b2b50 0x5627e84cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.751+0000 7f187cc46740 4 client.4665 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T17:41:50.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.751+0000 7f187cc46740 1 -- 172.21.15.98:0/2909470103 --> [v2:172.21.15.98:6835/1925178583,v1:172.21.15.98:6837/1925178583] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5627e83d77c0 con 0x5627e84b2b50 2023-11-28T17:41:50.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.751+0000 7f186bfff640 1 --2- 172.21.15.98:0/2909470103 >> [v2:172.21.15.98:6835/1925178583,v1:172.21.15.98:6837/1925178583] conn(0x5627e84b2b50 0x5627e84cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.751+0000 7f186bfff640 1 --2- 172.21.15.98:0/2909470103 >> [v2:172.21.15.98:6835/1925178583,v1:172.21.15.98:6837/1925178583] conn(0x5627e84b2b50 0x5627e84cbdc0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=925e0ef9f0dca3f2 server_cookie=a67696fecd8e9901 in_seq=0 out_seq=0 2023-11-28T17:41:50.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.751+0000 7f18527fc640 10 client.4665 ms_handle_connect on v2:172.21.15.98:6835/1925178583 2023-11-28T17:41:50.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.752+0000 7f18527fc640 1 -- 172.21.15.98:0/2909470103 <== mds.3 v2:172.21.15.98:6835/1925178583 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5627e83d77c0 con 0x5627e84b2b50 2023-11-28T17:41:50.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.752+0000 7f18527fc640 10 client.4665 handle_command_reply: tid=0 2023-11-28T17:41:50.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.752+0000 7f187cc46740 1 client.4665 shutdown 2023-11-28T17:41:50.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.753+0000 7f187cc46740 1 -- 172.21.15.98:0/2909470103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 msgr2=0x5627e84b86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:50.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.753+0000 7f187cc46740 1 --2- 172.21.15.98:0/2909470103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e84b86e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1864083320 tx=0x7f18640046c0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.753+0000 7f187cc46740 1 -- 172.21.15.98:0/2909470103 shutdown_connections 2023-11-28T17:41:50.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.753+0000 7f187cc46740 1 --2- 172.21.15.98:0/2909470103 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84b4330 0x5627e84b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.753+0000 7f187cc46740 1 --2- 172.21.15.98:0/2909470103 >> [v2:172.21.15.98:6835/1925178583,v1:172.21.15.98:6837/1925178583] conn(0x5627e84b2b50 0x5627e84cbdc0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.753+0000 7f187cc46740 1 --2- 172.21.15.98:0/2909470103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84c1710 0x5627e84b2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.753+0000 7f187cc46740 1 --2- 172.21.15.98:0/2909470103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e84b86e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.753+0000 7f18527fc640 0 client.4665 ms_handle_reset on v2:172.21.15.98:6835/1925178583 2023-11-28T17:41:50.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.753+0000 7f187cc46740 1 -- 172.21.15.98:0/2909470103 >> 172.21.15.98:0/2909470103 conn(0x5627e84762f0 msgr2=0x5627e8479150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:50.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.753+0000 7f187cc46740 1 -- 172.21.15.98:0/2909470103 shutdown_connections 2023-11-28T17:41:50.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.753+0000 7f187cc46740 1 -- 172.21.15.98:0/2909470103 wait complete. 2023-11-28T17:41:50.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.753+0000 7f187cc46740 20 client.4665 trim_cache size 0 max 16384 2023-11-28T17:41:50.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.770+0000 7f187cc46740 1 Processor -- start 2023-11-28T17:41:50.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.770+0000 7f187cc46740 1 -- start start 2023-11-28T17:41:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.771+0000 7f187cc46740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e849bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.771+0000 7f187cc46740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84b4330 0x5627e84d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.771+0000 7f187cc46740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84c1710 0x5627e84d2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.771+0000 7f187cc46740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e8141f20 con 0x5627e8477890 2023-11-28T17:41:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.771+0000 7f187cc46740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e84c82c0 con 0x5627e84c1710 2023-11-28T17:41:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.771+0000 7f187cc46740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x5627e84747c0 con 0x5627e84b4330 2023-11-28T17:41:50.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.771+0000 7f18737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e849bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.771+0000 7f18737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e849bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56828/0 (socket says 172.21.15.98:56828) 2023-11-28T17:41:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.771+0000 7f18737fe640 1 -- 172.21.15.98:0/129921860 learned_addr learned my addr 172.21.15.98:0/129921860 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.771+0000 7f18737fe640 1 -- 172.21.15.98:0/129921860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84b4330 msgr2=0x5627e84d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.771+0000 7f18737fe640 1 --2- 172.21.15.98:0/129921860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84b4330 0x5627e84d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.771+0000 7f18737fe640 1 -- 172.21.15.98:0/129921860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84c1710 msgr2=0x5627e84d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.771+0000 7f18737fe640 1 --2- 172.21.15.98:0/129921860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84c1710 0x5627e84d2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.771+0000 7f18737fe640 1 -- 172.21.15.98:0/129921860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627e837ab80 con 0x5627e8477890 2023-11-28T17:41:50.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.772+0000 7f18737fe640 1 --2- 172.21.15.98:0/129921860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e849bcb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f186407b400 tx=0x7f1864094000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a02ceb62c9722376 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:50.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.772+0000 7f1869ffb640 1 -- 172.21.15.98:0/129921860 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864083850 con 0x5627e8477890 2023-11-28T17:41:50.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.772+0000 7f1869ffb640 1 -- 172.21.15.98:0/129921860 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1864079600 con 0x5627e8477890 2023-11-28T17:41:50.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.772+0000 7f1869ffb640 1 -- 172.21.15.98:0/129921860 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18640772b0 con 0x5627e8477890 2023-11-28T17:41:50.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.772+0000 7f187cc46740 1 -- 172.21.15.98:0/129921860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 msgr2=0x5627e849bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:50.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.772+0000 7f187cc46740 1 --2- 172.21.15.98:0/129921860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e849bcb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f186407b400 tx=0x7f1864094000 comp rx=0 tx=0).stop 2023-11-28T17:41:50.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.772+0000 7f187cc46740 1 -- 172.21.15.98:0/129921860 shutdown_connections 2023-11-28T17:41:50.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.772+0000 7f187cc46740 1 --2- 172.21.15.98:0/129921860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84c1710 0x5627e84d2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.772+0000 7f187cc46740 1 --2- 172.21.15.98:0/129921860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84b4330 0x5627e84d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.772+0000 7f187cc46740 1 --2- 172.21.15.98:0/129921860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e849bcb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.772+0000 7f187cc46740 1 -- 172.21.15.98:0/129921860 >> 172.21.15.98:0/129921860 conn(0x5627e84762f0 msgr2=0x5627e846d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:50.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.773+0000 7f187cc46740 1 -- 172.21.15.98:0/129921860 shutdown_connections 2023-11-28T17:41:50.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.773+0000 7f187cc46740 1 -- 172.21.15.98:0/129921860 wait complete. 2023-11-28T17:41:50.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.773+0000 7f187cc46740 1 -- start start 2023-11-28T17:41:50.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.773+0000 7f187cc46740 1 Processor -- start 2023-11-28T17:41:50.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.774+0000 7f187cc46740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e84d21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.774+0000 7f187cc46740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84b4330 0x5627e8522ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.774+0000 7f187cc46740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84c1710 0x5627e851cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.774+0000 7f187cc46740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e84c1350 con 0x5627e8477890 2023-11-28T17:41:50.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.774+0000 7f187cc46740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627e84d2730 con 0x5627e84c1710 2023-11-28T17:41:50.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.774+0000 7f187cc46740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x5627e85238f0 con 0x5627e84b4330 2023-11-28T17:41:50.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.774+0000 7f187cc46740 10 client.0 _ll_register_callbacks cb 0x7f186407b090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:41:50.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.775+0000 7f18737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e84d21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.775+0000 7f18737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e84d21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56842/0 (socket says 172.21.15.98:56842) 2023-11-28T17:41:50.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.775+0000 7f18737fe640 1 -- 172.21.15.98:0/2330084402 learned_addr learned my addr 172.21.15.98:0/2330084402 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:50.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.775+0000 7f186bfff640 1 --2- 172.21.15.98:0/2330084402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84b4330 0x5627e8522ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.775+0000 7f1873fff640 1 --2- 172.21.15.98:0/2330084402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84c1710 0x5627e851cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.775+0000 7f18737fe640 1 -- 172.21.15.98:0/2330084402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84b4330 msgr2=0x5627e8522ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:50.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.775+0000 7f18737fe640 1 --2- 172.21.15.98:0/2330084402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84b4330 0x5627e8522ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.775+0000 7f18737fe640 1 -- 172.21.15.98:0/2330084402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84c1710 msgr2=0x5627e851cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:50.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.775+0000 7f18737fe640 1 --2- 172.21.15.98:0/2330084402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84c1710 0x5627e851cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.775+0000 7f18737fe640 1 -- 172.21.15.98:0/2330084402 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627e83dd140 con 0x5627e8477890 2023-11-28T17:41:50.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.775+0000 7f18737fe640 1 --2- 172.21.15.98:0/2330084402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e84d21f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f18640779c0 tx=0x7f1864077880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f457537b51102ec server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:50.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.775+0000 7f18527fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-11-28T17:41:50.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.775+0000 7f18527fc640 1 -- 172.21.15.98:0/2330084402 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864083850 con 0x5627e8477890 2023-11-28T17:41:50.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.775+0000 7f18527fc640 1 -- 172.21.15.98:0/2330084402 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f186407cd00 con 0x5627e8477890 2023-11-28T17:41:50.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.775+0000 7f187cc46740 1 -- 172.21.15.98:0/2330084402 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5627e84b32d0 con 0x5627e8477890 2023-11-28T17:41:50.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.776+0000 7f18527fc640 1 -- 172.21.15.98:0/2330084402 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f186407a420 con 0x5627e8477890 2023-11-28T17:41:50.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.776+0000 7f187cc46740 10 client.4671 fetch_fsmap learned FSMap version 16 2023-11-28T17:41:50.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.776+0000 7f187cc46740 1 -- 172.21.15.98:0/2330084402 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x5627e83a0170 con 0x5627e8477890 2023-11-28T17:41:50.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.776+0000 7f18527fc640 1 -- 172.21.15.98:0/2330084402 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5627e84b32d0 con 0x5627e8477890 2023-11-28T17:41:50.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.776+0000 7f18527fc640 1 -- 172.21.15.98:0/2330084402 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f186409d030 con 0x5627e8477890 2023-11-28T17:41:50.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.776+0000 7f187cc46740 10 client.4671 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T17:41:50.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.776+0000 7f187cc46740 10 client.4671 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T17:41:50.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.776+0000 7f187cc46740 20 client.4671 populate_metadata read hostname 'smithi098' 2023-11-28T17:41:50.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.776+0000 7f187cc46740 1 --2- 172.21.15.98:0/2330084402 >> [v2:172.21.15.98:6835/1925178583,v1:172.21.15.98:6837/1925178583] conn(0x5627e851d130 0x5627e84b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:50.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.776+0000 7f187cc46740 4 client.4671 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T17:41:50.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.776+0000 7f187cc46740 1 -- 172.21.15.98:0/2330084402 --> [v2:172.21.15.98:6835/1925178583,v1:172.21.15.98:6837/1925178583] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f1864079860 con 0x5627e851d130 2023-11-28T17:41:50.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.776+0000 7f186bfff640 1 --2- 172.21.15.98:0/2330084402 >> [v2:172.21.15.98:6835/1925178583,v1:172.21.15.98:6837/1925178583] conn(0x5627e851d130 0x5627e84b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:50.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f186bfff640 1 --2- 172.21.15.98:0/2330084402 >> [v2:172.21.15.98:6835/1925178583,v1:172.21.15.98:6837/1925178583] conn(0x5627e851d130 0x5627e84b5180 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=799c1a3d088e919e server_cookie=688cfeb35f192de0 in_seq=0 out_seq=0 2023-11-28T17:41:50.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f18527fc640 10 client.4671 ms_handle_connect on v2:172.21.15.98:6835/1925178583 2023-11-28T17:41:50.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f18527fc640 1 -- 172.21.15.98:0/2330084402 <== mds.3 v2:172.21.15.98:6835/1925178583 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x7f1864079860 con 0x5627e851d130 2023-11-28T17:41:50.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f18527fc640 10 client.4671 handle_command_reply: tid=0 2023-11-28T17:41:50.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f187cc46740 1 client.4671 shutdown 2023-11-28T17:41:50.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f187cc46740 1 -- 172.21.15.98:0/2330084402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 msgr2=0x5627e84d21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:50.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f187cc46740 1 --2- 172.21.15.98:0/2330084402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e84d21f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f18640779c0 tx=0x7f1864077880 comp rx=0 tx=0).stop 2023-11-28T17:41:50.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f187cc46740 1 -- 172.21.15.98:0/2330084402 shutdown_connections 2023-11-28T17:41:50.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f187cc46740 1 --2- 172.21.15.98:0/2330084402 >> [v2:172.21.15.98:6835/1925178583,v1:172.21.15.98:6837/1925178583] conn(0x5627e851d130 0x5627e84b5180 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f187cc46740 1 --2- 172.21.15.98:0/2330084402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627e84c1710 0x5627e851cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f187cc46740 1 --2- 172.21.15.98:0/2330084402 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x5627e84b4330 0x5627e8522ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f187cc46740 1 --2- 172.21.15.98:0/2330084402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627e8477890 0x5627e84d21f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f18527fc640 0 client.4671 ms_handle_reset on v2:172.21.15.98:6835/1925178583 2023-11-28T17:41:50.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f187cc46740 1 -- 172.21.15.98:0/2330084402 >> 172.21.15.98:0/2330084402 conn(0x5627e84762f0 msgr2=0x5627e84b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:50.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f187cc46740 1 -- 172.21.15.98:0/2330084402 shutdown_connections 2023-11-28T17:41:50.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.777+0000 7f187cc46740 1 -- 172.21.15.98:0/2330084402 wait complete. 2023-11-28T17:41:50.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.778+0000 7f187cc46740 20 client.4671 trim_cache size 0 max 16384 2023-11-28T17:41:50.783 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:41:50.784 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:41:50.784 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 3, 2023-11-28T17:41:50.784 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4270, 2023-11-28T17:41:50.784 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-11-28T17:41:50.784 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-11-28T17:41:50.784 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-11-28T17:41:50.784 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 16.559325596000001, 2023-11-28T17:41:50.784 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 16, 2023-11-28T17:41:50.784 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 22, 2023-11-28T17:41:50.784 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 22, 2023-11-28T17:41:50.785 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 40.927259978999999 2023-11-28T17:41:50.785 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:41:50.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.778+0000 7f187a0b4640 1 -- 172.21.15.98:0/2066641507 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f185406b2a0 msgr2=0x7f185406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:50.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.778+0000 7f187a0b4640 1 --2- 172.21.15.98:0/2066641507 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f185406b2a0 0x7f185406d760 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f18740c54c0 tx=0x7f185c007e00 comp rx=0 tx=0).stop 2023-11-28T17:41:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.778+0000 7f187a0b4640 1 -- 172.21.15.98:0/2066641507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187406dcf0 msgr2=0x7f1874158ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.778+0000 7f187a0b4640 1 --2- 172.21.15.98:0/2066641507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187406dcf0 0x7f1874158ed0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f186402ba70 tx=0x7f1864004730 comp rx=0 tx=0).stop 2023-11-28T17:41:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.778+0000 7f187a0b4640 1 -- 172.21.15.98:0/2066641507 shutdown_connections 2023-11-28T17:41:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.778+0000 7f187a0b4640 1 --2- 172.21.15.98:0/2066641507 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1874154030 0x7f1874146c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.778+0000 7f187a0b4640 1 --2- 172.21.15.98:0/2066641507 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f185406b2a0 0x7f185406d760 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.778+0000 7f187a0b4640 1 --2- 172.21.15.98:0/2066641507 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18740c8600 0x7f18741466e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.778+0000 7f187a0b4640 1 --2- 172.21.15.98:0/2066641507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f187406dcf0 0x7f1874158ed0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.778+0000 7f187a0b4640 1 -- 172.21.15.98:0/2066641507 >> 172.21.15.98:0/2066641507 conn(0x7f187406c0f0 msgr2=0x7f1874158030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.778+0000 7f187a0b4640 1 -- 172.21.15.98:0/2066641507 shutdown_connections 2023-11-28T17:41:50.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:50.778+0000 7f187a0b4640 1 -- 172.21.15.98:0/2066641507 wait complete. 2023-11-28T17:41:50.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-11-28T17:41:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.001+0000 7fbe7bc64640 1 Processor -- start 2023-11-28T17:41:51.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.001+0000 7fbe7bc64640 1 -- start start 2023-11-28T17:41:51.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.002+0000 7fbe7bc64640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbe741540d0 0x7fbe7415dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.003+0000 7fbe7bc64640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe740cb070 0x7fbe7406b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.003+0000 7fbe7bc64640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7406a660 con 0x7fbe740ca280 2023-11-28T17:41:51.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.003+0000 7fbe7bc64640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe740cbdd0 con 0x7fbe741540d0 2023-11-28T17:41:51.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.003+0000 7fbe7bc64640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe741544b0 con 0x7fbe740cb070 2023-11-28T17:41:51.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.003+0000 7fbe79a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbe741540d0 0x7fbe7415dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.003+0000 7fbe7a201640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbe740ca280 0x7fbe74159c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46848/0 (socket says 172.21.15.98:46848) 2023-11-28T17:41:51.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.003+0000 7fbe79a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbe741540d0 0x7fbe7415dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56856/0 (socket says 172.21.15.98:56856) 2023-11-28T17:41:51.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.003+0000 7fbe7a201640 1 -- 172.21.15.98:0/1991443101 learned_addr learned my addr 172.21.15.98:0/1991443101 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:51.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.003+0000 7fbe791ff640 1 --2- 172.21.15.98:0/1991443101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe740cb070 0x7fbe7406b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.003+0000 7fbe79a00640 1 -- 172.21.15.98:0/1991443101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe740cb070 msgr2=0x7fbe7406b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.003+0000 7fbe79a00640 1 --2- 172.21.15.98:0/1991443101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe740cb070 0x7fbe7406b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.003+0000 7fbe79a00640 1 -- 172.21.15.98:0/1991443101 >> v1:172.21.15.98:6789/0 conn(0x7fbe740ca280 legacy=0x7fbe74159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.003+0000 7fbe79a00640 1 -- 172.21.15.98:0/1991443101 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7415e1b0 con 0x7fbe741540d0 2023-11-28T17:41:51.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.004+0000 7fbe79a00640 1 --2- 172.21.15.98:0/1991443101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe741540d0 0x7fbe7415dc70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbe700097a0 tx=0x7fbe7002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6a63521d97c160 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.005+0000 7fbe789fe640 1 -- 172.21.15.98:0/1991443101 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7002d040 con 0x7fbe741540d0 2023-11-28T17:41:51.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.005+0000 7fbe789fe640 1 -- 172.21.15.98:0/1991443101 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe70031810 con 0x7fbe741540d0 2023-11-28T17:41:51.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.005+0000 7fbe789fe640 1 -- 172.21.15.98:0/1991443101 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70031ba0 con 0x7fbe741540d0 2023-11-28T17:41:51.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.005+0000 7fbe7bc64640 1 -- 172.21.15.98:0/1991443101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe741540d0 msgr2=0x7fbe7415dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.005+0000 7fbe7bc64640 1 --2- 172.21.15.98:0/1991443101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe741540d0 0x7fbe7415dc70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbe700097a0 tx=0x7fbe7002b8b0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.005+0000 7fbe7bc64640 1 -- 172.21.15.98:0/1991443101 shutdown_connections 2023-11-28T17:41:51.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.005+0000 7fbe7bc64640 1 --2- 172.21.15.98:0/1991443101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe740cb070 0x7fbe7406b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.005+0000 7fbe7bc64640 1 --2- 172.21.15.98:0/1991443101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe741540d0 0x7fbe7415dc70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.005+0000 7fbe7bc64640 1 -- 172.21.15.98:0/1991443101 >> 172.21.15.98:0/1991443101 conn(0x7fbe740cf490 msgr2=0x7fbe740cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.006+0000 7fbe7bc64640 1 -- 172.21.15.98:0/1991443101 shutdown_connections 2023-11-28T17:41:51.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.006+0000 7fbe7bc64640 1 -- 172.21.15.98:0/1991443101 wait complete. 2023-11-28T17:41:51.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.007+0000 7fbe7bc64640 1 Processor -- start 2023-11-28T17:41:51.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.007+0000 7fbe7bc64640 1 -- start start 2023-11-28T17:41:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.008+0000 7fbe7bc64640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe740ca280 0x7fbe741f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.008+0000 7fbe7bc64640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe740cb070 0x7fbe7415bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.008+0000 7fbe7bc64640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe741540d0 0x7fbe7414f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.008+0000 7fbe7bc64640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe741f3f50 con 0x7fbe741540d0 2023-11-28T17:41:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.008+0000 7fbe7bc64640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7415b270 con 0x7fbe740ca280 2023-11-28T17:41:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.008+0000 7fbe7bc64640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe741efde0 con 0x7fbe740cb070 2023-11-28T17:41:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.009+0000 7fbe7a201640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe741540d0 0x7fbe7414f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.009+0000 7fbe7a201640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe741540d0 0x7fbe7414f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56864/0 (socket says 172.21.15.98:56864) 2023-11-28T17:41:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.009+0000 7fbe7a201640 1 -- 172.21.15.98:0/3645063868 learned_addr learned my addr 172.21.15.98:0/3645063868 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.009+0000 7fbe79a00640 1 --2- 172.21.15.98:0/3645063868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe740ca280 0x7fbe741f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.009+0000 7fbe791ff640 1 --2- 172.21.15.98:0/3645063868 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe740cb070 0x7fbe7415bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.009+0000 7fbe7a201640 1 -- 172.21.15.98:0/3645063868 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe740cb070 msgr2=0x7fbe7415bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.009+0000 7fbe7a201640 1 --2- 172.21.15.98:0/3645063868 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe740cb070 0x7fbe7415bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.009+0000 7fbe7a201640 1 -- 172.21.15.98:0/3645063868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe740ca280 msgr2=0x7fbe741f3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.009+0000 7fbe7a201640 1 --2- 172.21.15.98:0/3645063868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe740ca280 0x7fbe741f3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.009+0000 7fbe7a201640 1 -- 172.21.15.98:0/3645063868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe70009400 con 0x7fbe741540d0 2023-11-28T17:41:51.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.009+0000 7fbe7a201640 1 --2- 172.21.15.98:0/3645063868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe741540d0 0x7fbe7414f480 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00ca70 tx=0x7fbe6c00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.010+0000 7fbe667fc640 1 -- 172.21.15.98:0/3645063868 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c00abd0 con 0x7fbe741540d0 2023-11-28T17:41:51.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.010+0000 7fbe667fc640 1 -- 172.21.15.98:0/3645063868 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6c00ad30 con 0x7fbe741540d0 2023-11-28T17:41:51.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.010+0000 7fbe667fc640 1 -- 172.21.15.98:0/3645063868 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c012600 con 0x7fbe741540d0 2023-11-28T17:41:51.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.010+0000 7fbe7bc64640 1 -- 172.21.15.98:0/3645063868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe741f00c0 con 0x7fbe741540d0 2023-11-28T17:41:51.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.010+0000 7fbe7bc64640 1 -- 172.21.15.98:0/3645063868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe741f0680 con 0x7fbe741540d0 2023-11-28T17:41:51.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.012+0000 7fbe667fc640 1 -- 172.21.15.98:0/3645063868 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fbe6c007070 con 0x7fbe741540d0 2023-11-28T17:41:51.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.013+0000 7fbe667fc640 1 --2- 172.21.15.98:0/3645063868 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbe5806b2d0 0x7fbe5806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.013+0000 7fbe667fc640 1 -- 172.21.15.98:0/3645063868 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe6c089a00 con 0x7fbe741540d0 2023-11-28T17:41:51.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.013+0000 7fbe79a00640 1 --2- 172.21.15.98:0/3645063868 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbe5806b2d0 0x7fbe5806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.014+0000 7fbe79a00640 1 --2- 172.21.15.98:0/3645063868 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbe5806b2d0 0x7fbe5806d790 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbe70036000 tx=0x7fbe7003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.014+0000 7fbe7e7fe740 1 Processor -- start 2023-11-28T17:41:51.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.014+0000 7fbe7e7fe740 1 -- start start 2023-11-28T17:41:51.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.015+0000 7fbe7e7fe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559065fd9890 0x559065fd9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.015+0000 7fbe7e7fe740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559066023710 0x559065fda1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.015+0000 7fbe7e7fe740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd82f0 0x559066019210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.015+0000 7fbe7e7fe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559065f38880 con 0x559065fd9890 2023-11-28T17:41:51.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.015+0000 7fbe7e7fe740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x559065f47ae0 con 0x559066023710 2023-11-28T17:41:51.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.015+0000 7fbe7e7fe740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x559065f451d0 con 0x559065fd82f0 2023-11-28T17:41:51.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.015+0000 7fbe79a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559065fd9890 0x559065fd9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.015+0000 7fbe79a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559065fd9890 0x559065fd9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56866/0 (socket says 172.21.15.98:56866) 2023-11-28T17:41:51.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.015+0000 7fbe79a00640 1 -- 172.21.15.98:0/2994104826 learned_addr learned my addr 172.21.15.98:0/2994104826 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.016+0000 7fbe791ff640 1 --2- 172.21.15.98:0/2994104826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559066023710 0x559065fda1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.016+0000 7fbe79a00640 1 -- 172.21.15.98:0/2994104826 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd82f0 msgr2=0x559066019210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.016+0000 7fbe79a00640 1 --2- 172.21.15.98:0/2994104826 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd82f0 0x559066019210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.016+0000 7fbe79a00640 1 -- 172.21.15.98:0/2994104826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559066023710 msgr2=0x559065fda1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.016+0000 7fbe79a00640 1 --2- 172.21.15.98:0/2994104826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559066023710 0x559065fda1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.016+0000 7fbe79a00640 1 -- 172.21.15.98:0/2994104826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559065f08860 con 0x559065fd9890 2023-11-28T17:41:51.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.016+0000 7fbe79a00640 1 --2- 172.21.15.98:0/2994104826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559065fd9890 0x559065fd9c70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbe700097a0 tx=0x7fbe70030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c97204001304871f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.016+0000 7fbe577fe640 1 -- 172.21.15.98:0/2994104826 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7003adb0 con 0x559065fd9890 2023-11-28T17:41:51.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.016+0000 7fbe577fe640 1 -- 172.21.15.98:0/2994104826 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7004f400 con 0x559065fd9890 2023-11-28T17:41:51.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.016+0000 7fbe577fe640 1 -- 172.21.15.98:0/2994104826 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7004f770 con 0x559065fd9890 2023-11-28T17:41:51.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.017+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/2994104826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559065fd9890 msgr2=0x559065fd9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.017+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/2994104826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559065fd9890 0x559065fd9c70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbe700097a0 tx=0x7fbe70030000 comp rx=0 tx=0).stop 2023-11-28T17:41:51.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.017+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/2994104826 shutdown_connections 2023-11-28T17:41:51.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.017+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/2994104826 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd82f0 0x559066019210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.017+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/2994104826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559066023710 0x559065fda1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.017+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/2994104826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559065fd9890 0x559065fd9c70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.017+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/2994104826 >> 172.21.15.98:0/2994104826 conn(0x559065fd6290 msgr2=0x559065fd8940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.017+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/2994104826 shutdown_connections 2023-11-28T17:41:51.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.017+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/2994104826 wait complete. 2023-11-28T17:41:51.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.018+0000 7fbe7e7fe740 1 -- start start 2023-11-28T17:41:51.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.018+0000 7fbe7e7fe740 1 Processor -- start 2023-11-28T17:41:51.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.019+0000 7fbe7e7fe740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd9890 0x5590660009d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.019+0000 7fbe7e7fe740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559066023710 0x5590660016c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.019+0000 7fbe7e7fe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906601d150 0x559066017460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.019+0000 7fbe7e7fe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559065f40ac0 con 0x55906601d150 2023-11-28T17:41:51.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.019+0000 7fbe7e7fe740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x559065edd2f0 con 0x559066023710 2023-11-28T17:41:51.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.019+0000 7fbe7e7fe740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x559065efac70 con 0x559065fd9890 2023-11-28T17:41:51.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.019+0000 7fbe7e7fe740 10 client.0 _ll_register_callbacks cb 0x559065c52ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:41:51.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.019+0000 7fbe7a201640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906601d150 0x559066017460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.019+0000 7fbe7a201640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906601d150 0x559066017460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56880/0 (socket says 172.21.15.98:56880) 2023-11-28T17:41:51.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.019+0000 7fbe7a201640 1 -- 172.21.15.98:0/2484551377 learned_addr learned my addr 172.21.15.98:0/2484551377 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.020+0000 7fbe7a201640 1 -- 172.21.15.98:0/2484551377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd9890 msgr2=0x5590660009d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.020+0000 7fbe7a201640 1 --2- 172.21.15.98:0/2484551377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd9890 0x5590660009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.020+0000 7fbe7a201640 1 -- 172.21.15.98:0/2484551377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559066023710 msgr2=0x5590660016c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.020+0000 7fbe7a201640 1 --2- 172.21.15.98:0/2484551377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559066023710 0x5590660016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.020+0000 7fbe7a201640 1 -- 172.21.15.98:0/2484551377 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7415e1b0 con 0x55906601d150 2023-11-28T17:41:51.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.020+0000 7fbe7a201640 1 --2- 172.21.15.98:0/2484551377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906601d150 0x559066017460 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c0012e0 tx=0x7fbe6c05c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ac2753fbe50ad1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.020+0000 7fbe54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-11-28T17:41:51.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.020+0000 7fbe54ff9640 1 -- 172.21.15.98:0/2484551377 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c05c370 con 0x55906601d150 2023-11-28T17:41:51.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.020+0000 7fbe54ff9640 1 -- 172.21.15.98:0/2484551377 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe6c057a40 con 0x55906601d150 2023-11-28T17:41:51.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.020+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/2484551377 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559065f39aa0 con 0x55906601d150 2023-11-28T17:41:51.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.020+0000 7fbe54ff9640 1 -- 172.21.15.98:0/2484551377 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe6c057c60 con 0x55906601d150 2023-11-28T17:41:51.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.020+0000 7fbe7e7fe740 10 client.4686 fetch_fsmap learned FSMap version 16 2023-11-28T17:41:51.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.020+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/2484551377 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x559065f3f140 con 0x55906601d150 2023-11-28T17:41:51.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.020+0000 7fbe54ff9640 1 -- 172.21.15.98:0/2484551377 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6c073080 con 0x55906601d150 2023-11-28T17:41:51.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.021+0000 7fbe54ff9640 1 -- 172.21.15.98:0/2484551377 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fbe6c0603a0 con 0x55906601d150 2023-11-28T17:41:51.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.021+0000 7fbe7e7fe740 10 client.4686 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T17:41:51.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.021+0000 7fbe7e7fe740 10 client.4686 resolve_mds: resolved name 'f' to daemon mds.f 2023-11-28T17:41:51.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.021+0000 7fbe7e7fe740 20 client.4686 populate_metadata read hostname 'smithi098' 2023-11-28T17:41:51.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.021+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/2484551377 >> [v2:172.21.15.113:6836/1732613907,v1:172.21.15.113:6837/1732613907] conn(0x5590660247e0 0x559065ffd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.021+0000 7fbe7e7fe740 4 client.4686 mds_command: new command op to 4272 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T17:41:51.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.021+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/2484551377 --> [v2:172.21.15.113:6836/1732613907,v1:172.21.15.113:6837/1732613907] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559065f397c0 con 0x5590660247e0 2023-11-28T17:41:51.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.021+0000 7fbe79a00640 1 --2- 172.21.15.98:0/2484551377 >> [v2:172.21.15.113:6836/1732613907,v1:172.21.15.113:6837/1732613907] conn(0x5590660247e0 0x559065ffd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.022+0000 7fbe79a00640 1 --2- 172.21.15.98:0/2484551377 >> [v2:172.21.15.113:6836/1732613907,v1:172.21.15.113:6837/1732613907] conn(0x5590660247e0 0x559065ffd650 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=320141a998f76e16 server_cookie=e3e0d70fed0ed4dc in_seq=0 out_seq=0 2023-11-28T17:41:51.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.022+0000 7fbe54ff9640 10 client.4686 ms_handle_connect on v2:172.21.15.113:6836/1732613907 2023-11-28T17:41:51.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.024+0000 7fbe54ff9640 1 -- 172.21.15.98:0/2484551377 <== mds.2 v2:172.21.15.113:6836/1732613907 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559065f397c0 con 0x5590660247e0 2023-11-28T17:41:51.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.024+0000 7fbe54ff9640 10 client.4686 handle_command_reply: tid=0 2023-11-28T17:41:51.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.024+0000 7fbe7e7fe740 1 client.4686 shutdown 2023-11-28T17:41:51.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.024+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/2484551377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906601d150 msgr2=0x559066017460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.024+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/2484551377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906601d150 0x559066017460 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c0012e0 tx=0x7fbe6c05c6b0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.024+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/2484551377 shutdown_connections 2023-11-28T17:41:51.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.024+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/2484551377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906601d150 0x559066017460 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.024+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/2484551377 >> [v2:172.21.15.113:6836/1732613907,v1:172.21.15.113:6837/1732613907] conn(0x5590660247e0 0x559065ffd650 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.024+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/2484551377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559066023710 0x5590660016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.024+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/2484551377 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd9890 0x5590660009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.024+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/2484551377 >> 172.21.15.98:0/2484551377 conn(0x559065fd82f0 msgr2=0x559065fd31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.024+0000 7fbe54ff9640 0 client.4686 ms_handle_reset on v2:172.21.15.113:6836/1732613907 2023-11-28T17:41:51.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.025+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/2484551377 shutdown_connections 2023-11-28T17:41:51.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.025+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/2484551377 wait complete. 2023-11-28T17:41:51.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.025+0000 7fbe7e7fe740 20 client.4686 trim_cache size 0 max 16384 2023-11-28T17:41:51.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.042+0000 7fbe7e7fe740 1 Processor -- start 2023-11-28T17:41:51.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.042+0000 7fbe7e7fe740 1 -- start start 2023-11-28T17:41:51.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.043+0000 7fbe7e7fe740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd9890 0x559065ffde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.043+0000 7fbe7e7fe740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55906601d150 0x559065ffe390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.043+0000 7fbe7e7fe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559066023710 0x559066002860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.043+0000 7fbe7e7fe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559065ca3f20 con 0x559066023710 2023-11-28T17:41:51.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.043+0000 7fbe7e7fe740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x559065ffcd20 con 0x55906601d150 2023-11-28T17:41:51.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.043+0000 7fbe7e7fe740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x559066016c50 con 0x559065fd9890 2023-11-28T17:41:51.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.043+0000 7fbe7a201640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559066023710 0x559066002860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.043+0000 7fbe7a201640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559066023710 0x559066002860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56892/0 (socket says 172.21.15.98:56892) 2023-11-28T17:41:51.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.043+0000 7fbe7a201640 1 -- 172.21.15.98:0/3041996330 learned_addr learned my addr 172.21.15.98:0/3041996330 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.043+0000 7fbe791ff640 1 --2- 172.21.15.98:0/3041996330 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55906601d150 0x559065ffe390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.043+0000 7fbe79a00640 1 --2- 172.21.15.98:0/3041996330 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd9890 0x559065ffde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.044+0000 7fbe7a201640 1 -- 172.21.15.98:0/3041996330 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd9890 msgr2=0x559065ffde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.044+0000 7fbe7a201640 1 --2- 172.21.15.98:0/3041996330 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd9890 0x559065ffde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.044+0000 7fbe7a201640 1 -- 172.21.15.98:0/3041996330 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55906601d150 msgr2=0x559065ffe390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.044+0000 7fbe7a201640 1 --2- 172.21.15.98:0/3041996330 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55906601d150 0x559065ffe390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.044+0000 7fbe7a201640 1 -- 172.21.15.98:0/3041996330 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559065edcb80 con 0x559066023710 2023-11-28T17:41:51.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.044+0000 7fbe79a00640 1 --2- 172.21.15.98:0/3041996330 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd9890 0x559065ffde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:51.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.044+0000 7fbe7a201640 1 --2- 172.21.15.98:0/3041996330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559066023710 0x559066002860 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c002230 tx=0x7fbe6c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0487f02adf653f2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.044+0000 7fbe57fff640 1 -- 172.21.15.98:0/3041996330 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c0603a0 con 0x559066023710 2023-11-28T17:41:51.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.044+0000 7fbe57fff640 1 -- 172.21.15.98:0/3041996330 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559065f39aa0 con 0x559066023710 2023-11-28T17:41:51.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.044+0000 7fbe57fff640 1 -- 172.21.15.98:0/3041996330 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c05c3b0 con 0x559066023710 2023-11-28T17:41:51.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.045+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/3041996330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559066023710 msgr2=0x559066002860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.045+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/3041996330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559066023710 0x559066002860 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c002230 tx=0x7fbe6c013000 comp rx=0 tx=0).stop 2023-11-28T17:41:51.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.045+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/3041996330 shutdown_connections 2023-11-28T17:41:51.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.045+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/3041996330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559066023710 0x559066002860 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.045+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/3041996330 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55906601d150 0x559065ffe390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.045+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/3041996330 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd9890 0x559065ffde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.045+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/3041996330 >> 172.21.15.98:0/3041996330 conn(0x559065fd82f0 msgr2=0x5590660013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.045+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/3041996330 shutdown_connections 2023-11-28T17:41:51.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.045+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/3041996330 wait complete. 2023-11-28T17:41:51.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.045+0000 7fbe7e7fe740 1 -- start start 2023-11-28T17:41:51.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.046+0000 7fbe7e7fe740 1 Processor -- start 2023-11-28T17:41:51.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe7e7fe740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd9890 0x559066002c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe7e7fe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906601d150 0x55906601e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe7e7fe740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559066023710 0x55906601ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe7e7fe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559065fd7600 con 0x55906601d150 2023-11-28T17:41:51.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe7e7fe740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x559066003180 con 0x559066023710 2023-11-28T17:41:51.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe7e7fe740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x559066000ee0 con 0x559065fd9890 2023-11-28T17:41:51.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe7e7fe740 10 client.0 _ll_register_callbacks cb 0x7fbe6c001bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:41:51.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe791ff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906601d150 0x55906601e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe791ff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906601d150 0x55906601e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56894/0 (socket says 172.21.15.98:56894) 2023-11-28T17:41:51.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe791ff640 1 -- 172.21.15.98:0/3199118000 learned_addr learned my addr 172.21.15.98:0/3199118000 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe7a201640 1 --2- 172.21.15.98:0/3199118000 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559066023710 0x55906601ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe79a00640 1 --2- 172.21.15.98:0/3199118000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd9890 0x559066002c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe791ff640 1 -- 172.21.15.98:0/3199118000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd9890 msgr2=0x559066002c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe791ff640 1 --2- 172.21.15.98:0/3199118000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd9890 0x559066002c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe791ff640 1 -- 172.21.15.98:0/3199118000 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559066023710 msgr2=0x55906601ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe791ff640 1 --2- 172.21.15.98:0/3199118000 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559066023710 0x55906601ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.047+0000 7fbe791ff640 1 -- 172.21.15.98:0/3199118000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559065f3f140 con 0x55906601d150 2023-11-28T17:41:51.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.048+0000 7fbe791ff640 1 --2- 172.21.15.98:0/3199118000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906601d150 0x55906601e6f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbe74150e30 tx=0x7fbe68012640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c4ee5f26e7ae478 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.048+0000 7fbe54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-11-28T17:41:51.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.048+0000 7fbe54ff9640 1 -- 172.21.15.98:0/3199118000 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe68015600 con 0x55906601d150 2023-11-28T17:41:51.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.048+0000 7fbe54ff9640 1 -- 172.21.15.98:0/3199118000 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe68015760 con 0x55906601d150 2023-11-28T17:41:51.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.048+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/3199118000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe7004f400 con 0x55906601d150 2023-11-28T17:41:51.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.048+0000 7fbe54ff9640 1 -- 172.21.15.98:0/3199118000 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe68015920 con 0x55906601d150 2023-11-28T17:41:51.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.048+0000 7fbe7e7fe740 10 client.4692 fetch_fsmap learned FSMap version 16 2023-11-28T17:41:51.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.048+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/3199118000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x559065f02170 con 0x55906601d150 2023-11-28T17:41:51.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.048+0000 7fbe54ff9640 1 -- 172.21.15.98:0/3199118000 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7004f400 con 0x55906601d150 2023-11-28T17:41:51.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.049+0000 7fbe54ff9640 1 -- 172.21.15.98:0/3199118000 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fbe68015a80 con 0x55906601d150 2023-11-28T17:41:51.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.049+0000 7fbe7e7fe740 10 client.4692 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T17:41:51.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.049+0000 7fbe7e7fe740 10 client.4692 resolve_mds: resolved name 'f' to daemon mds.f 2023-11-28T17:41:51.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.049+0000 7fbe7e7fe740 20 client.4692 populate_metadata read hostname 'smithi098' 2023-11-28T17:41:51.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.049+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/3199118000 >> [v2:172.21.15.113:6836/1732613907,v1:172.21.15.113:6837/1732613907] conn(0x559065ff6430 0x55906607e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.049+0000 7fbe7e7fe740 4 client.4692 mds_command: new command op to 4272 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T17:41:51.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.049+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/3199118000 --> [v2:172.21.15.113:6836/1732613907,v1:172.21.15.113:6837/1732613907] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559065c416c0 con 0x559065ff6430 2023-11-28T17:41:51.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.049+0000 7fbe79a00640 1 --2- 172.21.15.98:0/3199118000 >> [v2:172.21.15.113:6836/1732613907,v1:172.21.15.113:6837/1732613907] conn(0x559065ff6430 0x55906607e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.050+0000 7fbe79a00640 1 --2- 172.21.15.98:0/3199118000 >> [v2:172.21.15.113:6836/1732613907,v1:172.21.15.113:6837/1732613907] conn(0x559065ff6430 0x55906607e640 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c1d96a12ca23782d server_cookie=c64b0d43b97ebbe6 in_seq=0 out_seq=0 2023-11-28T17:41:51.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.050+0000 7fbe54ff9640 10 client.4692 ms_handle_connect on v2:172.21.15.113:6836/1732613907 2023-11-28T17:41:51.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.051+0000 7fbe54ff9640 1 -- 172.21.15.98:0/3199118000 <== mds.2 v2:172.21.15.113:6836/1732613907 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x559065c416c0 con 0x559065ff6430 2023-11-28T17:41:51.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.051+0000 7fbe54ff9640 10 client.4692 handle_command_reply: tid=0 2023-11-28T17:41:51.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.051+0000 7fbe7e7fe740 1 client.4692 shutdown 2023-11-28T17:41:51.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.051+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/3199118000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906601d150 msgr2=0x55906601e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.051+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/3199118000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906601d150 0x55906601e6f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbe74150e30 tx=0x7fbe68012640 comp rx=0 tx=0).stop 2023-11-28T17:41:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.051+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/3199118000 shutdown_connections 2023-11-28T17:41:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.051+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/3199118000 >> [v2:172.21.15.113:6836/1732613907,v1:172.21.15.113:6837/1732613907] conn(0x559065ff6430 0x55906607e640 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.051+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/3199118000 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559066023710 0x55906601ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.051+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/3199118000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55906601d150 0x55906601e6f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.051+0000 7fbe54ff9640 0 client.4692 ms_handle_reset on v2:172.21.15.113:6836/1732613907 2023-11-28T17:41:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.051+0000 7fbe7e7fe740 1 --2- 172.21.15.98:0/3199118000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x559065fd9890 0x559066002c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.051+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/3199118000 >> 172.21.15.98:0/3199118000 conn(0x559065fd82f0 msgr2=0x559065ffde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.051+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/3199118000 shutdown_connections 2023-11-28T17:41:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.051+0000 7fbe7e7fe740 1 -- 172.21.15.98:0/3199118000 wait complete. 2023-11-28T17:41:51.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.051+0000 7fbe7e7fe740 20 client.4692 trim_cache size 0 max 16384 2023-11-28T17:41:51.058 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:41:51.058 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:41:51.058 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2023-11-28T17:41:51.058 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4272, 2023-11-28T17:41:51.058 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-11-28T17:41:51.058 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-11-28T17:41:51.058 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-11-28T17:41:51.058 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 22.839717793999998, 2023-11-28T17:41:51.058 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 16, 2023-11-28T17:41:51.059 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 22, 2023-11-28T17:41:51.059 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 22, 2023-11-28T17:41:51.059 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 41.162499662000002 2023-11-28T17:41:51.059 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:41:51.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.052+0000 7fbe7bc64640 1 -- 172.21.15.98:0/3645063868 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbe5806b2d0 msgr2=0x7fbe5806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.052+0000 7fbe7bc64640 1 --2- 172.21.15.98:0/3645063868 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbe5806b2d0 0x7fbe5806d790 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbe70036000 tx=0x7fbe7003c040 comp rx=0 tx=0).stop 2023-11-28T17:41:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.052+0000 7fbe7bc64640 1 -- 172.21.15.98:0/3645063868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe741540d0 msgr2=0x7fbe7414f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.052+0000 7fbe7bc64640 1 --2- 172.21.15.98:0/3645063868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe741540d0 0x7fbe7414f480 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00ca70 tx=0x7fbe6c00cf30 comp rx=0 tx=0).stop 2023-11-28T17:41:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.052+0000 7fbe7bc64640 1 -- 172.21.15.98:0/3645063868 shutdown_connections 2023-11-28T17:41:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.052+0000 7fbe7bc64640 1 --2- 172.21.15.98:0/3645063868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe741540d0 0x7fbe7414f480 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.052+0000 7fbe7bc64640 1 --2- 172.21.15.98:0/3645063868 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbe740cb070 0x7fbe7415bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.052+0000 7fbe7bc64640 1 --2- 172.21.15.98:0/3645063868 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbe5806b2d0 0x7fbe5806d790 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.052+0000 7fbe7bc64640 1 --2- 172.21.15.98:0/3645063868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe740ca280 0x7fbe741f3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.052+0000 7fbe7bc64640 1 -- 172.21.15.98:0/3645063868 >> 172.21.15.98:0/3645063868 conn(0x7fbe740cf490 msgr2=0x7fbe740cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.052+0000 7fbe7bc64640 1 -- 172.21.15.98:0/3645063868 shutdown_connections 2023-11-28T17:41:51.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.052+0000 7fbe7bc64640 1 -- 172.21.15.98:0/3645063868 wait complete. 2023-11-28T17:41:51.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-11-28T17:41:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.266+0000 7f8734ce7640 1 Processor -- start 2023-11-28T17:41:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.266+0000 7f8734ce7640 1 -- start start 2023-11-28T17:41:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.267+0000 7f8734ce7640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8730154030 0x7f8730070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.267+0000 7f8734ce7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f873006dcf0 0x7f873006e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.267+0000 7f8734ce7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8730154410 con 0x7f87300c8600 2023-11-28T17:41:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.267+0000 7f8734ce7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87300715a0 con 0x7f873006dcf0 2023-11-28T17:41:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.267+0000 7f8734ce7640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87300c5da0 con 0x7f8730154030 2023-11-28T17:41:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.268+0000 7f872ed9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f873006dcf0 0x7f873006e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.268+0000 7f872ed9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f873006dcf0 0x7f873006e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56910/0 (socket says 172.21.15.98:56910) 2023-11-28T17:41:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.268+0000 7f8727fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f87300c8600 0x7f8730070e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46858/0 (socket says 172.21.15.98:46858) 2023-11-28T17:41:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.268+0000 7f872ed9d640 1 -- 172.21.15.98:0/678690053 learned_addr learned my addr 172.21.15.98:0/678690053 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.268+0000 7f872e59c640 1 --2- 172.21.15.98:0/678690053 >> v2:172.21.15.113:3300/0 conn(0x7f8730154030 0x7f8730070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.268+0000 7f872ed9d640 1 -- 172.21.15.98:0/678690053 >> v2:172.21.15.113:3300/0 conn(0x7f8730154030 msgr2=0x7f8730070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.268+0000 7f872ed9d640 1 --2- 172.21.15.98:0/678690053 >> v2:172.21.15.113:3300/0 conn(0x7f8730154030 0x7f8730070660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.268+0000 7f872ed9d640 1 -- 172.21.15.98:0/678690053 >> v1:172.21.15.98:6789/0 conn(0x7f87300c8600 legacy=0x7f8730070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.268+0000 7f872ed9d640 1 -- 172.21.15.98:0/678690053 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873015d370 con 0x7f873006dcf0 2023-11-28T17:41:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.269+0000 7f872ed9d640 1 --2- 172.21.15.98:0/678690053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f873006dcf0 0x7f873006e120 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f873006eba0 tx=0x7f872802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ada4496d45dc2a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.270+0000 7f872dd9b640 1 -- 172.21.15.98:0/678690053 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872802cbb0 con 0x7f873006dcf0 2023-11-28T17:41:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.270+0000 7f872dd9b640 1 -- 172.21.15.98:0/678690053 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872802cd10 con 0x7f873006dcf0 2023-11-28T17:41:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.270+0000 7f872dd9b640 1 -- 172.21.15.98:0/678690053 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8728034610 con 0x7f873006dcf0 2023-11-28T17:41:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.270+0000 7f8734ce7640 1 -- 172.21.15.98:0/678690053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f873006dcf0 msgr2=0x7f873006e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.270+0000 7f8734ce7640 1 --2- 172.21.15.98:0/678690053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f873006dcf0 0x7f873006e120 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f873006eba0 tx=0x7f872802c010 comp rx=0 tx=0).stop 2023-11-28T17:41:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.270+0000 7f8734ce7640 1 -- 172.21.15.98:0/678690053 shutdown_connections 2023-11-28T17:41:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.270+0000 7f8734ce7640 1 --2- 172.21.15.98:0/678690053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f873006dcf0 0x7f873006e120 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.270+0000 7f8734ce7640 1 --2- 172.21.15.98:0/678690053 >> v2:172.21.15.113:3300/0 conn(0x7f8730154030 0x7f8730070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.270+0000 7f8734ce7640 1 -- 172.21.15.98:0/678690053 >> 172.21.15.98:0/678690053 conn(0x7f873006c0f0 msgr2=0x7f873006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.270+0000 7f8734ce7640 1 -- 172.21.15.98:0/678690053 shutdown_connections 2023-11-28T17:41:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.270+0000 7f8734ce7640 1 -- 172.21.15.98:0/678690053 wait complete. 2023-11-28T17:41:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.272+0000 7f8734ce7640 1 Processor -- start 2023-11-28T17:41:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.272+0000 7f8734ce7640 1 -- start start 2023-11-28T17:41:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.273+0000 7f8734ce7640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f873006dcf0 0x7f8730143370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.273+0000 7f8734ce7640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87300c8600 0x7f873014daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.273+0000 7f8734ce7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8730154030 0x7f87301588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.273+0000 7f8734ce7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87301522d0 con 0x7f8730154030 2023-11-28T17:41:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.273+0000 7f8734ce7640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87301f0020 con 0x7f873006dcf0 2023-11-28T17:41:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.273+0000 7f8734ce7640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87301e70b0 con 0x7f87300c8600 2023-11-28T17:41:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.274+0000 7f872ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8730154030 0x7f87301588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.274+0000 7f872ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8730154030 0x7f87301588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56914/0 (socket says 172.21.15.98:56914) 2023-11-28T17:41:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.274+0000 7f872ed9d640 1 -- 172.21.15.98:0/2935999446 learned_addr learned my addr 172.21.15.98:0/2935999446 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.274+0000 7f872ed9d640 1 -- 172.21.15.98:0/2935999446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87300c8600 msgr2=0x7f873014daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.274+0000 7f872ed9d640 1 --2- 172.21.15.98:0/2935999446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87300c8600 0x7f873014daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.274+0000 7f872ed9d640 1 -- 172.21.15.98:0/2935999446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f873006dcf0 msgr2=0x7f8730143370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.274+0000 7f872e59c640 1 --2- 172.21.15.98:0/2935999446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f873006dcf0 0x7f8730143370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.274+0000 7f872ed9d640 1 --2- 172.21.15.98:0/2935999446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f873006dcf0 0x7f8730143370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.274+0000 7f872ed9d640 1 -- 172.21.15.98:0/2935999446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8728009b30 con 0x7f8730154030 2023-11-28T17:41:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.274+0000 7f872ed9d640 1 --2- 172.21.15.98:0/2935999446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8730154030 0x7f87301588d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8728036040 tx=0x7f8728004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.274+0000 7f8726ffd640 1 -- 172.21.15.98:0/2935999446 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8728009510 con 0x7f8730154030 2023-11-28T17:41:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.275+0000 7f8726ffd640 1 -- 172.21.15.98:0/2935999446 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872803e450 con 0x7f8730154030 2023-11-28T17:41:51.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.275+0000 7f8726ffd640 1 -- 172.21.15.98:0/2935999446 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872803e800 con 0x7f8730154030 2023-11-28T17:41:51.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.275+0000 7f8734ce7640 1 -- 172.21.15.98:0/2935999446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8730159790 con 0x7f8730154030 2023-11-28T17:41:51.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.275+0000 7f8734ce7640 1 -- 172.21.15.98:0/2935999446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87301ef6f0 con 0x7f8730154030 2023-11-28T17:41:51.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.277+0000 7f8726ffd640 1 -- 172.21.15.98:0/2935999446 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f872803e960 con 0x7f8730154030 2023-11-28T17:41:51.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.278+0000 7f8726ffd640 1 --2- 172.21.15.98:0/2935999446 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f871006b2a0 0x7f871006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.278+0000 7f8726ffd640 1 -- 172.21.15.98:0/2935999446 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87280aca40 con 0x7f8730154030 2023-11-28T17:41:51.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.278+0000 7f872e59c640 1 --2- 172.21.15.98:0/2935999446 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f871006b2a0 0x7f871006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.279+0000 7f872e59c640 1 --2- 172.21.15.98:0/2935999446 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f871006b2a0 0x7f871006d760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8720006fd0 tx=0x7f8720008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.279+0000 7f8737884740 1 Processor -- start 2023-11-28T17:41:51.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.279+0000 7f8737884740 1 -- start start 2023-11-28T17:41:51.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f8737884740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb306890 0x55c5eb306c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f8737884740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb350710 0x55c5eb3071b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f8737884740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb3052f0 0x55c5eb346210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f8737884740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5eb265880 con 0x55c5eb350710 2023-11-28T17:41:51.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f8737884740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5eb274ae0 con 0x55c5eb306890 2023-11-28T17:41:51.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f8737884740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5eb2721d0 con 0x55c5eb3052f0 2023-11-28T17:41:51.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f8727fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb350710 0x55c5eb3071b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f8727fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb350710 0x55c5eb3071b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56916/0 (socket says 172.21.15.98:56916) 2023-11-28T17:41:51.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f8727fff640 1 -- 172.21.15.98:0/4197345596 learned_addr learned my addr 172.21.15.98:0/4197345596 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f872e59c640 1 --2- 172.21.15.98:0/4197345596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb306890 0x55c5eb306c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f872ed9d640 1 --2- 172.21.15.98:0/4197345596 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb3052f0 0x55c5eb346210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f8727fff640 1 -- 172.21.15.98:0/4197345596 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb3052f0 msgr2=0x55c5eb346210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f8727fff640 1 --2- 172.21.15.98:0/4197345596 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb3052f0 0x55c5eb346210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f8727fff640 1 -- 172.21.15.98:0/4197345596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb306890 msgr2=0x55c5eb306c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f8727fff640 1 --2- 172.21.15.98:0/4197345596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb306890 0x55c5eb306c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f8727fff640 1 -- 172.21.15.98:0/4197345596 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5eb235860 con 0x55c5eb350710 2023-11-28T17:41:51.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.280+0000 7f872ed9d640 1 --2- 172.21.15.98:0/4197345596 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb3052f0 0x55c5eb346210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:51.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.281+0000 7f8727fff640 1 --2- 172.21.15.98:0/4197345596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb350710 0x55c5eb3071b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8718001460 tx=0x7f8718001920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5991c8937e6c32 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.281+0000 7f870ffff640 1 -- 172.21.15.98:0/4197345596 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87180074e0 con 0x55c5eb350710 2023-11-28T17:41:51.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.281+0000 7f870ffff640 1 -- 172.21.15.98:0/4197345596 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8718007640 con 0x55c5eb350710 2023-11-28T17:41:51.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.281+0000 7f870ffff640 1 -- 172.21.15.98:0/4197345596 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8718004a20 con 0x55c5eb350710 2023-11-28T17:41:51.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.281+0000 7f8737884740 1 -- 172.21.15.98:0/4197345596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb350710 msgr2=0x55c5eb3071b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.281+0000 7f8737884740 1 --2- 172.21.15.98:0/4197345596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb350710 0x55c5eb3071b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8718001460 tx=0x7f8718001920 comp rx=0 tx=0).stop 2023-11-28T17:41:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.282+0000 7f8737884740 1 -- 172.21.15.98:0/4197345596 shutdown_connections 2023-11-28T17:41:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.282+0000 7f8737884740 1 --2- 172.21.15.98:0/4197345596 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb3052f0 0x55c5eb346210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.282+0000 7f8737884740 1 --2- 172.21.15.98:0/4197345596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb350710 0x55c5eb3071b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.282+0000 7f8737884740 1 --2- 172.21.15.98:0/4197345596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb306890 0x55c5eb306c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.282+0000 7f8737884740 1 -- 172.21.15.98:0/4197345596 >> 172.21.15.98:0/4197345596 conn(0x55c5eb303290 msgr2=0x55c5eb305940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.282+0000 7f8737884740 1 -- 172.21.15.98:0/4197345596 shutdown_connections 2023-11-28T17:41:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.282+0000 7f8737884740 1 -- 172.21.15.98:0/4197345596 wait complete. 2023-11-28T17:41:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.282+0000 7f8737884740 1 -- start start 2023-11-28T17:41:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.283+0000 7f8737884740 1 Processor -- start 2023-11-28T17:41:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.283+0000 7f8737884740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb306890 0x55c5eb34f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f8737884740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb350710 0x55c5eb342be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f8737884740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb343b90 0x55c5eb32a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f8737884740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5eb26dac0 con 0x55c5eb306890 2023-11-28T17:41:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f8737884740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5eb20a2f0 con 0x55c5eb350710 2023-11-28T17:41:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f8737884740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5eb227c70 con 0x55c5eb343b90 2023-11-28T17:41:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f8737884740 10 client.0 _ll_register_callbacks cb 0x55c5eaf7fac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:41:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f872e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb306890 0x55c5eb34f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f872e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb306890 0x55c5eb34f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56926/0 (socket says 172.21.15.98:56926) 2023-11-28T17:41:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f872e59c640 1 -- 172.21.15.98:0/2877195630 learned_addr learned my addr 172.21.15.98:0/2877195630 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f8727fff640 1 --2- 172.21.15.98:0/2877195630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb350710 0x55c5eb342be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f872ed9d640 1 --2- 172.21.15.98:0/2877195630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb343b90 0x55c5eb32a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f872e59c640 1 -- 172.21.15.98:0/2877195630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb343b90 msgr2=0x55c5eb32a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f872e59c640 1 --2- 172.21.15.98:0/2877195630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb343b90 0x55c5eb32a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f872e59c640 1 -- 172.21.15.98:0/2877195630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb350710 msgr2=0x55c5eb342be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f872e59c640 1 --2- 172.21.15.98:0/2877195630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb350710 0x55c5eb342be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f872e59c640 1 -- 172.21.15.98:0/2877195630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8730154410 con 0x55c5eb306890 2023-11-28T17:41:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f872ed9d640 1 --2- 172.21.15.98:0/2877195630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb343b90 0x55c5eb32a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.284+0000 7f872e59c640 1 --2- 172.21.15.98:0/2877195630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb306890 0x55c5eb34f1f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f872000eea0 tx=0x7f8720004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192c737a85f16c62 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.285+0000 7f870d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-11-28T17:41:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.285+0000 7f870d7fa640 1 -- 172.21.15.98:0/2877195630 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8720015c90 con 0x55c5eb306890 2023-11-28T17:41:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.285+0000 7f870d7fa640 1 -- 172.21.15.98:0/2877195630 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8720015df0 con 0x55c5eb306890 2023-11-28T17:41:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.285+0000 7f8737884740 1 -- 172.21.15.98:0/2877195630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c5eb266aa0 con 0x55c5eb306890 2023-11-28T17:41:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.285+0000 7f870d7fa640 1 -- 172.21.15.98:0/2877195630 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8720021420 con 0x55c5eb306890 2023-11-28T17:41:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.285+0000 7f8737884740 10 client.4704 fetch_fsmap learned FSMap version 16 2023-11-28T17:41:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.285+0000 7f8737884740 1 -- 172.21.15.98:0/2877195630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55c5eb26c140 con 0x55c5eb306890 2023-11-28T17:41:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.285+0000 7f870d7fa640 1 -- 172.21.15.98:0/2877195630 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c5eb266aa0 con 0x55c5eb306890 2023-11-28T17:41:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.285+0000 7f870d7fa640 1 -- 172.21.15.98:0/2877195630 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f8720021580 con 0x55c5eb306890 2023-11-28T17:41:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.285+0000 7f8737884740 10 client.4704 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T17:41:51.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.285+0000 7f8737884740 10 client.4704 resolve_mds: resolved name 'e' to daemon mds.e 2023-11-28T17:41:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.285+0000 7f8737884740 20 client.4704 populate_metadata read hostname 'smithi098' 2023-11-28T17:41:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.285+0000 7f8737884740 1 --2- 172.21.15.98:0/2877195630 >> [v2:172.21.15.98:6838/3585869153,v1:172.21.15.98:6839/3585869153] conn(0x55c5eb35b0f0 0x55c5eb365f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.285+0000 7f8737884740 4 client.4704 mds_command: new command op to 4273 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T17:41:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.285+0000 7f8737884740 1 -- 172.21.15.98:0/2877195630 --> [v2:172.21.15.98:6838/3585869153,v1:172.21.15.98:6839/3585869153] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c5eb2667c0 con 0x55c5eb35b0f0 2023-11-28T17:41:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.286+0000 7f8727fff640 1 --2- 172.21.15.98:0/2877195630 >> [v2:172.21.15.98:6838/3585869153,v1:172.21.15.98:6839/3585869153] conn(0x55c5eb35b0f0 0x55c5eb365f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.286+0000 7f8727fff640 1 --2- 172.21.15.98:0/2877195630 >> [v2:172.21.15.98:6838/3585869153,v1:172.21.15.98:6839/3585869153] conn(0x55c5eb35b0f0 0x55c5eb365f40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=baea685dfb2d1217 server_cookie=b01511722e0722c2 in_seq=0 out_seq=0 2023-11-28T17:41:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.286+0000 7f870d7fa640 10 client.4704 ms_handle_connect on v2:172.21.15.98:6838/3585869153 2023-11-28T17:41:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.287+0000 7f870d7fa640 1 -- 172.21.15.98:0/2877195630 <== mds.1 v2:172.21.15.98:6838/3585869153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c5eb2667c0 con 0x55c5eb35b0f0 2023-11-28T17:41:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.287+0000 7f870d7fa640 10 client.4704 handle_command_reply: tid=0 2023-11-28T17:41:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.287+0000 7f8737884740 1 client.4704 shutdown 2023-11-28T17:41:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.287+0000 7f8737884740 1 -- 172.21.15.98:0/2877195630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb306890 msgr2=0x55c5eb34f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.287+0000 7f8737884740 1 --2- 172.21.15.98:0/2877195630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb306890 0x55c5eb34f1f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f872000eea0 tx=0x7f8720004730 comp rx=0 tx=0).stop 2023-11-28T17:41:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.288+0000 7f8737884740 1 -- 172.21.15.98:0/2877195630 shutdown_connections 2023-11-28T17:41:51.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.288+0000 7f8737884740 1 --2- 172.21.15.98:0/2877195630 >> [v2:172.21.15.98:6838/3585869153,v1:172.21.15.98:6839/3585869153] conn(0x55c5eb35b0f0 0x55c5eb365f40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.288+0000 7f8737884740 1 --2- 172.21.15.98:0/2877195630 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb343b90 0x55c5eb32a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.288+0000 7f8737884740 1 --2- 172.21.15.98:0/2877195630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb350710 0x55c5eb342be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.288+0000 7f8737884740 1 --2- 172.21.15.98:0/2877195630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb306890 0x55c5eb34f1f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.288+0000 7f870d7fa640 0 client.4704 ms_handle_reset on v2:172.21.15.98:6838/3585869153 2023-11-28T17:41:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.288+0000 7f8737884740 1 -- 172.21.15.98:0/2877195630 >> 172.21.15.98:0/2877195630 conn(0x55c5eb3052f0 msgr2=0x55c5eb3001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.288+0000 7f8737884740 1 -- 172.21.15.98:0/2877195630 shutdown_connections 2023-11-28T17:41:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.288+0000 7f8737884740 1 -- 172.21.15.98:0/2877195630 wait complete. 2023-11-28T17:41:51.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.288+0000 7f8737884740 20 client.4704 trim_cache size 0 max 16384 2023-11-28T17:41:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.305+0000 7f8737884740 1 Processor -- start 2023-11-28T17:41:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.305+0000 7f8737884740 1 -- start start 2023-11-28T17:41:51.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.306+0000 7f8737884740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb306890 0x55c5eb305940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.306+0000 7f8737884740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb343b90 0x55c5eb305e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.306+0000 7f8737884740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb350710 0x55c5eb329750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.306+0000 7f8737884740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5eafd0f20 con 0x55c5eb343b90 2023-11-28T17:41:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.306+0000 7f8737884740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5eb308430 con 0x55c5eb306890 2023-11-28T17:41:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.306+0000 7f8737884740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5eb303a00 con 0x55c5eb350710 2023-11-28T17:41:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.306+0000 7f8727fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb343b90 0x55c5eb305e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.306+0000 7f8727fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb343b90 0x55c5eb305e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56932/0 (socket says 172.21.15.98:56932) 2023-11-28T17:41:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.306+0000 7f8727fff640 1 -- 172.21.15.98:0/2007052090 learned_addr learned my addr 172.21.15.98:0/2007052090 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.306+0000 7f8727fff640 1 -- 172.21.15.98:0/2007052090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb350710 msgr2=0x55c5eb329750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.306+0000 7f8727fff640 1 --2- 172.21.15.98:0/2007052090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb350710 0x55c5eb329750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.306+0000 7f8727fff640 1 -- 172.21.15.98:0/2007052090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb306890 msgr2=0x55c5eb305940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.306+0000 7f8727fff640 1 --2- 172.21.15.98:0/2007052090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb306890 0x55c5eb305940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.306+0000 7f8727fff640 1 -- 172.21.15.98:0/2007052090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5eb209b80 con 0x55c5eb343b90 2023-11-28T17:41:51.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.307+0000 7f8727fff640 1 --2- 172.21.15.98:0/2007052090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb343b90 0x55c5eb305e80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8718007fd0 tx=0x7f871800a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73e49d857cdeb3c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.307+0000 7f8724ff9640 1 -- 172.21.15.98:0/2007052090 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8718026d00 con 0x55c5eb343b90 2023-11-28T17:41:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.307+0000 7f8724ff9640 1 -- 172.21.15.98:0/2007052090 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8718026e60 con 0x55c5eb343b90 2023-11-28T17:41:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.307+0000 7f8724ff9640 1 -- 172.21.15.98:0/2007052090 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8718004aa0 con 0x55c5eb343b90 2023-11-28T17:41:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.307+0000 7f8737884740 1 -- 172.21.15.98:0/2007052090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb343b90 msgr2=0x55c5eb305e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.307+0000 7f8737884740 1 --2- 172.21.15.98:0/2007052090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb343b90 0x55c5eb305e80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8718007fd0 tx=0x7f871800a000 comp rx=0 tx=0).stop 2023-11-28T17:41:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.307+0000 7f8737884740 1 -- 172.21.15.98:0/2007052090 shutdown_connections 2023-11-28T17:41:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.307+0000 7f8737884740 1 --2- 172.21.15.98:0/2007052090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb350710 0x55c5eb329750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.307+0000 7f8737884740 1 --2- 172.21.15.98:0/2007052090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb343b90 0x55c5eb305e80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.307+0000 7f8737884740 1 --2- 172.21.15.98:0/2007052090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb306890 0x55c5eb305940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.307+0000 7f8737884740 1 -- 172.21.15.98:0/2007052090 >> 172.21.15.98:0/2007052090 conn(0x55c5eb3052f0 msgr2=0x55c5eb34d950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.308+0000 7f8737884740 1 -- 172.21.15.98:0/2007052090 shutdown_connections 2023-11-28T17:41:51.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.308+0000 7f8737884740 1 -- 172.21.15.98:0/2007052090 wait complete. 2023-11-28T17:41:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.308+0000 7f8737884740 1 -- start start 2023-11-28T17:41:51.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.308+0000 7f8737884740 1 Processor -- start 2023-11-28T17:41:51.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.309+0000 7f8737884740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb306890 0x55c5eb329530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.309+0000 7f8737884740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb343b90 0x55c5eb3ab990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.309+0000 7f8737884740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb350710 0x55c5eb3abed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.309+0000 7f8737884740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5eb340f70 con 0x55c5eb350710 2023-11-28T17:41:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.309+0000 7f8737884740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5eb35de40 con 0x55c5eb343b90 2023-11-28T17:41:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.309+0000 7f8737884740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c5eb328970 con 0x55c5eb306890 2023-11-28T17:41:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.309+0000 7f8737884740 10 client.0 _ll_register_callbacks cb 0x7f8720009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:41:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.310+0000 7f872ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb350710 0x55c5eb3abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.310+0000 7f872ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb350710 0x55c5eb3abed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56936/0 (socket says 172.21.15.98:56936) 2023-11-28T17:41:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.310+0000 7f872ed9d640 1 -- 172.21.15.98:0/3324713392 learned_addr learned my addr 172.21.15.98:0/3324713392 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.310+0000 7f872ed9d640 1 -- 172.21.15.98:0/3324713392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb306890 msgr2=0x55c5eb329530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.310+0000 7f872ed9d640 1 --2- 172.21.15.98:0/3324713392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb306890 0x55c5eb329530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.310+0000 7f872ed9d640 1 -- 172.21.15.98:0/3324713392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb343b90 msgr2=0x55c5eb3ab990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.310+0000 7f872ed9d640 1 --2- 172.21.15.98:0/3324713392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb343b90 0x55c5eb3ab990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.310+0000 7f872ed9d640 1 -- 172.21.15.98:0/3324713392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5eb235860 con 0x55c5eb350710 2023-11-28T17:41:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.310+0000 7f872ed9d640 1 --2- 172.21.15.98:0/3324713392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb350710 0x55c5eb3abed0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f872807b1d0 tx=0x7f872807aef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38923b60aea88a5e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.310+0000 7f870d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-11-28T17:41:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.310+0000 7f870d7fa640 1 -- 172.21.15.98:0/3324713392 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87280837d0 con 0x55c5eb350710 2023-11-28T17:41:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.310+0000 7f870d7fa640 1 -- 172.21.15.98:0/3324713392 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f872807a9c0 con 0x55c5eb350710 2023-11-28T17:41:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.310+0000 7f8737884740 1 -- 172.21.15.98:0/3324713392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8718007640 con 0x55c5eb350710 2023-11-28T17:41:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.311+0000 7f870d7fa640 1 -- 172.21.15.98:0/3324713392 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f872807ab80 con 0x55c5eb350710 2023-11-28T17:41:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.311+0000 7f8737884740 10 client.4710 fetch_fsmap learned FSMap version 16 2023-11-28T17:41:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.311+0000 7f8737884740 1 -- 172.21.15.98:0/3324713392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55c5eb22f170 con 0x55c5eb350710 2023-11-28T17:41:51.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.311+0000 7f870d7fa640 1 -- 172.21.15.98:0/3324713392 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8718007640 con 0x55c5eb350710 2023-11-28T17:41:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.311+0000 7f870d7fa640 1 -- 172.21.15.98:0/3324713392 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f872804f020 con 0x55c5eb350710 2023-11-28T17:41:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.311+0000 7f8737884740 10 client.4710 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T17:41:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.311+0000 7f8737884740 10 client.4710 resolve_mds: resolved name 'e' to daemon mds.e 2023-11-28T17:41:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.311+0000 7f8737884740 20 client.4710 populate_metadata read hostname 'smithi098' 2023-11-28T17:41:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.311+0000 7f8737884740 1 --2- 172.21.15.98:0/3324713392 >> [v2:172.21.15.98:6838/3585869153,v1:172.21.15.98:6839/3585869153] conn(0x55c5eb357310 0x55c5eb3576f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.311+0000 7f8737884740 4 client.4710 mds_command: new command op to 4273 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T17:41:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.311+0000 7f8737884740 1 -- 172.21.15.98:0/3324713392 --> [v2:172.21.15.98:6838/3585869153,v1:172.21.15.98:6839/3585869153] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c5eaf6e6c0 con 0x55c5eb357310 2023-11-28T17:41:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.311+0000 7f872e59c640 1 --2- 172.21.15.98:0/3324713392 >> [v2:172.21.15.98:6838/3585869153,v1:172.21.15.98:6839/3585869153] conn(0x55c5eb357310 0x55c5eb3576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f872e59c640 1 --2- 172.21.15.98:0/3324713392 >> [v2:172.21.15.98:6838/3585869153,v1:172.21.15.98:6839/3585869153] conn(0x55c5eb357310 0x55c5eb3576f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3436fd693b92f5db server_cookie=69fd8e01e2f7c78b in_seq=0 out_seq=0 2023-11-28T17:41:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f870d7fa640 10 client.4710 ms_handle_connect on v2:172.21.15.98:6838/3585869153 2023-11-28T17:41:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f870d7fa640 1 -- 172.21.15.98:0/3324713392 <== mds.1 v2:172.21.15.98:6838/3585869153 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55c5eaf6e6c0 con 0x55c5eb357310 2023-11-28T17:41:51.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f870d7fa640 10 client.4710 handle_command_reply: tid=0 2023-11-28T17:41:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f8737884740 1 client.4710 shutdown 2023-11-28T17:41:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f8737884740 1 -- 172.21.15.98:0/3324713392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb350710 msgr2=0x55c5eb3abed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f8737884740 1 --2- 172.21.15.98:0/3324713392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb350710 0x55c5eb3abed0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f872807b1d0 tx=0x7f872807aef0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f8737884740 1 -- 172.21.15.98:0/3324713392 shutdown_connections 2023-11-28T17:41:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f8737884740 1 --2- 172.21.15.98:0/3324713392 >> [v2:172.21.15.98:6838/3585869153,v1:172.21.15.98:6839/3585869153] conn(0x55c5eb357310 0x55c5eb3576f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f8737884740 1 --2- 172.21.15.98:0/3324713392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c5eb350710 0x55c5eb3abed0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f8737884740 1 --2- 172.21.15.98:0/3324713392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5eb343b90 0x55c5eb3ab990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f8737884740 1 --2- 172.21.15.98:0/3324713392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x55c5eb306890 0x55c5eb329530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f870d7fa640 0 client.4710 ms_handle_reset on v2:172.21.15.98:6838/3585869153 2023-11-28T17:41:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f8737884740 1 -- 172.21.15.98:0/3324713392 >> 172.21.15.98:0/3324713392 conn(0x55c5eb3052f0 msgr2=0x55c5eb369f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f8737884740 1 -- 172.21.15.98:0/3324713392 shutdown_connections 2023-11-28T17:41:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.312+0000 7f8737884740 1 -- 172.21.15.98:0/3324713392 wait complete. 2023-11-28T17:41:51.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.313+0000 7f8737884740 20 client.4710 trim_cache size 0 max 16384 2023-11-28T17:41:51.319 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:41:51.319 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:41:51.319 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2023-11-28T17:41:51.319 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4273, 2023-11-28T17:41:51.319 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-11-28T17:41:51.319 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-11-28T17:41:51.319 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-11-28T17:41:51.319 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 29.001963954000001, 2023-11-28T17:41:51.319 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 16, 2023-11-28T17:41:51.320 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 22, 2023-11-28T17:41:51.320 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 22, 2023-11-28T17:41:51.320 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 41.453249274999997 2023-11-28T17:41:51.320 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:41:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.313+0000 7f8734ce7640 1 -- 172.21.15.98:0/2935999446 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f871006b2a0 msgr2=0x7f871006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.313+0000 7f8734ce7640 1 --2- 172.21.15.98:0/2935999446 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f871006b2a0 0x7f871006d760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8720006fd0 tx=0x7f8720008040 comp rx=0 tx=0).stop 2023-11-28T17:41:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.313+0000 7f8734ce7640 1 -- 172.21.15.98:0/2935999446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8730154030 msgr2=0x7f87301588d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.313+0000 7f8734ce7640 1 --2- 172.21.15.98:0/2935999446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8730154030 0x7f87301588d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8728036040 tx=0x7f8728004a40 comp rx=0 tx=0).stop 2023-11-28T17:41:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.313+0000 7f8734ce7640 1 -- 172.21.15.98:0/2935999446 shutdown_connections 2023-11-28T17:41:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.313+0000 7f8734ce7640 1 --2- 172.21.15.98:0/2935999446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8730154030 0x7f87301588d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.313+0000 7f8734ce7640 1 --2- 172.21.15.98:0/2935999446 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f87300c8600 0x7f873014daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.313+0000 7f8734ce7640 1 --2- 172.21.15.98:0/2935999446 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f871006b2a0 0x7f871006d760 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.313+0000 7f8734ce7640 1 --2- 172.21.15.98:0/2935999446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f873006dcf0 0x7f8730143370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.313+0000 7f8734ce7640 1 -- 172.21.15.98:0/2935999446 >> 172.21.15.98:0/2935999446 conn(0x7f873006c0f0 msgr2=0x7f8730157930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.313+0000 7f8734ce7640 1 -- 172.21.15.98:0/2935999446 shutdown_connections 2023-11-28T17:41:51.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.313+0000 7f8734ce7640 1 -- 172.21.15.98:0/2935999446 wait complete. 2023-11-28T17:41:51.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-28T17:41:51.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.540+0000 7f30b2168640 1 Processor -- start 2023-11-28T17:41:51.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.540+0000 7f30b2168640 1 -- start start 2023-11-28T17:41:51.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.542+0000 7f30b2168640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f30ac0c4760 0x7f30ac0c2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.542+0000 7f30b2168640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30ac15a0f0 0x7f30ac15a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.542+0000 7f30b2168640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac15aaa0 con 0x7f30ac0c4380 2023-11-28T17:41:51.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.542+0000 7f30b2168640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30ac0c2a60 con 0x7f30ac0c4760 2023-11-28T17:41:51.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.542+0000 7f30b2168640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac156980 con 0x7f30ac15a0f0 2023-11-28T17:41:51.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.542+0000 7f30a3fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f30ac0c4760 0x7f30ac0c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.542+0000 7f30a3fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f30ac0c4760 0x7f30ac0c2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56952/0 (socket says 172.21.15.98:56952) 2023-11-28T17:41:51.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.542+0000 7f30a3fff640 1 -- 172.21.15.98:0/1025194329 learned_addr learned my addr 172.21.15.98:0/1025194329 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.543+0000 7f30abfff640 1 --2- 172.21.15.98:0/1025194329 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30ac15a0f0 0x7f30ac15a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.543+0000 7f30a3fff640 1 -- 172.21.15.98:0/1025194329 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30ac15a0f0 msgr2=0x7f30ac15a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.543+0000 7f30a3fff640 1 --2- 172.21.15.98:0/1025194329 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30ac15a0f0 0x7f30ac15a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.543+0000 7f30a3fff640 1 -- 172.21.15.98:0/1025194329 >> v1:172.21.15.98:6789/0 conn(0x7f30ac0c4380 legacy=0x7f30ac0c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.543+0000 7f30a3fff640 1 -- 172.21.15.98:0/1025194329 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac156c40 con 0x7f30ac0c4760 2023-11-28T17:41:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.544+0000 7f30a3fff640 1 --2- 172.21.15.98:0/1025194329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac0c4760 0x7f30ac0c2520 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3094009870 tx=0x7f309402dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22dae703977e3890 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.544+0000 7f30aaffd640 1 -- 172.21.15.98:0/1025194329 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309402e9e0 con 0x7f30ac0c4760 2023-11-28T17:41:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.544+0000 7f30aaffd640 1 -- 172.21.15.98:0/1025194329 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309402eb40 con 0x7f30ac0c4760 2023-11-28T17:41:51.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.544+0000 7f30aaffd640 1 -- 172.21.15.98:0/1025194329 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3094036440 con 0x7f30ac0c4760 2023-11-28T17:41:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.545+0000 7f30b2168640 1 -- 172.21.15.98:0/1025194329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac0c4760 msgr2=0x7f30ac0c2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.545+0000 7f30b2168640 1 --2- 172.21.15.98:0/1025194329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac0c4760 0x7f30ac0c2520 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3094009870 tx=0x7f309402dc30 comp rx=0 tx=0).stop 2023-11-28T17:41:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.545+0000 7f30b2168640 1 -- 172.21.15.98:0/1025194329 shutdown_connections 2023-11-28T17:41:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.545+0000 7f30b2168640 1 --2- 172.21.15.98:0/1025194329 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30ac15a0f0 0x7f30ac15a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.545+0000 7f30b2168640 1 --2- 172.21.15.98:0/1025194329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac0c4760 0x7f30ac0c2520 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.545+0000 7f30b2168640 1 -- 172.21.15.98:0/1025194329 >> 172.21.15.98:0/1025194329 conn(0x7f30ac06c0f0 msgr2=0x7f30ac06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.545+0000 7f30b2168640 1 -- 172.21.15.98:0/1025194329 shutdown_connections 2023-11-28T17:41:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.545+0000 7f30b2168640 1 -- 172.21.15.98:0/1025194329 wait complete. 2023-11-28T17:41:51.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.547+0000 7f30b2168640 1 Processor -- start 2023-11-28T17:41:51.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.547+0000 7f30b2168640 1 -- start start 2023-11-28T17:41:51.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.548+0000 7f30b2168640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30ac0c4380 0x7f30ac1ef500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.548+0000 7f30b2168640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30ac0c4760 0x7f30ac14c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.548+0000 7f30b2168640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac15a0f0 0x7f30ac158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.548+0000 7f30b2168640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac14adb0 con 0x7f30ac15a0f0 2023-11-28T17:41:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.548+0000 7f30b2168640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac14d2c0 con 0x7f30ac0c4380 2023-11-28T17:41:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.548+0000 7f30b2168640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac150950 con 0x7f30ac0c4760 2023-11-28T17:41:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.548+0000 7f30abfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac15a0f0 0x7f30ac158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.548+0000 7f30abfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac15a0f0 0x7f30ac158fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56960/0 (socket says 172.21.15.98:56960) 2023-11-28T17:41:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.548+0000 7f30abfff640 1 -- 172.21.15.98:0/1499139845 learned_addr learned my addr 172.21.15.98:0/1499139845 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.549+0000 7f30ab7fe640 1 --2- 172.21.15.98:0/1499139845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30ac0c4380 0x7f30ac1ef500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.549+0000 7f30abfff640 1 -- 172.21.15.98:0/1499139845 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30ac0c4760 msgr2=0x7f30ac14c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.549+0000 7f30abfff640 1 --2- 172.21.15.98:0/1499139845 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30ac0c4760 0x7f30ac14c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.549+0000 7f30abfff640 1 -- 172.21.15.98:0/1499139845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30ac0c4380 msgr2=0x7f30ac1ef500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.549+0000 7f30abfff640 1 --2- 172.21.15.98:0/1499139845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30ac0c4380 0x7f30ac1ef500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.549+0000 7f30abfff640 1 -- 172.21.15.98:0/1499139845 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30940094d0 con 0x7f30ac15a0f0 2023-11-28T17:41:51.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.549+0000 7f30abfff640 1 --2- 172.21.15.98:0/1499139845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac15a0f0 0x7f30ac158fb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f30a4007e60 tx=0x7f30a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.549+0000 7f30a8ff9640 1 -- 172.21.15.98:0/1499139845 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4014420 con 0x7f30ac15a0f0 2023-11-28T17:41:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.549+0000 7f30a8ff9640 1 -- 172.21.15.98:0/1499139845 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a40145a0 con 0x7f30ac15a0f0 2023-11-28T17:41:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.549+0000 7f30b2168640 1 -- 172.21.15.98:0/1499139845 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ac14aa80 con 0x7f30ac15a0f0 2023-11-28T17:41:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.549+0000 7f30a8ff9640 1 -- 172.21.15.98:0/1499139845 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4014950 con 0x7f30ac15a0f0 2023-11-28T17:41:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.549+0000 7f30b2168640 1 -- 172.21.15.98:0/1499139845 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ac14ca20 con 0x7f30ac15a0f0 2023-11-28T17:41:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.552+0000 7f30a8ff9640 1 -- 172.21.15.98:0/1499139845 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f30a4014ab0 con 0x7f30ac15a0f0 2023-11-28T17:41:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.552+0000 7f30a8ff9640 1 --2- 172.21.15.98:0/1499139845 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f308c06b2d0 0x7f308c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.553+0000 7f30a8ff9640 1 -- 172.21.15.98:0/1499139845 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30a408a660 con 0x7f30ac15a0f0 2023-11-28T17:41:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.553+0000 7f30ab7fe640 1 --2- 172.21.15.98:0/1499139845 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f308c06b2d0 0x7f308c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.553+0000 7f30b4dd7740 1 Processor -- start 2023-11-28T17:41:51.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.553+0000 7f30b4dd7740 1 -- start start 2023-11-28T17:41:51.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.553+0000 7f30ab7fe640 1 --2- 172.21.15.98:0/1499139845 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f308c06b2d0 0x7f308c06d790 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f309c000f60 tx=0x7f309c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.554+0000 7f30b4dd7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 0x563a2632bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.554+0000 7f30b4dd7740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a26375710 0x563a2632c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.554+0000 7f30b4dd7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a2632a2f0 0x563a2636b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.554+0000 7f30b4dd7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2628a880 con 0x563a2632a2f0 2023-11-28T17:41:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.554+0000 7f30b4dd7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a26299ae0 con 0x563a2632b890 2023-11-28T17:41:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.554+0000 7f30b4dd7740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x563a262971d0 con 0x563a26375710 2023-11-28T17:41:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.554+0000 7f30abfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a2632a2f0 0x563a2636b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.554+0000 7f30abfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a2632a2f0 0x563a2636b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56972/0 (socket says 172.21.15.98:56972) 2023-11-28T17:41:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.554+0000 7f30abfff640 1 -- 172.21.15.98:0/2141162039 learned_addr learned my addr 172.21.15.98:0/2141162039 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.555+0000 7f30ab7fe640 1 --2- 172.21.15.98:0/2141162039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 0x563a2632bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.555+0000 7f30abfff640 1 -- 172.21.15.98:0/2141162039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a26375710 msgr2=0x563a2632c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.555+0000 7f30a3fff640 1 --2- 172.21.15.98:0/2141162039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a26375710 0x563a2632c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.555+0000 7f30abfff640 1 --2- 172.21.15.98:0/2141162039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a26375710 0x563a2632c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.555+0000 7f30abfff640 1 -- 172.21.15.98:0/2141162039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 msgr2=0x563a2632bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.555+0000 7f30abfff640 1 --2- 172.21.15.98:0/2141162039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 0x563a2632bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.555+0000 7f30abfff640 1 -- 172.21.15.98:0/2141162039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a2625a860 con 0x563a2632a2f0 2023-11-28T17:41:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.555+0000 7f30ab7fe640 1 --2- 172.21.15.98:0/2141162039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 0x563a2632bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.555+0000 7f30abfff640 1 --2- 172.21.15.98:0/2141162039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a2632a2f0 0x563a2636b210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f30a400c040 tx=0x7f30a4059760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=564e7f381dcbcd50 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.555+0000 7f30a17fa640 1 -- 172.21.15.98:0/2141162039 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a40613f0 con 0x563a2632a2f0 2023-11-28T17:41:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.555+0000 7f30a17fa640 1 -- 172.21.15.98:0/2141162039 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a4057400 con 0x563a2632a2f0 2023-11-28T17:41:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.555+0000 7f30a17fa640 1 -- 172.21.15.98:0/2141162039 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4058550 con 0x563a2632a2f0 2023-11-28T17:41:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.556+0000 7f30b4dd7740 1 -- 172.21.15.98:0/2141162039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a2632a2f0 msgr2=0x563a2636b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.556+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/2141162039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a2632a2f0 0x563a2636b210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f30a400c040 tx=0x7f30a4059760 comp rx=0 tx=0).stop 2023-11-28T17:41:51.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.556+0000 7f30b4dd7740 1 -- 172.21.15.98:0/2141162039 shutdown_connections 2023-11-28T17:41:51.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.556+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/2141162039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a2632a2f0 0x563a2636b210 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.556+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/2141162039 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a26375710 0x563a2632c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.556+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/2141162039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 0x563a2632bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.556+0000 7f30b4dd7740 1 -- 172.21.15.98:0/2141162039 >> 172.21.15.98:0/2141162039 conn(0x563a26328290 msgr2=0x563a2632a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.556+0000 7f30b4dd7740 1 -- 172.21.15.98:0/2141162039 shutdown_connections 2023-11-28T17:41:51.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.556+0000 7f30b4dd7740 1 -- 172.21.15.98:0/2141162039 wait complete. 2023-11-28T17:41:51.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.557+0000 7f30b4dd7740 1 -- start start 2023-11-28T17:41:51.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.557+0000 7f30b4dd7740 1 Processor -- start 2023-11-28T17:41:51.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30b4dd7740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a2632b890 0x563a26367540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30b4dd7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26375710 0x563a263491b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30b4dd7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a26369100 0x563a263496f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30b4dd7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a26292ac0 con 0x563a26375710 2023-11-28T17:41:51.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30b4dd7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2622f2f0 con 0x563a26369100 2023-11-28T17:41:51.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30b4dd7740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x563a2624cc70 con 0x563a2632b890 2023-11-28T17:41:51.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30b4dd7740 10 client.0 _ll_register_callbacks cb 0x563a25fa4ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:41:51.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30a3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26375710 0x563a263491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30a3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26375710 0x563a263491b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56982/0 (socket says 172.21.15.98:56982) 2023-11-28T17:41:51.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30a3fff640 1 -- 172.21.15.98:0/3949563141 learned_addr learned my addr 172.21.15.98:0/3949563141 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30a3fff640 1 -- 172.21.15.98:0/3949563141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a2632b890 msgr2=0x563a26367540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30a3fff640 1 --2- 172.21.15.98:0/3949563141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a2632b890 0x563a26367540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30a3fff640 1 -- 172.21.15.98:0/3949563141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a26369100 msgr2=0x563a263496f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30abfff640 1 --2- 172.21.15.98:0/3949563141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a26369100 0x563a263496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30a3fff640 1 --2- 172.21.15.98:0/3949563141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a26369100 0x563a263496f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.558+0000 7f30a3fff640 1 -- 172.21.15.98:0/3949563141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ac14ca20 con 0x563a26375710 2023-11-28T17:41:51.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.559+0000 7f30a3fff640 1 --2- 172.21.15.98:0/3949563141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26375710 0x563a263491b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3094009990 tx=0x7f30940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d37b10dc9b371ea7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.559+0000 7f308a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-11-28T17:41:51.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.559+0000 7f308a7fc640 1 -- 172.21.15.98:0/3949563141 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3094036c00 con 0x563a26375710 2023-11-28T17:41:51.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.559+0000 7f308a7fc640 1 -- 172.21.15.98:0/3949563141 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3094036d60 con 0x563a26375710 2023-11-28T17:41:51.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.559+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3949563141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563a2628baa0 con 0x563a26375710 2023-11-28T17:41:51.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.559+0000 7f308a7fc640 1 -- 172.21.15.98:0/3949563141 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3094041470 con 0x563a26375710 2023-11-28T17:41:51.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.559+0000 7f30b4dd7740 10 client.4722 fetch_fsmap learned FSMap version 16 2023-11-28T17:41:51.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.559+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3949563141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x563a26291140 con 0x563a26375710 2023-11-28T17:41:51.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.560+0000 7f308a7fc640 1 -- 172.21.15.98:0/3949563141 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563a2628baa0 con 0x563a26375710 2023-11-28T17:41:51.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.560+0000 7f308a7fc640 1 -- 172.21.15.98:0/3949563141 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f30940415d0 con 0x563a26375710 2023-11-28T17:41:51.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.560+0000 7f30b4dd7740 10 client.4722 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T17:41:51.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.560+0000 7f30b4dd7740 10 client.4722 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-28T17:41:51.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.560+0000 7f30b4dd7740 20 client.4722 populate_metadata read hostname 'smithi098' 2023-11-28T17:41:51.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.560+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3949563141 >> [v2:172.21.15.113:6832/2183585971,v1:172.21.15.113:6834/2183585971] conn(0x563a2637c890 0x563a2638abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.560+0000 7f30b4dd7740 4 client.4722 mds_command: new command op to 4291 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T17:41:51.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.560+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3949563141 --> [v2:172.21.15.113:6832/2183585971,v1:172.21.15.113:6834/2183585971] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563a2628b7c0 con 0x563a2637c890 2023-11-28T17:41:51.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.560+0000 7f30ab7fe640 1 --2- 172.21.15.98:0/3949563141 >> [v2:172.21.15.113:6832/2183585971,v1:172.21.15.113:6834/2183585971] conn(0x563a2637c890 0x563a2638abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.561+0000 7f30ab7fe640 1 --2- 172.21.15.98:0/3949563141 >> [v2:172.21.15.113:6832/2183585971,v1:172.21.15.113:6834/2183585971] conn(0x563a2637c890 0x563a2638abe0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ab1967617547eca server_cookie=d9793133613902f3 in_seq=0 out_seq=0 2023-11-28T17:41:51.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.561+0000 7f308a7fc640 10 client.4722 ms_handle_connect on v2:172.21.15.113:6832/2183585971 2023-11-28T17:41:51.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.563+0000 7f308a7fc640 1 -- 172.21.15.98:0/3949563141 <== mds.0 v2:172.21.15.113:6832/2183585971 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563a2628b7c0 con 0x563a2637c890 2023-11-28T17:41:51.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.563+0000 7f308a7fc640 10 client.4722 handle_command_reply: tid=0 2023-11-28T17:41:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.563+0000 7f30b4dd7740 1 client.4722 shutdown 2023-11-28T17:41:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.563+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3949563141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26375710 msgr2=0x563a263491b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.563+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3949563141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26375710 0x563a263491b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3094009990 tx=0x7f30940046c0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.563+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3949563141 shutdown_connections 2023-11-28T17:41:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.563+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3949563141 >> [v2:172.21.15.113:6832/2183585971,v1:172.21.15.113:6834/2183585971] conn(0x563a2637c890 0x563a2638abe0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.563+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3949563141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a26369100 0x563a263496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.563+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3949563141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26375710 0x563a263491b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.563+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3949563141 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a2632b890 0x563a26367540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.563+0000 7f308a7fc640 0 client.4722 ms_handle_reset on v2:172.21.15.113:6832/2183585971 2023-11-28T17:41:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.563+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3949563141 >> 172.21.15.98:0/3949563141 conn(0x563a2632a2f0 msgr2=0x563a263251c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.563+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3949563141 shutdown_connections 2023-11-28T17:41:51.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.563+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3949563141 wait complete. 2023-11-28T17:41:51.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.564+0000 7f30b4dd7740 20 client.4722 trim_cache size 0 max 16384 2023-11-28T17:41:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.580+0000 7f30b4dd7740 1 Processor -- start 2023-11-28T17:41:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.580+0000 7f30b4dd7740 1 -- start start 2023-11-28T17:41:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.581+0000 7f30b4dd7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 0x563a2632a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.581+0000 7f30b4dd7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26369100 0x563a2632ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.581+0000 7f30b4dd7740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a26375710 0x563a2636f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.581+0000 7f30b4dd7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a25ff5f20 con 0x563a26369100 2023-11-28T17:41:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.581+0000 7f30b4dd7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a263684a0 con 0x563a2632b890 2023-11-28T17:41:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.581+0000 7f30b4dd7740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x563a2632c610 con 0x563a26375710 2023-11-28T17:41:51.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.581+0000 7f30a3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26369100 0x563a2632ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.581+0000 7f30a3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26369100 0x563a2632ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56992/0 (socket says 172.21.15.98:56992) 2023-11-28T17:41:51.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.581+0000 7f30a3fff640 1 -- 172.21.15.98:0/3045381955 learned_addr learned my addr 172.21.15.98:0/3045381955 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.582+0000 7f30ab7fe640 1 --2- 172.21.15.98:0/3045381955 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 0x563a2632a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.582+0000 7f30a3fff640 1 -- 172.21.15.98:0/3045381955 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a26375710 msgr2=0x563a2636f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.582+0000 7f30a3fff640 1 --2- 172.21.15.98:0/3045381955 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a26375710 0x563a2636f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.582+0000 7f30a3fff640 1 -- 172.21.15.98:0/3045381955 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 msgr2=0x563a2632a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.582+0000 7f30a3fff640 1 --2- 172.21.15.98:0/3045381955 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 0x563a2632a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.582+0000 7f30a3fff640 1 -- 172.21.15.98:0/3045381955 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a2622eb80 con 0x563a26369100 2023-11-28T17:41:51.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.582+0000 7f30ab7fe640 1 --2- 172.21.15.98:0/3045381955 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 0x563a2632a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.582+0000 7f30a3fff640 1 --2- 172.21.15.98:0/3045381955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26369100 0x563a2632ae80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f309402dc70 tx=0x7f3094040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21c76142ca9b561 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.582+0000 7f30a1ffb640 1 -- 172.21.15.98:0/3045381955 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3094036440 con 0x563a26369100 2023-11-28T17:41:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.582+0000 7f30a1ffb640 1 -- 172.21.15.98:0/3045381955 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30940365a0 con 0x563a26369100 2023-11-28T17:41:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.582+0000 7f30a1ffb640 1 -- 172.21.15.98:0/3045381955 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3094041670 con 0x563a26369100 2023-11-28T17:41:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.583+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3045381955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26369100 msgr2=0x563a2632ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.583+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3045381955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26369100 0x563a2632ae80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f309402dc70 tx=0x7f3094040000 comp rx=0 tx=0).stop 2023-11-28T17:41:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.583+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3045381955 shutdown_connections 2023-11-28T17:41:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.583+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3045381955 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a26375710 0x563a2636f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.583+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3045381955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26369100 0x563a2632ae80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.583+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3045381955 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 0x563a2632a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.583+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3045381955 >> 172.21.15.98:0/3045381955 conn(0x563a2632a2f0 msgr2=0x563a26367520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.583+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3045381955 shutdown_connections 2023-11-28T17:41:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.583+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3045381955 wait complete. 2023-11-28T17:41:51.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.584+0000 7f30b4dd7740 1 -- start start 2023-11-28T17:41:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.584+0000 7f30b4dd7740 1 Processor -- start 2023-11-28T17:41:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30b4dd7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 0x563a26369b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30b4dd7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26369100 0x563a263700e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30b4dd7740 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a26375710 0x563a263d9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30b4dd7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2636a520 con 0x563a26369100 2023-11-28T17:41:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30b4dd7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a2637f720 con 0x563a2632b890 2023-11-28T17:41:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30b4dd7740 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x563a2636a050 con 0x563a26375710 2023-11-28T17:41:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30b4dd7740 10 client.0 _ll_register_callbacks cb 0x7f3094039070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:41:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30a3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26369100 0x563a263700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30a3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26369100 0x563a263700e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57002/0 (socket says 172.21.15.98:57002) 2023-11-28T17:41:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30a3fff640 1 -- 172.21.15.98:0/3075023359 learned_addr learned my addr 172.21.15.98:0/3075023359 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30ab7fe640 1 --2- 172.21.15.98:0/3075023359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 0x563a26369b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30a3fff640 1 -- 172.21.15.98:0/3075023359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a26375710 msgr2=0x563a263d9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30a3fff640 1 --2- 172.21.15.98:0/3075023359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a26375710 0x563a263d9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30a3fff640 1 -- 172.21.15.98:0/3075023359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 msgr2=0x563a26369b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30a3fff640 1 --2- 172.21.15.98:0/3075023359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 0x563a26369b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30a3fff640 1 -- 172.21.15.98:0/3075023359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563a26291140 con 0x563a26369100 2023-11-28T17:41:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.585+0000 7f30ab7fe640 1 --2- 172.21.15.98:0/3075023359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 0x563a26369b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.586+0000 7f30a3fff640 1 --2- 172.21.15.98:0/3075023359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26369100 0x563a263700e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3094054040 tx=0x7f309403d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4de42c78c6b7dad3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.586+0000 7f308a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-11-28T17:41:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.586+0000 7f308a7fc640 1 -- 172.21.15.98:0/3075023359 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3094004040 con 0x563a26369100 2023-11-28T17:41:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.586+0000 7f308a7fc640 1 -- 172.21.15.98:0/3075023359 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3094000e40 con 0x563a26369100 2023-11-28T17:41:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.586+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3075023359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30a4057400 con 0x563a26369100 2023-11-28T17:41:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.586+0000 7f308a7fc640 1 -- 172.21.15.98:0/3075023359 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30940041a0 con 0x563a26369100 2023-11-28T17:41:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.586+0000 7f30b4dd7740 10 client.4728 fetch_fsmap learned FSMap version 16 2023-11-28T17:41:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.586+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3075023359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x563a26254170 con 0x563a26369100 2023-11-28T17:41:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.586+0000 7f308a7fc640 1 -- 172.21.15.98:0/3075023359 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a4057400 con 0x563a26369100 2023-11-28T17:41:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.586+0000 7f308a7fc640 1 -- 172.21.15.98:0/3075023359 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f3094004300 con 0x563a26369100 2023-11-28T17:41:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.586+0000 7f30b4dd7740 10 client.4728 fetch_fsmap finished waiting for FSMap version 16 2023-11-28T17:41:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.586+0000 7f30b4dd7740 10 client.4728 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-28T17:41:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.587+0000 7f30b4dd7740 20 client.4728 populate_metadata read hostname 'smithi098' 2023-11-28T17:41:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.587+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3075023359 >> [v2:172.21.15.113:6832/2183585971,v1:172.21.15.113:6834/2183585971] conn(0x563a263da450 0x563a263da830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.587+0000 7f30b4dd7740 4 client.4728 mds_command: new command op to 4291 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T17:41:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.587+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3075023359 --> [v2:172.21.15.113:6832/2183585971,v1:172.21.15.113:6834/2183585971] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563a25f936c0 con 0x563a263da450 2023-11-28T17:41:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.587+0000 7f30ab7fe640 1 --2- 172.21.15.98:0/3075023359 >> [v2:172.21.15.113:6832/2183585971,v1:172.21.15.113:6834/2183585971] conn(0x563a263da450 0x563a263da830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:51.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.588+0000 7f30ab7fe640 1 --2- 172.21.15.98:0/3075023359 >> [v2:172.21.15.113:6832/2183585971,v1:172.21.15.113:6834/2183585971] conn(0x563a263da450 0x563a263da830 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74dcfda5ddaf041a server_cookie=b367bc7f71748092 in_seq=0 out_seq=0 2023-11-28T17:41:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.588+0000 7f308a7fc640 10 client.4728 ms_handle_connect on v2:172.21.15.113:6832/2183585971 2023-11-28T17:41:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.588+0000 7f308a7fc640 1 -- 172.21.15.98:0/3075023359 <== mds.0 v2:172.21.15.113:6832/2183585971 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x563a25f936c0 con 0x563a263da450 2023-11-28T17:41:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.588+0000 7f308a7fc640 10 client.4728 handle_command_reply: tid=0 2023-11-28T17:41:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.588+0000 7f30b4dd7740 1 client.4728 shutdown 2023-11-28T17:41:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.589+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3075023359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26369100 msgr2=0x563a263700e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.589+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3075023359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26369100 0x563a263700e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3094054040 tx=0x7f309403d000 comp rx=0 tx=0).stop 2023-11-28T17:41:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.589+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3075023359 shutdown_connections 2023-11-28T17:41:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.589+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3075023359 >> [v2:172.21.15.113:6832/2183585971,v1:172.21.15.113:6834/2183585971] conn(0x563a263da450 0x563a263da830 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.589+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3075023359 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x563a26375710 0x563a263d9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.589+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3075023359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a26369100 0x563a263700e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.589+0000 7f308a7fc640 0 client.4728 ms_handle_reset on v2:172.21.15.113:6832/2183585971 2023-11-28T17:41:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.589+0000 7f30b4dd7740 1 --2- 172.21.15.98:0/3075023359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a2632b890 0x563a26369b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.589+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3075023359 >> 172.21.15.98:0/3075023359 conn(0x563a2632a2f0 msgr2=0x563a2632be10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.589+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3075023359 shutdown_connections 2023-11-28T17:41:51.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.589+0000 7f30b4dd7740 1 -- 172.21.15.98:0/3075023359 wait complete. 2023-11-28T17:41:51.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.589+0000 7f30b4dd7740 20 client.4728 trim_cache size 0 max 16384 2023-11-28T17:41:51.594 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:41:51.594 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:41:51.594 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2023-11-28T17:41:51.594 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4291, 2023-11-28T17:41:51.595 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-11-28T17:41:51.595 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-11-28T17:41:51.595 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-11-28T17:41:51.595 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 34.336872206000002, 2023-11-28T17:41:51.595 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 16, 2023-11-28T17:41:51.595 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 22, 2023-11-28T17:41:51.595 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 22, 2023-11-28T17:41:51.595 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 41.705415107 2023-11-28T17:41:51.595 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:41:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.589+0000 7f30b2168640 1 -- 172.21.15.98:0/1499139845 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f308c06b2d0 msgr2=0x7f308c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.589+0000 7f30b2168640 1 --2- 172.21.15.98:0/1499139845 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f308c06b2d0 0x7f308c06d790 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f309c000f60 tx=0x7f309c009040 comp rx=0 tx=0).stop 2023-11-28T17:41:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.590+0000 7f30b2168640 1 -- 172.21.15.98:0/1499139845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac15a0f0 msgr2=0x7f30ac158fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.590+0000 7f30b2168640 1 --2- 172.21.15.98:0/1499139845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac15a0f0 0x7f30ac158fb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f30a4007e60 tx=0x7f30a400e4b0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.590+0000 7f30b2168640 1 -- 172.21.15.98:0/1499139845 shutdown_connections 2023-11-28T17:41:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.590+0000 7f30b2168640 1 --2- 172.21.15.98:0/1499139845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac15a0f0 0x7f30ac158fb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.590+0000 7f30b2168640 1 --2- 172.21.15.98:0/1499139845 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30ac0c4760 0x7f30ac14c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.590+0000 7f30b2168640 1 --2- 172.21.15.98:0/1499139845 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f308c06b2d0 0x7f308c06d790 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.590+0000 7f30b2168640 1 --2- 172.21.15.98:0/1499139845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30ac0c4380 0x7f30ac1ef500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.590+0000 7f30b2168640 1 -- 172.21.15.98:0/1499139845 >> 172.21.15.98:0/1499139845 conn(0x7f30ac06c0f0 msgr2=0x7f30ac0cb150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.590+0000 7f30b2168640 1 -- 172.21.15.98:0/1499139845 shutdown_connections 2023-11-28T17:41:51.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:51.590+0000 7f30b2168640 1 -- 172.21.15.98:0/1499139845 wait complete. 2023-11-28T17:41:51.608 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-11-28T17:41:51.630 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-11-28T17:41:51.631 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2023-11-28T17:41:51.631 INFO:tasks.ceph_fuse:client.0 config is {} 2023-11-28T17:41:51.631 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T17:41:51.631 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T17:41:51.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:51.631 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-11-28T17:41:51.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:51.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2023-11-28T17:41:51.789 INFO:teuthology.orchestra.run.smithi098.stderr:Cannot find device "ceph-brx" 2023-11-28T17:41:51.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:41:51.794 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-11-28T17:41:51.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T17:41:52.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.003+0000 7f8f55407640 1 Processor -- start 2023-11-28T17:41:52.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.004+0000 7f8f55407640 1 -- start start 2023-11-28T17:41:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.005+0000 7f8f55407640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f50153f60 0x7f8f500cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.005+0000 7f8f55407640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5006b010 con 0x7f8f5015d240 2023-11-28T17:41:52.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.005+0000 7f8f55407640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5015dd60 con 0x7f8f5006bd40 2023-11-28T17:41:52.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.005+0000 7f8f55407640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5015e040 con 0x7f8f50153f60 2023-11-28T17:41:52.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.005+0000 7f8f4f7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8f5015d240 0x7f8f500cc3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46880/0 (socket says 172.21.15.98:46880) 2023-11-28T17:41:52.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.005+0000 7f8f4f7fe640 1 -- 172.21.15.98:0/3018342484 learned_addr learned my addr 172.21.15.98:0/3018342484 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4effd640 1 --2- 172.21.15.98:0/3018342484 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f50153f60 0x7f8f500cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4e7fc640 1 -- 172.21.15.98:0/3018342484 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012520993 0 0) 0x7f8f5006b010 con 0x7f8f5015d240 2023-11-28T17:41:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4e7fc640 1 -- 172.21.15.98:0/3018342484 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f3c003520 con 0x7f8f5015d240 2023-11-28T17:41:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4e7fc640 1 -- 172.21.15.98:0/3018342484 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1352897985 0 0) 0x7f8f3c003520 con 0x7f8f5015d240 2023-11-28T17:41:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4e7fc640 1 -- 172.21.15.98:0/3018342484 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f50153f60 msgr2=0x7f8f500cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4e7fc640 1 --2- 172.21.15.98:0/3018342484 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f50153f60 0x7f8f500cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4e7fc640 1 -- 172.21.15.98:0/3018342484 >> v1:172.21.15.113:6789/0 conn(0x7f8f5006bd40 legacy=0x7f8f500c4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4e7fc640 1 -- 172.21.15.98:0/3018342484 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5006b260 con 0x7f8f5015d240 2023-11-28T17:41:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4e7fc640 1 -- 172.21.15.98:0/3018342484 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f8f48001670 con 0x7f8f5015d240 2023-11-28T17:41:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4e7fc640 1 -- 172.21.15.98:0/3018342484 >> v1:172.21.15.98:6789/0 conn(0x7f8f5015d240 legacy=0x7f8f500cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4e7fc640 1 --2- 172.21.15.98:0/3018342484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f3c0039a0 0x7f8f3c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4e7fc640 1 --2- 172.21.15.98:0/3018342484 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f50153f60 0x7f8f3c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4e7fc640 1 --2- 172.21.15.98:0/3018342484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f3c004d00 0x7f8f3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4e7fc640 1 -- 172.21.15.98:0/3018342484 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5015e040 con 0x7f8f3c0039a0 2023-11-28T17:41:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4e7fc640 1 -- 172.21.15.98:0/3018342484 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c00baf0 con 0x7f8f3c004d00 2023-11-28T17:41:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.006+0000 7f8f4e7fc640 1 -- 172.21.15.98:0/3018342484 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c00bcb0 con 0x7f8f50153f60 2023-11-28T17:41:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.007+0000 7f8f4effd640 1 --2- 172.21.15.98:0/3018342484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f3c0039a0 0x7f8f3c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.007+0000 7f8f4f7fe640 1 --2- 172.21.15.98:0/3018342484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f3c004d00 0x7f8f3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.007+0000 7f8f4effd640 1 -- 172.21.15.98:0/3018342484 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f50153f60 msgr2=0x7f8f3c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.007+0000 7f8f4effd640 1 --2- 172.21.15.98:0/3018342484 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f50153f60 0x7f8f3c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.007+0000 7f8f4effd640 1 -- 172.21.15.98:0/3018342484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f3c004d00 msgr2=0x7f8f3c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.007+0000 7f8f4effd640 1 --2- 172.21.15.98:0/3018342484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f3c004d00 0x7f8f3c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:52.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.007+0000 7f8f4effd640 1 -- 172.21.15.98:0/3018342484 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3c003520 con 0x7f8f3c0039a0 2023-11-28T17:41:52.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.007+0000 7f8f4f7fe640 1 --2- 172.21.15.98:0/3018342484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f3c004d00 0x7f8f3c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:52.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.008+0000 7f8f4effd640 1 --2- 172.21.15.98:0/3018342484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f3c0039a0 0x7f8f3c003dc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009f90 tx=0x7f8f3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b068c6291bb304fe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:52.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.008+0000 7f8f4e7fc640 1 -- 172.21.15.98:0/3018342484 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4002cc20 con 0x7f8f3c0039a0 2023-11-28T17:41:52.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.008+0000 7f8f4e7fc640 1 -- 172.21.15.98:0/3018342484 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4000f040 con 0x7f8f3c0039a0 2023-11-28T17:41:52.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.008+0000 7f8f4e7fc640 1 -- 172.21.15.98:0/3018342484 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f40034440 con 0x7f8f3c0039a0 2023-11-28T17:41:52.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.009+0000 7f8f55407640 1 -- 172.21.15.98:0/3018342484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f3c0039a0 msgr2=0x7f8f3c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:52.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.009+0000 7f8f55407640 1 --2- 172.21.15.98:0/3018342484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f3c0039a0 0x7f8f3c003dc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009f90 tx=0x7f8f3c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:52.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.009+0000 7f8f4f7fe640 1 -- 172.21.15.98:0/3018342484 reap_dead start 2023-11-28T17:41:52.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.009+0000 7f8f55407640 1 -- 172.21.15.98:0/3018342484 shutdown_connections 2023-11-28T17:41:52.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.009+0000 7f8f55407640 1 -- 172.21.15.98:0/3018342484 >> 172.21.15.98:0/3018342484 conn(0x7f8f50066410 msgr2=0x7f8f50066820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:52.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.009+0000 7f8f55407640 1 -- 172.21.15.98:0/3018342484 shutdown_connections 2023-11-28T17:41:52.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.009+0000 7f8f55407640 1 -- 172.21.15.98:0/3018342484 wait complete. 2023-11-28T17:41:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.011+0000 7f8f55407640 1 Processor -- start 2023-11-28T17:41:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.011+0000 7f8f55407640 1 -- start start 2023-11-28T17:41:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.012+0000 7f8f55407640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f3c0039a0 0x7f8f50156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.012+0000 7f8f55407640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f501561b0 0x7f8f501565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.012+0000 7f8f55407640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f501f2e50 0x7f8f501f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.012+0000 7f8f55407640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f500b3210 con 0x7f8f501561b0 2023-11-28T17:41:52.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.012+0000 7f8f55407640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f50155d20 con 0x7f8f501f2e50 2023-11-28T17:41:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.012+0000 7f8f55407640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f500b4e50 con 0x7f8f3c0039a0 2023-11-28T17:41:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.012+0000 7f8f467fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f501561b0 0x7f8f501565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.012+0000 7f8f467fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f501561b0 0x7f8f501565f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57020/0 (socket says 172.21.15.98:57020) 2023-11-28T17:41:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.012+0000 7f8f467fc640 1 -- 172.21.15.98:0/1731126043 learned_addr learned my addr 172.21.15.98:0/1731126043 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:52.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.012+0000 7f8f4effd640 1 --2- 172.21.15.98:0/1731126043 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f3c0039a0 0x7f8f50156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.012+0000 7f8f4f7fe640 1 --2- 172.21.15.98:0/1731126043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f501f2e50 0x7f8f501f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.012+0000 7f8f467fc640 1 -- 172.21.15.98:0/1731126043 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f3c0039a0 msgr2=0x7f8f50156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.013+0000 7f8f467fc640 1 --2- 172.21.15.98:0/1731126043 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f3c0039a0 0x7f8f50156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.013+0000 7f8f467fc640 1 -- 172.21.15.98:0/1731126043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f501f2e50 msgr2=0x7f8f501f32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:52.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.013+0000 7f8f467fc640 1 --2- 172.21.15.98:0/1731126043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f501f2e50 0x7f8f501f32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.013+0000 7f8f467fc640 1 -- 172.21.15.98:0/1731126043 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f501f39e0 con 0x7f8f501561b0 2023-11-28T17:41:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.013+0000 7f8f4f7fe640 1 --2- 172.21.15.98:0/1731126043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f501f2e50 0x7f8f501f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.013+0000 7f8f467fc640 1 --2- 172.21.15.98:0/1731126043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f501561b0 0x7f8f501565f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800c500 tx=0x7f8f3800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.013+0000 7f8f47fff640 1 -- 172.21.15.98:0/1731126043 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f380052d0 con 0x7f8f501561b0 2023-11-28T17:41:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.013+0000 7f8f47fff640 1 -- 172.21.15.98:0/1731126043 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f38005430 con 0x7f8f501561b0 2023-11-28T17:41:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.013+0000 7f8f47fff640 1 -- 172.21.15.98:0/1731126043 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38012870 con 0x7f8f501561b0 2023-11-28T17:41:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.013+0000 7f8f55407640 1 -- 172.21.15.98:0/1731126043 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f501e6ac0 con 0x7f8f501561b0 2023-11-28T17:41:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.013+0000 7f8f55407640 1 -- 172.21.15.98:0/1731126043 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f501e6d60 con 0x7f8f501561b0 2023-11-28T17:41:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.015+0000 7f8f55407640 1 -- 172.21.15.98:0/1731126043 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f50074f60 con 0x7f8f501561b0 2023-11-28T17:41:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.016+0000 7f8f47fff640 1 -- 172.21.15.98:0/1731126043 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f8f380129d0 con 0x7f8f501561b0 2023-11-28T17:41:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.017+0000 7f8f47fff640 1 --2- 172.21.15.98:0/1731126043 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f8f3006b2a0 0x7f8f3006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.017+0000 7f8f47fff640 1 -- 172.21.15.98:0/1731126043 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f38089790 con 0x7f8f501561b0 2023-11-28T17:41:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.017+0000 7f8f4effd640 1 --2- 172.21.15.98:0/1731126043 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f8f3006b2a0 0x7f8f3006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.018+0000 7f8f4effd640 1 --2- 172.21.15.98:0/1731126043 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f8f3006b2a0 0x7f8f3006d760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8f40036000 tx=0x7f8f4002e000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:52.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.021+0000 7f8f47fff640 1 -- 172.21.15.98:0/1731126043 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8f38054690 con 0x7f8f501561b0 2023-11-28T17:41:52.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.252+0000 7f8f55407640 1 -- 172.21.15.98:0/1731126043 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8f50155e60 con 0x7f8f501561b0 2023-11-28T17:41:52.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.253+0000 7f8f47fff640 1 -- 172.21.15.98:0/1731126043 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8f38057cb0 con 0x7f8f501561b0 2023-11-28T17:41:52.254 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T17:41:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.257+0000 7f8f55407640 1 -- 172.21.15.98:0/1731126043 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f8f3006b2a0 msgr2=0x7f8f3006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.257+0000 7f8f55407640 1 --2- 172.21.15.98:0/1731126043 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f8f3006b2a0 0x7f8f3006d760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8f40036000 tx=0x7f8f4002e000 comp rx=0 tx=0).stop 2023-11-28T17:41:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.257+0000 7f8f55407640 1 -- 172.21.15.98:0/1731126043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f501561b0 msgr2=0x7f8f501565f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.257+0000 7f8f55407640 1 --2- 172.21.15.98:0/1731126043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f501561b0 0x7f8f501565f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800c500 tx=0x7f8f3800c9c0 comp rx=0 tx=0).stop 2023-11-28T17:41:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.257+0000 7f8f55407640 1 -- 172.21.15.98:0/1731126043 shutdown_connections 2023-11-28T17:41:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.257+0000 7f8f55407640 1 --2- 172.21.15.98:0/1731126043 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f8f3006b2a0 0x7f8f3006d760 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.257+0000 7f8f55407640 1 --2- 172.21.15.98:0/1731126043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8f501f2e50 0x7f8f501f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.257+0000 7f8f55407640 1 --2- 172.21.15.98:0/1731126043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f501561b0 0x7f8f501565f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.257+0000 7f8f55407640 1 --2- 172.21.15.98:0/1731126043 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8f3c0039a0 0x7f8f50156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.257+0000 7f8f55407640 1 -- 172.21.15.98:0/1731126043 >> 172.21.15.98:0/1731126043 conn(0x7f8f50066410 msgr2=0x7f8f50152270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:52.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.257+0000 7f8f55407640 1 -- 172.21.15.98:0/1731126043 shutdown_connections 2023-11-28T17:41:52.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:52.257+0000 7f8f55407640 1 -- 172.21.15.98:0/1731126043 wait complete. 2023-11-28T17:41:52.275 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T17:41:52.275 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T17:41:52.276 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2023-11-28T17:41:52.276 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T17:41:52.276 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T17:41:52.276 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T17:41:52.276 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T17:41:52.276 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T17:41:52.276 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T17:41:52.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:52.276 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2023-11-28T17:41:52.331 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T17:41:52.331 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T17:41:52.331 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T17:41:52.332 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:41:52.332 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-11-28T17:41:52.332 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:41:52.332 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:41:52.332 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:41:52.332 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2023-11-28T17:41:52.332 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:41:52.333 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:41:52.333 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2023-11-28T17:41:52.333 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:41:52.333 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2023-11-28T17:41:52.333 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T17:41:52.333 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2023-11-28T17:41:52.333 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T17:41:52.333 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42366sec preferred_lft 42366sec 2023-11-28T17:41:52.334 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2023-11-28T17:41:52.334 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:41:52.334 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-28T17:41:52.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:52.335 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-11-28T17:41:52.335 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-11-28T17:41:52.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2023-11-28T17:41:52.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2023-11-28T17:41:52.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2023-11-28T17:41:52.335 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-28T17:41:52.336 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-11-28T17:41:52.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:52.818 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T17:41:52.897 INFO:teuthology.orchestra.run.smithi098.stdout:1 2023-11-28T17:41:52.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:52.901 DEBUG:teuthology.orchestra.run.smithi098:> route 2023-11-28T17:41:53.061 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2023-11-28T17:41:53.061 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T17:41:53.061 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T17:41:53.061 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T17:41:53.062 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T17:41:53.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:53.063 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-11-28T17:41:53.063 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-11-28T17:41:53.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T17:41:53.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T17:41:53.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T17:41:53.063 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-11-28T17:41:53.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:53.819 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-11-28T17:41:53.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:53.876 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2023-11-28T17:41:53.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:53.929 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-11-28T17:41:53.929 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-11-28T17:41:53.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:41:53.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-11-28T17:41:53.930 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-11-28T17:41:54.079 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-11-28T17:41:54.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:54.079 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-11-28T17:41:54.079 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-11-28T17:41:54.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-11-28T17:41:54.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-28T17:41:54.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-11-28T17:41:54.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-11-28T17:41:54.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-11-28T17:41:54.080 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-11-28T17:41:54.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:54.360 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-11-28T17:41:54.360 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-11-28T17:41:54.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2023-11-28T17:41:54.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2023-11-28T17:41:54.360 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-11-28T17:41:54.524 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-11-28T17:41:54.524 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:41:54.524 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T17:41:54.571 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T17:41:54.571 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:41:54.571 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T17:41:54.666 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-11-28T17:41:54.762 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:41:54.813 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2023-11-28T17:41:54.814 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2023-11-28T17:41:54.814 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2023-11-28T17:41:54.814 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2023-11-28T17:41:54.814 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2023-11-28T17:41:54.814 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2023-11-28T17:41:54.814 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2023-11-28T17:41:54.814 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2023-11-28T17:41:54.814 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2023-11-28T17:41:54.814 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2023-11-28T17:41:54.814 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/selinux 2023-11-28T17:41:54.814 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:41:54.815 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2023-11-28T17:41:54.815 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2023-11-28T17:41:54.815 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2023-11-28T17:41:54.815 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2023-11-28T17:41:54.815 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2023-11-28T17:41:54.815 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2023-11-28T17:41:54.815 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:41:54.815 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:41:54.815 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:41:54.815 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:41:54.815 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2023-11-28T17:41:54.815 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2023-11-28T17:41:54.816 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T17:41:54.816 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T17:41:54.816 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T17:41:54.816 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T17:41:54.816 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2023-11-28T17:41:54.816 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:41:54.816 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:41:54.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:54.817 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-11-28T17:41:54.866 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-28T17:41:54.867 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-11-28T17:41:54.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-11-28T17:41:54.950 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:41:54.996 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2023-11-28T17:41:54.996 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2023-11-28T17:41:54.996 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2023-11-28T17:41:54.997 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2023-11-28T17:41:54.997 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2023-11-28T17:41:54.997 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2023-11-28T17:41:54.997 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2023-11-28T17:41:54.997 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2023-11-28T17:41:54.997 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2023-11-28T17:41:54.998 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2023-11-28T17:41:54.998 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/selinux 2023-11-28T17:41:54.998 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:41:54.998 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2023-11-28T17:41:54.998 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2023-11-28T17:41:54.998 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2023-11-28T17:41:54.999 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2023-11-28T17:41:54.999 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2023-11-28T17:41:54.999 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2023-11-28T17:41:54.999 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:41:54.999 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:41:54.999 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:41:54.999 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:41:55.000 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2023-11-28T17:41:55.000 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2023-11-28T17:41:55.000 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T17:41:55.000 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T17:41:55.000 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T17:41:55.000 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T17:41:55.000 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2023-11-28T17:41:55.001 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:41:55.001 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:41:55.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:55.002 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-11-28T17:41:55.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:2023-11-28T17:41:55.065+0000 7f085edee480 -1 init, newargv = 0x562d3adae3c0 newargc=15 2023-11-28T17:41:55.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[95806]: starting ceph client 2023-11-28T17:41:55.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[95806]: starting fuse 2023-11-28T17:41:56.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-11-28T17:41:56.299 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:41:56.352 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2023-11-28T17:41:56.353 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2023-11-28T17:41:56.353 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2023-11-28T17:41:56.353 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2023-11-28T17:41:56.353 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2023-11-28T17:41:56.353 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2023-11-28T17:41:56.353 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2023-11-28T17:41:56.353 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2023-11-28T17:41:56.354 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2023-11-28T17:41:56.354 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2023-11-28T17:41:56.354 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/selinux 2023-11-28T17:41:56.354 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:41:56.354 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2023-11-28T17:41:56.354 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2023-11-28T17:41:56.355 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2023-11-28T17:41:56.355 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2023-11-28T17:41:56.355 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2023-11-28T17:41:56.356 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2023-11-28T17:41:56.356 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:41:56.356 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:41:56.356 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:41:56.356 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:41:56.356 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2023-11-28T17:41:56.357 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2023-11-28T17:41:56.357 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T17:41:56.357 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T17:41:56.357 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T17:41:56.357 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T17:41:56.357 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2023-11-28T17:41:56.357 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:41:56.358 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:41:56.358 INFO:teuthology.orchestra.run.smithi098.stdout:/home/ubuntu/cephtest/mnt.0 2023-11-28T17:41:56.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:56.359 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-11-28T17:41:56.410 INFO:teuthology.orchestra.run.smithi098.stdout:39 2023-11-28T17:41:56.411 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2023-11-28T17:41:56.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo stdin-killer -- python3 -c ' 2023-11-28T17:41:56.411 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2023-11-28T17:41:56.411 DEBUG:teuthology.orchestra.run.smithi098:> import re 2023-11-28T17:41:56.411 DEBUG:teuthology.orchestra.run.smithi098:> import os 2023-11-28T17:41:56.411 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2023-11-28T17:41:56.411 DEBUG:teuthology.orchestra.run.smithi098:> 2023-11-28T17:41:56.411 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2023-11-28T17:41:56.411 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-11-28T17:41:56.411 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2023-11-28T17:41:56.411 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-11-28T17:41:56.412 DEBUG:teuthology.orchestra.run.smithi098:> 2023-11-28T17:41:56.412 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2023-11-28T17:41:56.412 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2023-11-28T17:41:56.412 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2023-11-28T17:41:56.412 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2023-11-28T17:41:56.412 DEBUG:teuthology.orchestra.run.smithi098:> 2023-11-28T17:41:56.412 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2023-11-28T17:41:56.412 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-28T17:41:56.412 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-28T17:41:56.412 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-28T17:41:56.412 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2023-11-28T17:41:56.413 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2023-11-28T17:41:56.413 DEBUG:teuthology.orchestra.run.smithi098:> return f 2023-11-28T17:41:56.413 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-28T17:41:56.413 DEBUG:teuthology.orchestra.run.smithi098:> 2023-11-28T17:41:56.413 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.0")) 2023-11-28T17:41:56.413 DEBUG:teuthology.orchestra.run.smithi098:> ' 2023-11-28T17:41:56.591 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.0.95806.asok 2023-11-28T17:41:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-11-28T17:41:56.609 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.95806.asok 2023-11-28T17:41:56.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:56.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.95806.asok status 2023-11-28T17:41:56.851 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:41:56.851 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2023-11-28T17:41:56.851 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "47c1c547372ed5bff7984647dc200bac0e019caf", 2023-11-28T17:41:56.851 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev)", 2023-11-28T17:41:56.851 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "0", 2023-11-28T17:41:56.851 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2023-11-28T17:41:56.851 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-11-28T17:41:56.852 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "95806", 2023-11-28T17:41:56.852 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2023-11-28T17:41:56.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:41:56.852 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2023-11-28T17:41:56.852 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2023-11-28T17:41:56.852 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4737, 2023-11-28T17:41:56.852 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2023-11-28T17:41:56.852 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2023-11-28T17:41:56.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2023-11-28T17:41:56.852 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4737 2023-11-28T17:41:56.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:41:56.852 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-11-28T17:41:56.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:41:56.853 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2023-11-28T17:41:56.853 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2619348234 2023-11-28T17:41:56.853 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:41:56.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:41:56.853 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-11-28T17:41:56.853 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:41:56.853 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2023-11-28T17:41:56.853 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2619348234 2023-11-28T17:41:56.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:41:56.853 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.4737 172.21.15.98:0/2619348234", 2023-11-28T17:41:56.853 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "172.21.15.98:0/2619348234", 2023-11-28T17:41:56.853 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2023-11-28T17:41:56.854 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 16, 2023-11-28T17:41:56.854 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 22, 2023-11-28T17:41:56.854 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2023-11-28T17:41:56.854 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2023-11-28T17:41:56.854 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2023-11-28T17:41:56.854 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:41:56.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:56.866 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2023-11-28T17:41:56.867 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.0 2023-11-28T17:41:56.918 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2023-11-28T17:41:56.919 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-11-28T17:41:56.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:41:56.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T17:41:57.017 INFO:teuthology.run_tasks:Running task mon_thrash... 2023-11-28T17:41:57.032 INFO:tasks.mon_thrash:Beginning mon_thrash... 2023-11-28T17:41:57.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:41:57.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.252+0000 7f10affff640 1 Processor -- start 2023-11-28T17:41:57.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.252+0000 7f10affff640 1 -- start start 2023-11-28T17:41:57.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.254+0000 7f10affff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10b006b1e0 0x7f10b0156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:57.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.254+0000 7f10affff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f10b00c71f0 0x7f10b0157130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:57.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.254+0000 7f10affff640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b0150ea0 con 0x7f10b0150ac0 2023-11-28T17:41:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.254+0000 7f10affff640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b015ad90 con 0x7f10b00c71f0 2023-11-28T17:41:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.254+0000 7f10affff640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b01567a0 con 0x7f10b006b1e0 2023-11-28T17:41:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.254+0000 7f10af7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f10b00c71f0 0x7f10b0157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.254+0000 7f10af7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f10b00c71f0 0x7f10b0157130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57036/0 (socket says 172.21.15.98:57036) 2023-11-28T17:41:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.254+0000 7f10af7fe640 1 -- 172.21.15.98:0/1853536427 learned_addr learned my addr 172.21.15.98:0/1853536427 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.255+0000 7f10a67fc640 1 --2- 172.21.15.98:0/1853536427 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10b006b1e0 0x7f10b0156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.255+0000 7f10af7fe640 1 -- 172.21.15.98:0/1853536427 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10b006b1e0 msgr2=0x7f10b0156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.255+0000 7f10af7fe640 1 --2- 172.21.15.98:0/1853536427 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10b006b1e0 0x7f10b0156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.255+0000 7f10af7fe640 1 -- 172.21.15.98:0/1853536427 >> v1:172.21.15.113:6789/0 conn(0x7f10b0150ac0 legacy=0x7f10b00716e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:57.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.255+0000 7f10af7fe640 1 -- 172.21.15.98:0/1853536427 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b0159f30 con 0x7f10b00c71f0 2023-11-28T17:41:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.256+0000 7f10af7fe640 1 --2- 172.21.15.98:0/1853536427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b00c71f0 0x7f10b0157130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f10a8009870 tx=0x7f10a802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e1cf020e358833 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.256+0000 7f10ae7fc640 1 -- 172.21.15.98:0/1853536427 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a802f830 con 0x7f10b00c71f0 2023-11-28T17:41:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.256+0000 7f10ae7fc640 1 -- 172.21.15.98:0/1853536427 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a802f990 con 0x7f10b00c71f0 2023-11-28T17:41:57.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.256+0000 7f10ae7fc640 1 -- 172.21.15.98:0/1853536427 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a802fd20 con 0x7f10b00c71f0 2023-11-28T17:41:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.257+0000 7f10affff640 1 -- 172.21.15.98:0/1853536427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b00c71f0 msgr2=0x7f10b0157130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.257+0000 7f10affff640 1 --2- 172.21.15.98:0/1853536427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b00c71f0 0x7f10b0157130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f10a8009870 tx=0x7f10a802ba10 comp rx=0 tx=0).stop 2023-11-28T17:41:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.257+0000 7f10affff640 1 -- 172.21.15.98:0/1853536427 shutdown_connections 2023-11-28T17:41:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.257+0000 7f10affff640 1 --2- 172.21.15.98:0/1853536427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b00c71f0 0x7f10b0157130 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.257+0000 7f10affff640 1 --2- 172.21.15.98:0/1853536427 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10b006b1e0 0x7f10b0156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.257+0000 7f10affff640 1 -- 172.21.15.98:0/1853536427 >> 172.21.15.98:0/1853536427 conn(0x7f10b0066230 msgr2=0x7f10b006a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.257+0000 7f10affff640 1 -- 172.21.15.98:0/1853536427 shutdown_connections 2023-11-28T17:41:57.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.257+0000 7f10affff640 1 -- 172.21.15.98:0/1853536427 wait complete. 2023-11-28T17:41:57.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.259+0000 7f10affff640 1 Processor -- start 2023-11-28T17:41:57.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.259+0000 7f10affff640 1 -- start start 2023-11-28T17:41:57.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b006b1e0 0x7f10b01ef430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:57.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10affff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10b00c71f0 0x7f10b0145ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:57.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10affff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10b0150ac0 0x7f10b013e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:57.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10affff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b0159a50 con 0x7f10b006b1e0 2023-11-28T17:41:57.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10affff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b00af000 con 0x7f10b0150ac0 2023-11-28T17:41:57.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10affff640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b01ef970 con 0x7f10b00c71f0 2023-11-28T17:41:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10aeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b006b1e0 0x7f10b01ef430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10aeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b006b1e0 0x7f10b01ef430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57040/0 (socket says 172.21.15.98:57040) 2023-11-28T17:41:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10aeffd640 1 -- 172.21.15.98:0/704246840 learned_addr learned my addr 172.21.15.98:0/704246840 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10a67fc640 1 --2- 172.21.15.98:0/704246840 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10b00c71f0 0x7f10b0145ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10aeffd640 1 -- 172.21.15.98:0/704246840 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10b00c71f0 msgr2=0x7f10b0145ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10af7fe640 1 --2- 172.21.15.98:0/704246840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10b0150ac0 0x7f10b013e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10aeffd640 1 --2- 172.21.15.98:0/704246840 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10b00c71f0 0x7f10b0145ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10aeffd640 1 -- 172.21.15.98:0/704246840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10b0150ac0 msgr2=0x7f10b013e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10aeffd640 1 --2- 172.21.15.98:0/704246840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10b0150ac0 0x7f10b013e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.260+0000 7f10aeffd640 1 -- 172.21.15.98:0/704246840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a80094d0 con 0x7f10b006b1e0 2023-11-28T17:41:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.261+0000 7f10aeffd640 1 --2- 172.21.15.98:0/704246840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b006b1e0 0x7f10b01ef430 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f10a000c760 tx=0x7f10a000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.261+0000 7f10a7fff640 1 -- 172.21.15.98:0/704246840 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a0003b70 con 0x7f10b006b1e0 2023-11-28T17:41:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.261+0000 7f10a7fff640 1 -- 172.21.15.98:0/704246840 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a0003cf0 con 0x7f10b006b1e0 2023-11-28T17:41:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.261+0000 7f10a7fff640 1 -- 172.21.15.98:0/704246840 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a000aa90 con 0x7f10b006b1e0 2023-11-28T17:41:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.261+0000 7f10affff640 1 -- 172.21.15.98:0/704246840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b00ad9d0 con 0x7f10b006b1e0 2023-11-28T17:41:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.261+0000 7f10affff640 1 -- 172.21.15.98:0/704246840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b0159770 con 0x7f10b006b1e0 2023-11-28T17:41:57.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.262+0000 7f10affff640 1 -- 172.21.15.98:0/704246840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b0074db0 con 0x7f10b006b1e0 2023-11-28T17:41:57.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.263+0000 7f10a7fff640 1 -- 172.21.15.98:0/704246840 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f10a000abf0 con 0x7f10b006b1e0 2023-11-28T17:41:57.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.264+0000 7f10a7fff640 1 --2- 172.21.15.98:0/704246840 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f109006b2a0 0x7f109006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:57.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.264+0000 7f10a7fff640 1 -- 172.21.15.98:0/704246840 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10a0089a80 con 0x7f10b006b1e0 2023-11-28T17:41:57.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.264+0000 7f10a67fc640 1 --2- 172.21.15.98:0/704246840 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f109006b2a0 0x7f109006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:57.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.265+0000 7f10a67fc640 1 --2- 172.21.15.98:0/704246840 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f109006b2a0 0x7f109006d760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f10b01558d0 tx=0x7f1098008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:57.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.269+0000 7f10a7fff640 1 -- 172.21.15.98:0/704246840 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10a0054860 con 0x7f10b006b1e0 2023-11-28T17:41:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.462+0000 7f10affff640 1 -- 172.21.15.98:0/704246840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10b0142990 con 0x7f10b006b1e0 2023-11-28T17:41:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.465+0000 7f10a7fff640 1 -- 172.21.15.98:0/704246840 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16517 (secure 0 0 0) 0x7f10a0003e60 con 0x7f10b006b1e0 2023-11-28T17:41:57.466 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:57.466 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:41:18.195933+0000","last_up_change":"2023-11-28T17:41:09.556993+0000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:41:10.303280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:41:14.890871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:41:15.704103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6810","nonce":2188704385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6813","nonce":2188704385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6819","nonce":2188704385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6816","nonce":2188704385}]},"public_addr":"172.21.15.98:6810/2188704385","cluster_addr":"172.21.15.98:6813/2188704385","heartbeat_back_addr":"172.21.15.98:6819/2188704385","heartbeat_front_addr":"172.21.15.98:6816/2188704385","state":["exists","up"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6801","nonce":3918226860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6803","nonce":3918226860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6807","nonce":3918226860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6805","nonce":3918226860}]},"public_addr":"172.21.15.98:6801/3918226860","cluster_addr":"172.21.15.98:6803/3918226860","heartbeat_back_addr":"172.21.15.98:6807/3918226860","heartbeat_front_addr":"172.21.15.98:6805/3918226860","state":["exists","up"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6812","nonce":2691532400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6815","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6818","nonce":2691532400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6823","nonce":2691532400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6821","nonce":2691532400}]},"public_addr":"172.21.15.98:6812/2691532400","cluster_addr":"172.21.15.98:6818/2691532400","heartbeat_back_addr":"172.21.15.98:6823/2691532400","heartbeat_front_addr":"172.21.15.98:6821/2691532400","state":["exists","up"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6825","nonce":1905487008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6827","nonce":1905487008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6831","nonce":1905487008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6829","nonce":1905487008}]},"public_addr":"172.21.15.98:6825/1905487008","cluster_addr":"172.21.15.98:6827/1905487008","heartbeat_back_addr":"172.21.15.98:6831/1905487008","heartbeat_front_addr":"172.21.15.98:6829/1905487008","state":["exists","up"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6801","nonce":2974677215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6803","nonce":2974677215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6807","nonce":2974677215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6805","nonce":2974677215}]},"public_addr":"172.21.15.113:6801/2974677215","cluster_addr":"172.21.15.113:6803/2974677215","heartbeat_back_addr":"172.21.15.113:6807/2974677215","heartbeat_front_addr":"172.21.15.113:6805/2974677215","state":["exists","up"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6817","nonce":2685000355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6821","nonce":2685000355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6829","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6831","nonce":2685000355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6823","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6826","nonce":2685000355}]},"public_addr":"172.21.15.113:6817/2685000355","cluster_addr":"172.21.15.113:6821/2685000355","heartbeat_back_addr":"172.21.15.113:6831/2685000355","heartbeat_front_addr":"172.21.15.113:6826/2685000355","state":["exists","up"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6809","nonce":2764300831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6811","nonce":2764300831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6815","nonce":2764300831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6813","nonce":2764300831}]},"public_addr":"172.21.15.113:6809/2764300831","cluster_addr":"172.21.15.113:6811/2764300831","heartbeat_back_addr":"172.21.15.113:6815/2764300831","heartbeat_front_addr":"172.21.15.113:6813/2764300831","state":["exists","up"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6819","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6820","nonce":1656248400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6824","nonce":1656248400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6830","nonce":1656248400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6825","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6827","nonce":1656248400}]},"public_addr":"172.21.15.113:6820/1656248400","cluster_addr":"172.21.15.113:6824/1656248400","heartbeat_back_addr":"172.21.15.113:6830/1656248400","heartbeat_front_addr":"172.21.15.113:6827/1656248400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.379549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.312196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.117494+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.388128+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.360608+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.411101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.355995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.390362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:41:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.467+0000 7f10affff640 1 -- 172.21.15.98:0/704246840 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f109006b2a0 msgr2=0x7f109006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.467+0000 7f10affff640 1 --2- 172.21.15.98:0/704246840 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f109006b2a0 0x7f109006d760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f10b01558d0 tx=0x7f1098008040 comp rx=0 tx=0).stop 2023-11-28T17:41:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.467+0000 7f10affff640 1 -- 172.21.15.98:0/704246840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b006b1e0 msgr2=0x7f10b01ef430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.467+0000 7f10affff640 1 --2- 172.21.15.98:0/704246840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b006b1e0 0x7f10b01ef430 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f10a000c760 tx=0x7f10a000cc20 comp rx=0 tx=0).stop 2023-11-28T17:41:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.468+0000 7f10affff640 1 -- 172.21.15.98:0/704246840 shutdown_connections 2023-11-28T17:41:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.468+0000 7f10affff640 1 --2- 172.21.15.98:0/704246840 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f109006b2a0 0x7f109006d760 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.468+0000 7f10affff640 1 --2- 172.21.15.98:0/704246840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10b0150ac0 0x7f10b013e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.468+0000 7f10affff640 1 --2- 172.21.15.98:0/704246840 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f10b00c71f0 0x7f10b0145ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.468+0000 7f10affff640 1 --2- 172.21.15.98:0/704246840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10b006b1e0 0x7f10b01ef430 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.468+0000 7f10affff640 1 -- 172.21.15.98:0/704246840 >> 172.21.15.98:0/704246840 conn(0x7f10b0066230 msgr2=0x7f10b00c6110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.468+0000 7f10affff640 1 -- 172.21.15.98:0/704246840 shutdown_connections 2023-11-28T17:41:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.468+0000 7f10affff640 1 -- 172.21.15.98:0/704246840 wait complete. 2023-11-28T17:41:57.488 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:41:10.303280+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:41:14.890871+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:41:15.704103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:41:57.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:41:57.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.696+0000 7f4fadc51640 1 Processor -- start 2023-11-28T17:41:57.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.697+0000 7f4fadc51640 1 -- start start 2023-11-28T17:41:57.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.698+0000 7f4fadc51640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4fa8153fb0 0x7f4fa80c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:57.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.698+0000 7f4fadc51640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4fa80b5270 0x7f4fa80ca300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:57.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.698+0000 7f4fadc51640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa815db80 con 0x7f4fa80ca840 2023-11-28T17:41:57.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.698+0000 7f4fadc51640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa80cb730 con 0x7f4fa8153fb0 2023-11-28T17:41:57.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.698+0000 7f4fadc51640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa815e0c0 con 0x7f4fa80b5270 2023-11-28T17:41:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.698+0000 7f4fa7fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4fa80ca840 0x7f4fa80c76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46892/0 (socket says 172.21.15.98:46892) 2023-11-28T17:41:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.698+0000 7f4fa7fff640 1 -- 172.21.15.98:0/2170440978 learned_addr learned my addr 172.21.15.98:0/2170440978 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.698+0000 7f4fa77fe640 1 --2- 172.21.15.98:0/2170440978 >> v2:172.21.15.98:3300/0 conn(0x7f4fa8153fb0 0x7f4fa80c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.699+0000 7f4fa6ffd640 1 --2- 172.21.15.98:0/2170440978 >> v2:172.21.15.113:3300/0 conn(0x7f4fa80b5270 0x7f4fa80ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.699+0000 7f4fa77fe640 1 -- 172.21.15.98:0/2170440978 >> v2:172.21.15.113:3300/0 conn(0x7f4fa80b5270 msgr2=0x7f4fa80ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.699+0000 7f4fa77fe640 1 --2- 172.21.15.98:0/2170440978 >> v2:172.21.15.113:3300/0 conn(0x7f4fa80b5270 0x7f4fa80ca300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.699+0000 7f4fa77fe640 1 -- 172.21.15.98:0/2170440978 >> v1:172.21.15.98:6789/0 conn(0x7f4fa80ca840 legacy=0x7f4fa80c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.699+0000 7f4fa77fe640 1 -- 172.21.15.98:0/2170440978 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa80d00a0 con 0x7f4fa8153fb0 2023-11-28T17:41:57.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.699+0000 7f4fa67fc640 1 -- 172.21.15.98:0/2170440978 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169626979 0 0) 0x7f4fa815db80 con 0x7f4fa80ca840 2023-11-28T17:41:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.700+0000 7f4fa77fe640 1 --2- 172.21.15.98:0/2170440978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fa8153fb0 0x7f4fa80c6ad0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4fa80c2fa0 tx=0x7f4f9802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b4584479505015 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.700+0000 7f4fa67fc640 1 -- 172.21.15.98:0/2170440978 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9802cbf0 con 0x7f4fa8153fb0 2023-11-28T17:41:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.700+0000 7f4fa67fc640 1 -- 172.21.15.98:0/2170440978 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9802cd50 con 0x7f4fa8153fb0 2023-11-28T17:41:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.700+0000 7f4fa67fc640 1 -- 172.21.15.98:0/2170440978 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f98034660 con 0x7f4fa8153fb0 2023-11-28T17:41:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.701+0000 7f4fadc51640 1 -- 172.21.15.98:0/2170440978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fa8153fb0 msgr2=0x7f4fa80c6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:57.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.701+0000 7f4fadc51640 1 --2- 172.21.15.98:0/2170440978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fa8153fb0 0x7f4fa80c6ad0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4fa80c2fa0 tx=0x7f4f9802be40 comp rx=0 tx=0).stop 2023-11-28T17:41:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.701+0000 7f4fadc51640 1 -- 172.21.15.98:0/2170440978 shutdown_connections 2023-11-28T17:41:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.701+0000 7f4fadc51640 1 --2- 172.21.15.98:0/2170440978 >> v2:172.21.15.113:3300/0 conn(0x7f4fa80b5270 0x7f4fa80ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.701+0000 7f4fadc51640 1 --2- 172.21.15.98:0/2170440978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fa8153fb0 0x7f4fa80c6ad0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.701+0000 7f4fadc51640 1 -- 172.21.15.98:0/2170440978 >> 172.21.15.98:0/2170440978 conn(0x7f4fa80cf4a0 msgr2=0x7f4fa80cf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.701+0000 7f4fadc51640 1 -- 172.21.15.98:0/2170440978 shutdown_connections 2023-11-28T17:41:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.701+0000 7f4fadc51640 1 -- 172.21.15.98:0/2170440978 wait complete. 2023-11-28T17:41:57.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.703+0000 7f4fadc51640 1 Processor -- start 2023-11-28T17:41:57.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.703+0000 7f4fadc51640 1 -- start start 2023-11-28T17:41:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.704+0000 7f4fadc51640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4fa80ca840 0x7f4fa80c2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.704+0000 7f4fadc51640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4fa8153fb0 0x7f4fa8158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.704+0000 7f4fadc51640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fa815ad90 0x7f4fa815b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.704+0000 7f4fadc51640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa8158aa0 con 0x7f4fa815ad90 2023-11-28T17:41:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.704+0000 7f4fadc51640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa815a8a0 con 0x7f4fa8153fb0 2023-11-28T17:41:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.704+0000 7f4fadc51640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa80c2e80 con 0x7f4fa80ca840 2023-11-28T17:41:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.704+0000 7f4fa7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fa815ad90 0x7f4fa815b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.704+0000 7f4fa7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fa815ad90 0x7f4fa815b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57062/0 (socket says 172.21.15.98:57062) 2023-11-28T17:41:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.704+0000 7f4fa7fff640 1 -- 172.21.15.98:0/3916405007 learned_addr learned my addr 172.21.15.98:0/3916405007 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.705+0000 7f4fa77fe640 1 --2- 172.21.15.98:0/3916405007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4fa80ca840 0x7f4fa80c2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.705+0000 7f4fa6ffd640 1 --2- 172.21.15.98:0/3916405007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4fa8153fb0 0x7f4fa8158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:57.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.705+0000 7f4fa7fff640 1 -- 172.21.15.98:0/3916405007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4fa80ca840 msgr2=0x7f4fa80c2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:57.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.705+0000 7f4fa7fff640 1 --2- 172.21.15.98:0/3916405007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4fa80ca840 0x7f4fa80c2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.705+0000 7f4fa7fff640 1 -- 172.21.15.98:0/3916405007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4fa8153fb0 msgr2=0x7f4fa8158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:57.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.705+0000 7f4fa7fff640 1 --2- 172.21.15.98:0/3916405007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4fa8153fb0 0x7f4fa8158f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.705+0000 7f4fa7fff640 1 -- 172.21.15.98:0/3916405007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f98009a70 con 0x7f4fa815ad90 2023-11-28T17:41:57.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.705+0000 7f4fa77fe640 1 --2- 172.21.15.98:0/3916405007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4fa80ca840 0x7f4fa80c2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:57.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.705+0000 7f4fa7fff640 1 --2- 172.21.15.98:0/3916405007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fa815ad90 0x7f4fa815b6f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4fa000c480 tx=0x7f4fa000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:57.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.705+0000 7f4f9ffff640 1 -- 172.21.15.98:0/3916405007 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0005ce0 con 0x7f4fa815ad90 2023-11-28T17:41:57.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.705+0000 7f4f9ffff640 1 -- 172.21.15.98:0/3916405007 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa0005e60 con 0x7f4fa815ad90 2023-11-28T17:41:57.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.705+0000 7f4f9ffff640 1 -- 172.21.15.98:0/3916405007 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa00072b0 con 0x7f4fa815ad90 2023-11-28T17:41:57.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.705+0000 7f4fadc51640 1 -- 172.21.15.98:0/3916405007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa8157b40 con 0x7f4fa815ad90 2023-11-28T17:41:57.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.706+0000 7f4fadc51640 1 -- 172.21.15.98:0/3916405007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa8158450 con 0x7f4fa815ad90 2023-11-28T17:41:57.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.707+0000 7f4fadc51640 1 -- 172.21.15.98:0/3916405007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa8158c50 con 0x7f4fa815ad90 2023-11-28T17:41:57.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.707+0000 7f4f9ffff640 1 -- 172.21.15.98:0/3916405007 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4fa0007410 con 0x7f4fa815ad90 2023-11-28T17:41:57.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.708+0000 7f4f9ffff640 1 --2- 172.21.15.98:0/3916405007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f8806b2d0 0x7f4f8806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:57.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.708+0000 7f4f9ffff640 1 -- 172.21.15.98:0/3916405007 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fa0089170 con 0x7f4fa815ad90 2023-11-28T17:41:57.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.709+0000 7f4fa77fe640 1 --2- 172.21.15.98:0/3916405007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f8806b2d0 0x7f4f8806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:57.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.709+0000 7f4fa77fe640 1 --2- 172.21.15.98:0/3916405007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f8806b2d0 0x7f4f8806d790 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4f98036000 tx=0x7f4f98030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:57.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.712+0000 7f4f9ffff640 1 -- 172.21.15.98:0/3916405007 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4fa8158c50 con 0x7f4fa815ad90 2023-11-28T17:41:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.903+0000 7f4fadc51640 1 -- 172.21.15.98:0/3916405007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4fa8158c50 con 0x7f4fa815ad90 2023-11-28T17:41:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.903+0000 7f4f9ffff640 1 -- 172.21.15.98:0/3916405007 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4fa8158c50 con 0x7f4fa815ad90 2023-11-28T17:41:57.904 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-11-28T17:41:57.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.906+0000 7f4fadc51640 1 -- 172.21.15.98:0/3916405007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f8806b2d0 msgr2=0x7f4f8806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:57.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.906+0000 7f4fadc51640 1 --2- 172.21.15.98:0/3916405007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f8806b2d0 0x7f4f8806d790 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4f98036000 tx=0x7f4f98030000 comp rx=0 tx=0).stop 2023-11-28T17:41:57.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.906+0000 7f4fadc51640 1 -- 172.21.15.98:0/3916405007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fa815ad90 msgr2=0x7f4fa815b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:57.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.906+0000 7f4fadc51640 1 --2- 172.21.15.98:0/3916405007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fa815ad90 0x7f4fa815b6f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4fa000c480 tx=0x7f4fa000c940 comp rx=0 tx=0).stop 2023-11-28T17:41:57.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.906+0000 7f4fadc51640 1 -- 172.21.15.98:0/3916405007 shutdown_connections 2023-11-28T17:41:57.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.906+0000 7f4fadc51640 1 --2- 172.21.15.98:0/3916405007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4fa815ad90 0x7f4fa815b6f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.906+0000 7f4fadc51640 1 --2- 172.21.15.98:0/3916405007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f8806b2d0 0x7f4f8806d790 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.906+0000 7f4fadc51640 1 --2- 172.21.15.98:0/3916405007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4fa8153fb0 0x7f4fa8158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.906+0000 7f4fadc51640 1 --2- 172.21.15.98:0/3916405007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4fa80ca840 0x7f4fa80c2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:57.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.906+0000 7f4fadc51640 1 -- 172.21.15.98:0/3916405007 >> 172.21.15.98:0/3916405007 conn(0x7f4fa80cf4a0 msgr2=0x7f4fa80cbbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:57.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.906+0000 7f4fadc51640 1 -- 172.21.15.98:0/3916405007 shutdown_connections 2023-11-28T17:41:57.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:57.906+0000 7f4fadc51640 1 -- 172.21.15.98:0/3916405007 wait complete. 2023-11-28T17:41:57.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:41:58.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.130+0000 7f71feada640 1 Processor -- start 2023-11-28T17:41:58.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.130+0000 7f71feada640 1 -- start start 2023-11-28T17:41:58.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.131+0000 7f71feada640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f71f8071b50 0x7f71f815e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.131+0000 7f71feada640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f80c6630 con 0x7f71f80c5a60 2023-11-28T17:41:58.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.131+0000 7f71feada640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f806e710 con 0x7f71f8071770 2023-11-28T17:41:58.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.131+0000 7f71feada640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f806de70 con 0x7f71f8071b50 2023-11-28T17:41:58.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.132+0000 7f71fd077640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f71f80c5a60 0x7f71f8168a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46906/0 (socket says 172.21.15.98:46906) 2023-11-28T17:41:58.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.132+0000 7f71fd077640 1 -- 172.21.15.98:0/3107980515 learned_addr learned my addr 172.21.15.98:0/3107980515 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:58.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.132+0000 7f71f77fe640 1 -- 172.21.15.98:0/3107980515 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292706054 0 0) 0x7f71f80c6630 con 0x7f71f80c5a60 2023-11-28T17:41:58.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.132+0000 7f71f77fe640 1 -- 172.21.15.98:0/3107980515 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71e8003520 con 0x7f71f80c5a60 2023-11-28T17:41:58.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.132+0000 7f71f7fff640 1 --2- 172.21.15.98:0/3107980515 >> v2:172.21.15.113:3300/0 conn(0x7f71f8071b50 0x7f71f815e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71f77fe640 1 -- 172.21.15.98:0/3107980515 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 395267776 0 0) 0x7f71e8003520 con 0x7f71f80c5a60 2023-11-28T17:41:58.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71f77fe640 1 -- 172.21.15.98:0/3107980515 >> v2:172.21.15.113:3300/0 conn(0x7f71f8071b50 msgr2=0x7f71f815e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:58.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71f77fe640 1 --2- 172.21.15.98:0/3107980515 >> v2:172.21.15.113:3300/0 conn(0x7f71f8071b50 0x7f71f815e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71f77fe640 1 -- 172.21.15.98:0/3107980515 >> v1:172.21.15.113:6789/0 conn(0x7f71f8071770 legacy=0x7f71f8157fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71f77fe640 1 -- 172.21.15.98:0/3107980515 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f806e150 con 0x7f71f80c5a60 2023-11-28T17:41:58.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71f77fe640 1 -- 172.21.15.98:0/3107980515 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f71f00019f0 con 0x7f71f80c5a60 2023-11-28T17:41:58.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71f77fe640 1 -- 172.21.15.98:0/3107980515 >> v1:172.21.15.98:6789/0 conn(0x7f71f80c5a60 legacy=0x7f71f8168a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71f77fe640 1 --2- 172.21.15.98:0/3107980515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71e80039c0 0x7f71e8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71f77fe640 1 --2- 172.21.15.98:0/3107980515 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71e80047c0 0x7f71e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71f77fe640 1 --2- 172.21.15.98:0/3107980515 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71e80052a0 0x7f71e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71f77fe640 1 -- 172.21.15.98:0/3107980515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f806de70 con 0x7f71e80039c0 2023-11-28T17:41:58.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71f77fe640 1 -- 172.21.15.98:0/3107980515 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e800c1c0 con 0x7f71e80047c0 2023-11-28T17:41:58.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71f77fe640 1 -- 172.21.15.98:0/3107980515 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e800c380 con 0x7f71e80052a0 2023-11-28T17:41:58.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71fc876640 1 --2- 172.21.15.98:0/3107980515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71e80039c0 0x7f71e8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71fc876640 1 -- 172.21.15.98:0/3107980515 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71e80052a0 msgr2=0x7f71e800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:58.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71fc876640 1 --2- 172.21.15.98:0/3107980515 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71e80052a0 0x7f71e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71fc876640 1 -- 172.21.15.98:0/3107980515 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71e80047c0 msgr2=0x7f71e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:58.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71fc876640 1 --2- 172.21.15.98:0/3107980515 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71e80047c0 0x7f71e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71fc876640 1 -- 172.21.15.98:0/3107980515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e8003520 con 0x7f71e80039c0 2023-11-28T17:41:58.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.133+0000 7f71fd077640 1 -- 172.21.15.98:0/3107980515 reap_dead start 2023-11-28T17:41:58.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.134+0000 7f71fc876640 1 --2- 172.21.15.98:0/3107980515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71e80039c0 0x7f71e8003de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f71ec0015b0 tx=0x7f71e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc17de14d58e6352 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:58.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.134+0000 7f71f77fe640 1 -- 172.21.15.98:0/3107980515 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec0052d0 con 0x7f71e80039c0 2023-11-28T17:41:58.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.135+0000 7f71f77fe640 1 -- 172.21.15.98:0/3107980515 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ec02e050 con 0x7f71e80039c0 2023-11-28T17:41:58.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.135+0000 7f71f77fe640 1 -- 172.21.15.98:0/3107980515 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec003830 con 0x7f71e80039c0 2023-11-28T17:41:58.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.135+0000 7f71feada640 1 -- 172.21.15.98:0/3107980515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71e80039c0 msgr2=0x7f71e8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.135+0000 7f71feada640 1 --2- 172.21.15.98:0/3107980515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71e80039c0 0x7f71e8003de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f71ec0015b0 tx=0x7f71e80057f0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.135+0000 7f71feada640 1 -- 172.21.15.98:0/3107980515 shutdown_connections 2023-11-28T17:41:58.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.135+0000 7f71feada640 1 --2- 172.21.15.98:0/3107980515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71e80039c0 0x7f71e8003de0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.135+0000 7f71feada640 1 -- 172.21.15.98:0/3107980515 >> 172.21.15.98:0/3107980515 conn(0x7f71f806c0f0 msgr2=0x7f71f806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:58.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.135+0000 7f71feada640 1 -- 172.21.15.98:0/3107980515 shutdown_connections 2023-11-28T17:41:58.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.135+0000 7f71feada640 1 -- 172.21.15.98:0/3107980515 wait complete. 2023-11-28T17:41:58.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.137+0000 7f71feada640 1 Processor -- start 2023-11-28T17:41:58.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.137+0000 7f71feada640 1 -- start start 2023-11-28T17:41:58.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.138+0000 7f71feada640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71e80039c0 0x7f71f814cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.138+0000 7f71feada640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71f80c90d0 0x7f71f81f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.138+0000 7f71feada640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71f81f5f90 0x7f71f81f69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.138+0000 7f71feada640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f815b3f0 con 0x7f71f80c90d0 2023-11-28T17:41:58.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.138+0000 7f71feada640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f81f52e0 con 0x7f71f81f5f90 2023-11-28T17:41:58.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.138+0000 7f71feada640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f81f55c0 con 0x7f71e80039c0 2023-11-28T17:41:58.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.138+0000 7f71f7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71f80c90d0 0x7f71f81f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.138+0000 7f71f7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71f80c90d0 0x7f71f81f5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57072/0 (socket says 172.21.15.98:57072) 2023-11-28T17:41:58.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.138+0000 7f71f7fff640 1 -- 172.21.15.98:0/822755035 learned_addr learned my addr 172.21.15.98:0/822755035 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:58.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.139+0000 7f71fc876640 1 --2- 172.21.15.98:0/822755035 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71e80039c0 0x7f71f814cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.139+0000 7f71f7fff640 1 -- 172.21.15.98:0/822755035 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71e80039c0 msgr2=0x7f71f814cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.139+0000 7f71f7fff640 1 --2- 172.21.15.98:0/822755035 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71e80039c0 0x7f71f814cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.139+0000 7f71f7fff640 1 -- 172.21.15.98:0/822755035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71f81f5f90 msgr2=0x7f71f81f69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.139+0000 7f71f7fff640 1 --2- 172.21.15.98:0/822755035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71f81f5f90 0x7f71f81f69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.139+0000 7f71f7fff640 1 -- 172.21.15.98:0/822755035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f81f6410 con 0x7f71f80c90d0 2023-11-28T17:41:58.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.139+0000 7f71f7fff640 1 --2- 172.21.15.98:0/822755035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71f80c90d0 0x7f71f81f5a50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f71e400d7d0 tx=0x7f71e400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:58.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.139+0000 7f71f57fa640 1 -- 172.21.15.98:0/822755035 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e400b6c0 con 0x7f71f80c90d0 2023-11-28T17:41:58.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.139+0000 7f71f57fa640 1 -- 172.21.15.98:0/822755035 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e400b860 con 0x7f71f80c90d0 2023-11-28T17:41:58.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.139+0000 7f71f57fa640 1 -- 172.21.15.98:0/822755035 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e400bc10 con 0x7f71f80c90d0 2023-11-28T17:41:58.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.139+0000 7f71feada640 1 -- 172.21.15.98:0/822755035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f81f7720 con 0x7f71f80c90d0 2023-11-28T17:41:58.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.139+0000 7f71feada640 1 -- 172.21.15.98:0/822755035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f815b920 con 0x7f71f80c90d0 2023-11-28T17:41:58.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.141+0000 7f71feada640 1 -- 172.21.15.98:0/822755035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f8074f60 con 0x7f71f80c90d0 2023-11-28T17:41:58.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.141+0000 7f71f57fa640 1 -- 172.21.15.98:0/822755035 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f71e401e460 con 0x7f71f80c90d0 2023-11-28T17:41:58.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.142+0000 7f71f57fa640 1 --2- 172.21.15.98:0/822755035 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f71dc06b2a0 0x7f71dc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.142+0000 7f71f57fa640 1 -- 172.21.15.98:0/822755035 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71e4089fa0 con 0x7f71f80c90d0 2023-11-28T17:41:58.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.142+0000 7f71fc876640 1 --2- 172.21.15.98:0/822755035 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f71dc06b2a0 0x7f71dc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.143+0000 7f71fc876640 1 --2- 172.21.15.98:0/822755035 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f71dc06b2a0 0x7f71dc06d760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f71ec033000 tx=0x7f71ec009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:58.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.147+0000 7f71f57fa640 1 -- 172.21.15.98:0/822755035 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f71e40bce90 con 0x7f71f80c90d0 2023-11-28T17:41:58.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.331+0000 7f71feada640 1 -- 172.21.15.98:0/822755035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f71f806c600 con 0x7f71f80c90d0 2023-11-28T17:41:58.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.332+0000 7f71f57fa640 1 -- 172.21.15.98:0/822755035 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f71e4058930 con 0x7f71f80c90d0 2023-11-28T17:41:58.333 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-11-28T17:41:58.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.334+0000 7f71feada640 1 -- 172.21.15.98:0/822755035 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f71dc06b2a0 msgr2=0x7f71dc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.334+0000 7f71feada640 1 --2- 172.21.15.98:0/822755035 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f71dc06b2a0 0x7f71dc06d760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f71ec033000 tx=0x7f71ec009030 comp rx=0 tx=0).stop 2023-11-28T17:41:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.334+0000 7f71feada640 1 -- 172.21.15.98:0/822755035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71f80c90d0 msgr2=0x7f71f81f5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.334+0000 7f71feada640 1 --2- 172.21.15.98:0/822755035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71f80c90d0 0x7f71f81f5a50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f71e400d7d0 tx=0x7f71e400dc90 comp rx=0 tx=0).stop 2023-11-28T17:41:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.334+0000 7f71feada640 1 -- 172.21.15.98:0/822755035 shutdown_connections 2023-11-28T17:41:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.334+0000 7f71feada640 1 --2- 172.21.15.98:0/822755035 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f71dc06b2a0 0x7f71dc06d760 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.334+0000 7f71feada640 1 --2- 172.21.15.98:0/822755035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f71f81f5f90 0x7f71f81f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.334+0000 7f71feada640 1 --2- 172.21.15.98:0/822755035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71f80c90d0 0x7f71f81f5a50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.334+0000 7f71feada640 1 --2- 172.21.15.98:0/822755035 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f71e80039c0 0x7f71f814cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.334+0000 7f71feada640 1 -- 172.21.15.98:0/822755035 >> 172.21.15.98:0/822755035 conn(0x7f71f806c0f0 msgr2=0x7f71f815e400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:58.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.335+0000 7f71feada640 1 -- 172.21.15.98:0/822755035 shutdown_connections 2023-11-28T17:41:58.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.335+0000 7f71feada640 1 -- 172.21.15.98:0/822755035 wait complete. 2023-11-28T17:41:58.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:41:58.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.555+0000 7f4a2cd05640 1 Processor -- start 2023-11-28T17:41:58.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.555+0000 7f4a2cd05640 1 -- start start 2023-11-28T17:41:58.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.557+0000 7f4a2cd05640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a28154030 0x7f4a28070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.557+0000 7f4a2cd05640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a28154410 con 0x7f4a2806dcf0 2023-11-28T17:41:58.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.557+0000 7f4a2cd05640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a280715a0 con 0x7f4a280c8600 2023-11-28T17:41:58.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.557+0000 7f4a2cd05640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a280c5da0 con 0x7f4a28154030 2023-11-28T17:41:58.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.557+0000 7f4a26d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4a2806dcf0 0x7f4a28164660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46908/0 (socket says 172.21.15.98:46908) 2023-11-28T17:41:58.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.557+0000 7f4a26d9d640 1 -- 172.21.15.98:0/1405214148 learned_addr learned my addr 172.21.15.98:0/1405214148 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:58.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.557+0000 7f4a2659c640 1 --2- 172.21.15.98:0/1405214148 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a28154030 0x7f4a28070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.558+0000 7f4a2559a640 1 -- 172.21.15.98:0/1405214148 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87499877 0 0) 0x7f4a28154410 con 0x7f4a2806dcf0 2023-11-28T17:41:58.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.558+0000 7f4a2559a640 1 -- 172.21.15.98:0/1405214148 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a10003520 con 0x7f4a2806dcf0 2023-11-28T17:41:58.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.558+0000 7f4a2659c640 1 -- 172.21.15.98:0/1405214148 >> v1:172.21.15.113:6789/0 conn(0x7f4a280c8600 legacy=0x7f4a28070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.558+0000 7f4a2559a640 1 -- 172.21.15.98:0/1405214148 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2990478294 0 0) 0x7f4a10003520 con 0x7f4a2806dcf0 2023-11-28T17:41:58.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.558+0000 7f4a2659c640 1 -- 172.21.15.98:0/1405214148 >> v1:172.21.15.98:6789/0 conn(0x7f4a2806dcf0 legacy=0x7f4a28164660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.558+0000 7f4a2659c640 1 -- 172.21.15.98:0/1405214148 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2815d370 con 0x7f4a28154030 2023-11-28T17:41:58.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.559+0000 7f4a2659c640 1 --2- 172.21.15.98:0/1405214148 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a28154030 0x7f4a28070660 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4a180097d0 tx=0x7f4a1802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7be851a632f194f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:58.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.560+0000 7f4a2559a640 1 -- 172.21.15.98:0/1405214148 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a20001a10 con 0x7f4a28154030 2023-11-28T17:41:58.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.560+0000 7f4a2559a640 1 -- 172.21.15.98:0/1405214148 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a18031580 con 0x7f4a28154030 2023-11-28T17:41:58.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.560+0000 7f4a2559a640 1 -- 172.21.15.98:0/1405214148 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a180318f0 con 0x7f4a28154030 2023-11-28T17:41:58.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.560+0000 7f4a2cd05640 1 -- 172.21.15.98:0/1405214148 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a28154030 msgr2=0x7f4a28070660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.560+0000 7f4a2cd05640 1 --2- 172.21.15.98:0/1405214148 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a28154030 0x7f4a28070660 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4a180097d0 tx=0x7f4a1802b8a0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.561+0000 7f4a2cd05640 1 -- 172.21.15.98:0/1405214148 shutdown_connections 2023-11-28T17:41:58.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.561+0000 7f4a2cd05640 1 --2- 172.21.15.98:0/1405214148 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a28154030 0x7f4a28070660 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.561+0000 7f4a2cd05640 1 -- 172.21.15.98:0/1405214148 >> 172.21.15.98:0/1405214148 conn(0x7f4a2806c0f0 msgr2=0x7f4a2806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:58.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.561+0000 7f4a2cd05640 1 -- 172.21.15.98:0/1405214148 shutdown_connections 2023-11-28T17:41:58.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.561+0000 7f4a2cd05640 1 -- 172.21.15.98:0/1405214148 wait complete. 2023-11-28T17:41:58.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.563+0000 7f4a2cd05640 1 Processor -- start 2023-11-28T17:41:58.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.563+0000 7f4a2cd05640 1 -- start start 2023-11-28T17:41:58.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.564+0000 7f4a2cd05640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a2806dcf0 0x7f4a281565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.564+0000 7f4a2cd05640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a280c8600 0x7f4a2815c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.564+0000 7f4a2cd05640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a28154030 0x7f4a281ef8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.564+0000 7f4a2cd05640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a281f2ee0 con 0x7f4a28154030 2023-11-28T17:41:58.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.564+0000 7f4a2cd05640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a28156c00 con 0x7f4a2806dcf0 2023-11-28T17:41:58.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.564+0000 7f4a2cd05640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a281f3150 con 0x7f4a280c8600 2023-11-28T17:41:58.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.564+0000 7f4a26d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a28154030 0x7f4a281ef8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.564+0000 7f4a26d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a28154030 0x7f4a281ef8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57078/0 (socket says 172.21.15.98:57078) 2023-11-28T17:41:58.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.564+0000 7f4a26d9d640 1 -- 172.21.15.98:0/1874934552 learned_addr learned my addr 172.21.15.98:0/1874934552 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:58.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.564+0000 7f4a25d9b640 1 --2- 172.21.15.98:0/1874934552 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a280c8600 0x7f4a2815c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.565+0000 7f4a2659c640 1 --2- 172.21.15.98:0/1874934552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a2806dcf0 0x7f4a281565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.565+0000 7f4a26d9d640 1 -- 172.21.15.98:0/1874934552 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a280c8600 msgr2=0x7f4a2815c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.565+0000 7f4a26d9d640 1 --2- 172.21.15.98:0/1874934552 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a280c8600 0x7f4a2815c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.565+0000 7f4a26d9d640 1 -- 172.21.15.98:0/1874934552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a2806dcf0 msgr2=0x7f4a281565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.565+0000 7f4a26d9d640 1 --2- 172.21.15.98:0/1874934552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a2806dcf0 0x7f4a281565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.565+0000 7f4a26d9d640 1 -- 172.21.15.98:0/1874934552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a18009400 con 0x7f4a28154030 2023-11-28T17:41:58.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.565+0000 7f4a26d9d640 1 --2- 172.21.15.98:0/1874934552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a28154030 0x7f4a281ef8d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000caf0 tx=0x7f4a20002080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:58.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.565+0000 7f4a16ffd640 1 -- 172.21.15.98:0/1874934552 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2000a700 con 0x7f4a28154030 2023-11-28T17:41:58.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.565+0000 7f4a16ffd640 1 -- 172.21.15.98:0/1874934552 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2000a860 con 0x7f4a28154030 2023-11-28T17:41:58.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.565+0000 7f4a2cd05640 1 -- 172.21.15.98:0/1874934552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a281f3430 con 0x7f4a28154030 2023-11-28T17:41:58.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.565+0000 7f4a16ffd640 1 -- 172.21.15.98:0/1874934552 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2000ac10 con 0x7f4a28154030 2023-11-28T17:41:58.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.566+0000 7f4a2cd05640 1 -- 172.21.15.98:0/1874934552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a28155670 con 0x7f4a28154030 2023-11-28T17:41:58.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.567+0000 7f4a2cd05640 1 -- 172.21.15.98:0/1874934552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a281f3950 con 0x7f4a28154030 2023-11-28T17:41:58.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.567+0000 7f4a16ffd640 1 -- 172.21.15.98:0/1874934552 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4a200070d0 con 0x7f4a28154030 2023-11-28T17:41:58.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.568+0000 7f4a16ffd640 1 --2- 172.21.15.98:0/1874934552 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4a0806b2d0 0x7f4a0806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.568+0000 7f4a16ffd640 1 -- 172.21.15.98:0/1874934552 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a200892b0 con 0x7f4a28154030 2023-11-28T17:41:58.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.569+0000 7f4a2659c640 1 --2- 172.21.15.98:0/1874934552 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4a0806b2d0 0x7f4a0806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.569+0000 7f4a2659c640 1 --2- 172.21.15.98:0/1874934552 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4a0806b2d0 0x7f4a0806d790 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4a18036000 tx=0x7f4a1803d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:58.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.572+0000 7f4a16ffd640 1 -- 172.21.15.98:0/1874934552 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a281f3950 con 0x7f4a28154030 2023-11-28T17:41:58.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.745+0000 7f4a2cd05640 1 -- 172.21.15.98:0/1874934552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4a281f3950 con 0x7f4a28154030 2023-11-28T17:41:58.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.746+0000 7f4a16ffd640 1 -- 172.21.15.98:0/1874934552 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4a281f3950 con 0x7f4a28154030 2023-11-28T17:41:58.747 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-11-28T17:41:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.749+0000 7f4a2cd05640 1 -- 172.21.15.98:0/1874934552 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4a0806b2d0 msgr2=0x7f4a0806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.749+0000 7f4a2cd05640 1 --2- 172.21.15.98:0/1874934552 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4a0806b2d0 0x7f4a0806d790 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4a18036000 tx=0x7f4a1803d040 comp rx=0 tx=0).stop 2023-11-28T17:41:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.749+0000 7f4a2cd05640 1 -- 172.21.15.98:0/1874934552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a28154030 msgr2=0x7f4a281ef8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.749+0000 7f4a2cd05640 1 --2- 172.21.15.98:0/1874934552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a28154030 0x7f4a281ef8d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000caf0 tx=0x7f4a20002080 comp rx=0 tx=0).stop 2023-11-28T17:41:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.749+0000 7f4a2cd05640 1 -- 172.21.15.98:0/1874934552 shutdown_connections 2023-11-28T17:41:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.749+0000 7f4a2cd05640 1 --2- 172.21.15.98:0/1874934552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a28154030 0x7f4a281ef8d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.749+0000 7f4a2cd05640 1 --2- 172.21.15.98:0/1874934552 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4a280c8600 0x7f4a2815c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.749+0000 7f4a2cd05640 1 --2- 172.21.15.98:0/1874934552 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4a0806b2d0 0x7f4a0806d790 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.749+0000 7f4a2cd05640 1 --2- 172.21.15.98:0/1874934552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4a2806dcf0 0x7f4a281565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.749+0000 7f4a2cd05640 1 -- 172.21.15.98:0/1874934552 >> 172.21.15.98:0/1874934552 conn(0x7f4a2806c0f0 msgr2=0x7f4a280c6160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.749+0000 7f4a2cd05640 1 -- 172.21.15.98:0/1874934552 shutdown_connections 2023-11-28T17:41:58.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.749+0000 7f4a2cd05640 1 -- 172.21.15.98:0/1874934552 wait complete. 2023-11-28T17:41:58.766 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2023-11-28T17:41:58.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:41:58.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.971+0000 7fde436cf640 1 Processor -- start 2023-11-28T17:41:58.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.972+0000 7fde436cf640 1 -- start start 2023-11-28T17:41:58.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.973+0000 7fde436cf640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde3c151d80 0x7fde3c157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.973+0000 7fde436cf640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fde3c15bc10 0x7fde3c0c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.973+0000 7fde436cf640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde3c15b150 con 0x7fde3c0c6140 2023-11-28T17:41:58.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.973+0000 7fde436cf640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde3c06e1f0 con 0x7fde3c15bc10 2023-11-28T17:41:58.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.973+0000 7fde436cf640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde3c06e4d0 con 0x7fde3c151d80 2023-11-28T17:41:58.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.974+0000 7fde41c6c640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fde3c0c6140 0x7fde3c0c4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.98:39324/0 (socket says 172.21.15.98:39324) 2023-11-28T17:41:58.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.974+0000 7fde41c6c640 1 -- 172.21.15.98:0/2675283068 learned_addr learned my addr 172.21.15.98:0/2675283068 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:58.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.974+0000 7fde4146b640 1 --2- 172.21.15.98:0/2675283068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde3c151d80 0x7fde3c157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.974+0000 7fde40c6a640 1 --2- 172.21.15.98:0/2675283068 >> v2:172.21.15.113:3300/0 conn(0x7fde3c15bc10 0x7fde3c0c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.975+0000 7fde33fff640 1 -- 172.21.15.98:0/2675283068 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344445016 0 0) 0x7fde3c15b150 con 0x7fde3c0c6140 2023-11-28T17:41:58.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.975+0000 7fde4146b640 1 -- 172.21.15.98:0/2675283068 >> v2:172.21.15.113:3300/0 conn(0x7fde3c15bc10 msgr2=0x7fde3c0c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:58.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.975+0000 7fde4146b640 1 --2- 172.21.15.98:0/2675283068 >> v2:172.21.15.113:3300/0 conn(0x7fde3c15bc10 0x7fde3c0c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.975+0000 7fde4146b640 1 -- 172.21.15.98:0/2675283068 >> v1:172.21.15.113:6789/0 conn(0x7fde3c0c6140 legacy=0x7fde3c0c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.975+0000 7fde4146b640 1 -- 172.21.15.98:0/2675283068 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3c06e7b0 con 0x7fde3c151d80 2023-11-28T17:41:58.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.976+0000 7fde4146b640 1 --2- 172.21.15.98:0/2675283068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde3c151d80 0x7fde3c157b60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fde380097a0 tx=0x7fde3802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4060283ed064e632 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:58.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.976+0000 7fde33fff640 1 -- 172.21.15.98:0/2675283068 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde38030610 con 0x7fde3c151d80 2023-11-28T17:41:58.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.977+0000 7fde33fff640 1 -- 172.21.15.98:0/2675283068 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde38030770 con 0x7fde3c151d80 2023-11-28T17:41:58.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.977+0000 7fde33fff640 1 -- 172.21.15.98:0/2675283068 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde38030b00 con 0x7fde3c151d80 2023-11-28T17:41:58.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.977+0000 7fde436cf640 1 -- 172.21.15.98:0/2675283068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde3c151d80 msgr2=0x7fde3c157b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.977+0000 7fde436cf640 1 --2- 172.21.15.98:0/2675283068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde3c151d80 0x7fde3c157b60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fde380097a0 tx=0x7fde3802b960 comp rx=0 tx=0).stop 2023-11-28T17:41:58.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.977+0000 7fde436cf640 1 -- 172.21.15.98:0/2675283068 shutdown_connections 2023-11-28T17:41:58.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.977+0000 7fde436cf640 1 --2- 172.21.15.98:0/2675283068 >> v2:172.21.15.113:3300/0 conn(0x7fde3c15bc10 0x7fde3c0c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.977+0000 7fde436cf640 1 --2- 172.21.15.98:0/2675283068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde3c151d80 0x7fde3c157b60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.977+0000 7fde436cf640 1 -- 172.21.15.98:0/2675283068 >> 172.21.15.98:0/2675283068 conn(0x7fde3c06a4d0 msgr2=0x7fde3c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:58.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.978+0000 7fde436cf640 1 -- 172.21.15.98:0/2675283068 shutdown_connections 2023-11-28T17:41:58.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.978+0000 7fde436cf640 1 -- 172.21.15.98:0/2675283068 wait complete. 2023-11-28T17:41:58.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.979+0000 7fde436cf640 1 Processor -- start 2023-11-28T17:41:58.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.979+0000 7fde436cf640 1 -- start start 2023-11-28T17:41:58.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.980+0000 7fde436cf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c0c6140 0x7fde3c072500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.980+0000 7fde436cf640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde3c151d80 0x7fde3c152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.980+0000 7fde436cf640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde3c15bc10 0x7fde3c14caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.980+0000 7fde436cf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3c1570d0 con 0x7fde3c0c6140 2023-11-28T17:41:58.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.980+0000 7fde436cf640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3c1f3790 con 0x7fde3c151d80 2023-11-28T17:41:58.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.980+0000 7fde436cf640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde3c156df0 con 0x7fde3c15bc10 2023-11-28T17:41:58.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.980+0000 7fde4146b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c0c6140 0x7fde3c072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.981+0000 7fde4146b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c0c6140 0x7fde3c072500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57084/0 (socket says 172.21.15.98:57084) 2023-11-28T17:41:58.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.981+0000 7fde4146b640 1 -- 172.21.15.98:0/1564928950 learned_addr learned my addr 172.21.15.98:0/1564928950 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:58.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.981+0000 7fde41c6c640 1 --2- 172.21.15.98:0/1564928950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde3c15bc10 0x7fde3c14caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.981+0000 7fde40c6a640 1 --2- 172.21.15.98:0/1564928950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde3c151d80 0x7fde3c152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.981+0000 7fde4146b640 1 -- 172.21.15.98:0/1564928950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde3c15bc10 msgr2=0x7fde3c14caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.981+0000 7fde4146b640 1 --2- 172.21.15.98:0/1564928950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde3c15bc10 0x7fde3c14caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.981+0000 7fde4146b640 1 -- 172.21.15.98:0/1564928950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde3c151d80 msgr2=0x7fde3c152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:58.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.981+0000 7fde4146b640 1 --2- 172.21.15.98:0/1564928950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde3c151d80 0x7fde3c152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:58.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.981+0000 7fde4146b640 1 -- 172.21.15.98:0/1564928950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde38009400 con 0x7fde3c0c6140 2023-11-28T17:41:58.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.981+0000 7fde4146b640 1 --2- 172.21.15.98:0/1564928950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c0c6140 0x7fde3c072500 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fde3802bf70 tx=0x7fde38004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:58.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.982+0000 7fde31ffb640 1 -- 172.21.15.98:0/1564928950 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3803e040 con 0x7fde3c0c6140 2023-11-28T17:41:58.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.982+0000 7fde31ffb640 1 -- 172.21.15.98:0/1564928950 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde38039cb0 con 0x7fde3c0c6140 2023-11-28T17:41:58.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.982+0000 7fde31ffb640 1 -- 172.21.15.98:0/1564928950 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde38043680 con 0x7fde3c0c6140 2023-11-28T17:41:58.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.982+0000 7fde436cf640 1 -- 172.21.15.98:0/1564928950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde3c1f3140 con 0x7fde3c0c6140 2023-11-28T17:41:58.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.982+0000 7fde436cf640 1 -- 172.21.15.98:0/1564928950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde3c14cfe0 con 0x7fde3c0c6140 2023-11-28T17:41:58.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.983+0000 7fde436cf640 1 -- 172.21.15.98:0/1564928950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde3c067bf0 con 0x7fde3c0c6140 2023-11-28T17:41:58.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.984+0000 7fde31ffb640 1 -- 172.21.15.98:0/1564928950 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fde380423e0 con 0x7fde3c0c6140 2023-11-28T17:41:58.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.985+0000 7fde31ffb640 1 --2- 172.21.15.98:0/1564928950 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fde2006b2a0 0x7fde2006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:58.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.985+0000 7fde31ffb640 1 -- 172.21.15.98:0/1564928950 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde380ac090 con 0x7fde3c0c6140 2023-11-28T17:41:58.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.986+0000 7fde40c6a640 1 --2- 172.21.15.98:0/1564928950 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fde2006b2a0 0x7fde2006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:58.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.986+0000 7fde40c6a640 1 --2- 172.21.15.98:0/1564928950 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fde2006b2a0 0x7fde2006d760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0c6bb0 tx=0x7fde2c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:58.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:58.990+0000 7fde31ffb640 1 -- 172.21.15.98:0/1564928950 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde38076dd0 con 0x7fde3c0c6140 2023-11-28T17:41:59.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.177+0000 7fde436cf640 1 -- 172.21.15.98:0/1564928950 --> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fde3c15c010 con 0x7fde2006b2a0 2023-11-28T17:41:59.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.186+0000 7fde31ffb640 1 -- 172.21.15.98:0/1564928950 <== mgr.4098 v2:172.21.15.98:6832/2274392793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264615 (secure 0 0 0) 0x7fde2c011970 con 0x7fde2006b2a0 2023-11-28T17:41:59.188 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:59.192 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-11-28T17:41:59.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.193+0000 7fde436cf640 1 -- 172.21.15.98:0/1564928950 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fde2006b2a0 msgr2=0x7fde2006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:59.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.193+0000 7fde436cf640 1 --2- 172.21.15.98:0/1564928950 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fde2006b2a0 0x7fde2006d760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0c6bb0 tx=0x7fde2c008040 comp rx=0 tx=0).stop 2023-11-28T17:41:59.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.193+0000 7fde436cf640 1 -- 172.21.15.98:0/1564928950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c0c6140 msgr2=0x7fde3c072500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:59.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.193+0000 7fde436cf640 1 --2- 172.21.15.98:0/1564928950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c0c6140 0x7fde3c072500 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fde3802bf70 tx=0x7fde38004730 comp rx=0 tx=0).stop 2023-11-28T17:41:59.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.193+0000 7fde436cf640 1 -- 172.21.15.98:0/1564928950 shutdown_connections 2023-11-28T17:41:59.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.193+0000 7fde436cf640 1 --2- 172.21.15.98:0/1564928950 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde3c15bc10 0x7fde3c14caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.193+0000 7fde436cf640 1 --2- 172.21.15.98:0/1564928950 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fde2006b2a0 0x7fde2006d760 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.193+0000 7fde436cf640 1 --2- 172.21.15.98:0/1564928950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde3c151d80 0x7fde3c152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.193+0000 7fde436cf640 1 --2- 172.21.15.98:0/1564928950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c0c6140 0x7fde3c072500 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.193+0000 7fde436cf640 1 -- 172.21.15.98:0/1564928950 >> 172.21.15.98:0/1564928950 conn(0x7fde3c06a4d0 msgr2=0x7fde3c0c7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:59.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.193+0000 7fde436cf640 1 -- 172.21.15.98:0/1564928950 shutdown_connections 2023-11-28T17:41:59.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.193+0000 7fde436cf640 1 -- 172.21.15.98:0/1564928950 wait complete. 2023-11-28T17:41:59.212 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2023-11-28T17:41:58.250229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600162,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":372,"num_write_kb":2486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":304,"ondisk_log_size":304,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218240,"kb_used_data":2112,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756480,"statfs":{"total":773094113280,"available":772870635520,"internally_reserved":0,"allocated":2162688,"data_stored":1811260,"data_compressed":65062,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1782,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004084"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339217+0000","last_change":"2023-11-28T17:41:17.238145+0000","last_active":"2023-11-28T17:41:18.339217+0000","last_peered":"2023-11-28T17:41:18.339217+0000","last_clean":"2023-11-28T17:41:18.339217+0000","last_became_active":"2023-11-28T17:41:17.237776+0000","last_became_peered":"2023-11-28T17:41:17.237776+0000","last_unstale":"2023-11-28T17:41:18.339217+0000","last_undegraded":"2023-11-28T17:41:18.339217+0000","last_fullsized":"2023-11-28T17:41:18.339217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:41:06.902665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632799+0000","last_change":"2023-11-28T17:41:17.231913+0000","last_active":"2023-11-28T17:41:19.632799+0000","last_peered":"2023-11-28T17:41:19.632799+0000","last_clean":"2023-11-28T17:41:19.632799+0000","last_became_active":"2023-11-28T17:41:17.231622+0000","last_became_peered":"2023-11-28T17:41:17.231622+0000","last_unstale":"2023-11-28T17:41:19.632799+0000","last_undegraded":"2023-11-28T17:41:19.632799+0000","last_fullsized":"2023-11-28T17:41:19.632799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:06.911223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649842+0000","last_change":"2023-11-28T17:41:17.230232+0000","last_active":"2023-11-28T17:41:18.649842+0000","last_peered":"2023-11-28T17:41:18.649842+0000","last_clean":"2023-11-28T17:41:18.649842+0000","last_became_active":"2023-11-28T17:41:17.229795+0000","last_became_peered":"2023-11-28T17:41:17.229795+0000","last_unstale":"2023-11-28T17:41:18.649842+0000","last_undegraded":"2023-11-28T17:41:18.649842+0000","last_fullsized":"2023-11-28T17:41:18.649842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:09:09.507547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207139+0000","last_change":"2023-11-28T17:41:17.234948+0000","last_active":"2023-11-28T17:41:18.207139+0000","last_peered":"2023-11-28T17:41:18.207139+0000","last_clean":"2023-11-28T17:41:18.207139+0000","last_became_active":"2023-11-28T17:41:17.233674+0000","last_became_peered":"2023-11-28T17:41:17.233674+0000","last_unstale":"2023-11-28T17:41:18.207139+0000","last_undegraded":"2023-11-28T17:41:18.207139+0000","last_fullsized":"2023-11-28T17:41:18.207139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:45:10.831719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338225+0000","last_change":"2023-11-28T17:41:17.233908+0000","last_active":"2023-11-28T17:41:18.338225+0000","last_peered":"2023-11-28T17:41:18.338225+0000","last_clean":"2023-11-28T17:41:18.338225+0000","last_became_active":"2023-11-28T17:41:17.233589+0000","last_became_peered":"2023-11-28T17:41:17.233589+0000","last_unstale":"2023-11-28T17:41:18.338225+0000","last_undegraded":"2023-11-28T17:41:18.338225+0000","last_fullsized":"2023-11-28T17:41:18.338225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:21:37.379103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206516+0000","last_change":"2023-11-28T17:41:17.239186+0000","last_active":"2023-11-28T17:41:18.206516+0000","last_peered":"2023-11-28T17:41:18.206516+0000","last_clean":"2023-11-28T17:41:18.206516+0000","last_became_active":"2023-11-28T17:41:17.233898+0000","last_became_peered":"2023-11-28T17:41:17.233898+0000","last_unstale":"2023-11-28T17:41:18.206516+0000","last_undegraded":"2023-11-28T17:41:18.206516+0000","last_fullsized":"2023-11-28T17:41:18.206516+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:54.918589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338198+0000","last_change":"2023-11-28T17:41:17.227923+0000","last_active":"2023-11-28T17:41:18.338198+0000","last_peered":"2023-11-28T17:41:18.338198+0000","last_clean":"2023-11-28T17:41:18.338198+0000","last_became_active":"2023-11-28T17:41:17.227642+0000","last_became_peered":"2023-11-28T17:41:17.227642+0000","last_unstale":"2023-11-28T17:41:18.338198+0000","last_undegraded":"2023-11-28T17:41:18.338198+0000","last_fullsized":"2023-11-28T17:41:18.338198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:33:29.400533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567710+0000","last_change":"2023-11-28T17:41:17.239818+0000","last_active":"2023-11-28T17:41:18.567710+0000","last_peered":"2023-11-28T17:41:18.567710+0000","last_clean":"2023-11-28T17:41:18.567710+0000","last_became_active":"2023-11-28T17:41:17.239594+0000","last_became_peered":"2023-11-28T17:41:17.239594+0000","last_unstale":"2023-11-28T17:41:18.567710+0000","last_undegraded":"2023-11-28T17:41:18.567710+0000","last_fullsized":"2023-11-28T17:41:18.567710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:23.969471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339318+0000","last_change":"2023-11-28T17:41:17.243214+0000","last_active":"2023-11-28T17:41:18.339318+0000","last_peered":"2023-11-28T17:41:18.339318+0000","last_clean":"2023-11-28T17:41:18.339318+0000","last_became_active":"2023-11-28T17:41:17.242933+0000","last_became_peered":"2023-11-28T17:41:17.242933+0000","last_unstale":"2023-11-28T17:41:18.339318+0000","last_undegraded":"2023-11-28T17:41:18.339318+0000","last_fullsized":"2023-11-28T17:41:18.339318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:47:47.697137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633703+0000","last_change":"2023-11-28T17:41:17.242507+0000","last_active":"2023-11-28T17:41:19.633703+0000","last_peered":"2023-11-28T17:41:19.633703+0000","last_clean":"2023-11-28T17:41:19.633703+0000","last_became_active":"2023-11-28T17:41:17.242196+0000","last_became_peered":"2023-11-28T17:41:17.242196+0000","last_unstale":"2023-11-28T17:41:19.633703+0000","last_undegraded":"2023-11-28T17:41:19.633703+0000","last_fullsized":"2023-11-28T17:41:19.633703+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:06.103318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633386+0000","last_change":"2023-11-28T17:41:17.244694+0000","last_active":"2023-11-28T17:41:19.633386+0000","last_peered":"2023-11-28T17:41:19.633386+0000","last_clean":"2023-11-28T17:41:19.633386+0000","last_became_active":"2023-11-28T17:41:17.244301+0000","last_became_peered":"2023-11-28T17:41:17.244301+0000","last_unstale":"2023-11-28T17:41:19.633386+0000","last_undegraded":"2023-11-28T17:41:19.633386+0000","last_fullsized":"2023-11-28T17:41:19.633386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:40.030685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208504+0000","last_change":"2023-11-28T17:41:17.242800+0000","last_active":"2023-11-28T17:41:18.208504+0000","last_peered":"2023-11-28T17:41:18.208504+0000","last_clean":"2023-11-28T17:41:18.208504+0000","last_became_active":"2023-11-28T17:41:17.242463+0000","last_became_peered":"2023-11-28T17:41:17.242463+0000","last_unstale":"2023-11-28T17:41:18.208504+0000","last_undegraded":"2023-11-28T17:41:18.208504+0000","last_fullsized":"2023-11-28T17:41:18.208504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:57:34.802348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568335+0000","last_change":"2023-11-28T17:41:17.232485+0000","last_active":"2023-11-28T17:41:18.568335+0000","last_peered":"2023-11-28T17:41:18.568335+0000","last_clean":"2023-11-28T17:41:18.568335+0000","last_became_active":"2023-11-28T17:41:17.232132+0000","last_became_peered":"2023-11-28T17:41:17.232132+0000","last_unstale":"2023-11-28T17:41:18.568335+0000","last_undegraded":"2023-11-28T17:41:18.568335+0000","last_fullsized":"2023-11-28T17:41:18.568335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:49.292565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649479+0000","last_change":"2023-11-28T17:41:17.234514+0000","last_active":"2023-11-28T17:41:18.649479+0000","last_peered":"2023-11-28T17:41:18.649479+0000","last_clean":"2023-11-28T17:41:18.649479+0000","last_became_active":"2023-11-28T17:41:17.234154+0000","last_became_peered":"2023-11-28T17:41:17.234154+0000","last_unstale":"2023-11-28T17:41:18.649479+0000","last_undegraded":"2023-11-28T17:41:18.649479+0000","last_fullsized":"2023-11-28T17:41:18.649479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:22.430159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632129+0000","last_change":"2023-11-28T17:41:17.225905+0000","last_active":"2023-11-28T17:41:19.632129+0000","last_peered":"2023-11-28T17:41:19.632129+0000","last_clean":"2023-11-28T17:41:19.632129+0000","last_became_active":"2023-11-28T17:41:17.223533+0000","last_became_peered":"2023-11-28T17:41:17.223533+0000","last_unstale":"2023-11-28T17:41:19.632129+0000","last_undegraded":"2023-11-28T17:41:19.632129+0000","last_fullsized":"2023-11-28T17:41:19.632129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:32.314586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.205612+0000","last_change":"2023-11-28T17:41:17.236242+0000","last_active":"2023-11-28T17:41:18.205612+0000","last_peered":"2023-11-28T17:41:18.205612+0000","last_clean":"2023-11-28T17:41:18.205612+0000","last_became_active":"2023-11-28T17:41:17.235890+0000","last_became_peered":"2023-11-28T17:41:17.235890+0000","last_unstale":"2023-11-28T17:41:18.205612+0000","last_undegraded":"2023-11-28T17:41:18.205612+0000","last_fullsized":"2023-11-28T17:41:18.205612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:20:09.004702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208885+0000","last_change":"2023-11-28T17:41:17.234610+0000","last_active":"2023-11-28T17:41:18.208885+0000","last_peered":"2023-11-28T17:41:18.208885+0000","last_clean":"2023-11-28T17:41:18.208885+0000","last_became_active":"2023-11-28T17:41:17.234255+0000","last_became_peered":"2023-11-28T17:41:17.234255+0000","last_unstale":"2023-11-28T17:41:18.208885+0000","last_undegraded":"2023-11-28T17:41:18.208885+0000","last_fullsized":"2023-11-28T17:41:18.208885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:25:51.137148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.202895+0000","last_change":"2023-11-28T17:41:17.231838+0000","last_active":"2023-11-28T17:41:18.202895+0000","last_peered":"2023-11-28T17:41:18.202895+0000","last_clean":"2023-11-28T17:41:18.202895+0000","last_became_active":"2023-11-28T17:41:17.231533+0000","last_became_peered":"2023-11-28T17:41:17.231533+0000","last_unstale":"2023-11-28T17:41:18.202895+0000","last_undegraded":"2023-11-28T17:41:18.202895+0000","last_fullsized":"2023-11-28T17:41:18.202895+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:00:56.554112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339038+0000","last_change":"2023-11-28T17:41:17.238888+0000","last_active":"2023-11-28T17:41:18.339038+0000","last_peered":"2023-11-28T17:41:18.339038+0000","last_clean":"2023-11-28T17:41:18.339038+0000","last_became_active":"2023-11-28T17:41:17.236140+0000","last_became_peered":"2023-11-28T17:41:17.236140+0000","last_unstale":"2023-11-28T17:41:18.339038+0000","last_undegraded":"2023-11-28T17:41:18.339038+0000","last_fullsized":"2023-11-28T17:41:18.339038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:20:57.936710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633201+0000","last_change":"2023-11-28T17:41:17.242150+0000","last_active":"2023-11-28T17:41:19.633201+0000","last_peered":"2023-11-28T17:41:19.633201+0000","last_clean":"2023-11-28T17:41:19.633201+0000","last_became_active":"2023-11-28T17:41:17.241832+0000","last_became_peered":"2023-11-28T17:41:17.241832+0000","last_unstale":"2023-11-28T17:41:19.633201+0000","last_undegraded":"2023-11-28T17:41:19.633201+0000","last_fullsized":"2023-11-28T17:41:19.633201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:30:37.837711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649928+0000","last_change":"2023-11-28T17:41:17.233651+0000","last_active":"2023-11-28T17:41:18.649928+0000","last_peered":"2023-11-28T17:41:18.649928+0000","last_clean":"2023-11-28T17:41:18.649928+0000","last_became_active":"2023-11-28T17:41:17.233393+0000","last_became_peered":"2023-11-28T17:41:17.233393+0000","last_unstale":"2023-11-28T17:41:18.649928+0000","last_undegraded":"2023-11-28T17:41:18.649928+0000","last_fullsized":"2023-11-28T17:41:18.649928+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:52:08.152046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206688+0000","last_change":"2023-11-28T17:41:17.239696+0000","last_active":"2023-11-28T17:41:18.206688+0000","last_peered":"2023-11-28T17:41:18.206688+0000","last_clean":"2023-11-28T17:41:18.206688+0000","last_became_active":"2023-11-28T17:41:17.239449+0000","last_became_peered":"2023-11-28T17:41:17.239449+0000","last_unstale":"2023-11-28T17:41:18.206688+0000","last_undegraded":"2023-11-28T17:41:18.206688+0000","last_fullsized":"2023-11-28T17:41:18.206688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:08:43.019689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567770+0000","last_change":"2023-11-28T17:41:17.244917+0000","last_active":"2023-11-28T17:41:18.567770+0000","last_peered":"2023-11-28T17:41:18.567770+0000","last_clean":"2023-11-28T17:41:18.567770+0000","last_became_active":"2023-11-28T17:41:17.244676+0000","last_became_peered":"2023-11-28T17:41:17.244676+0000","last_unstale":"2023-11-28T17:41:18.567770+0000","last_undegraded":"2023-11-28T17:41:18.567770+0000","last_fullsized":"2023-11-28T17:41:18.567770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:08:10.818556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206063+0000","last_change":"2023-11-28T17:41:17.236488+0000","last_active":"2023-11-28T17:41:18.206063+0000","last_peered":"2023-11-28T17:41:18.206063+0000","last_clean":"2023-11-28T17:41:18.206063+0000","last_became_active":"2023-11-28T17:41:17.235554+0000","last_became_peered":"2023-11-28T17:41:17.235554+0000","last_unstale":"2023-11-28T17:41:18.206063+0000","last_undegraded":"2023-11-28T17:41:18.206063+0000","last_fullsized":"2023-11-28T17:41:18.206063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:29:35.500772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568927+0000","last_change":"2023-11-28T17:41:17.245042+0000","last_active":"2023-11-28T17:41:18.568927+0000","last_peered":"2023-11-28T17:41:18.568927+0000","last_clean":"2023-11-28T17:41:18.568927+0000","last_became_active":"2023-11-28T17:41:17.244729+0000","last_became_peered":"2023-11-28T17:41:17.244729+0000","last_unstale":"2023-11-28T17:41:18.568927+0000","last_undegraded":"2023-11-28T17:41:18.568927+0000","last_fullsized":"2023-11-28T17:41:18.568927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:57:00.728653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633914+0000","last_change":"2023-11-28T17:41:17.249367+0000","last_active":"2023-11-28T17:41:19.633914+0000","last_peered":"2023-11-28T17:41:19.633914+0000","last_clean":"2023-11-28T17:41:19.633914+0000","last_became_active":"2023-11-28T17:41:17.249093+0000","last_became_peered":"2023-11-28T17:41:17.249093+0000","last_unstale":"2023-11-28T17:41:19.633914+0000","last_undegraded":"2023-11-28T17:41:19.633914+0000","last_fullsized":"2023-11-28T17:41:19.633914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:12:41.994354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632733+0000","last_change":"2023-11-28T17:41:17.230256+0000","last_active":"2023-11-28T17:41:19.632733+0000","last_peered":"2023-11-28T17:41:19.632733+0000","last_clean":"2023-11-28T17:41:19.632733+0000","last_became_active":"2023-11-28T17:41:17.229551+0000","last_became_peered":"2023-11-28T17:41:17.229551+0000","last_unstale":"2023-11-28T17:41:19.632733+0000","last_undegraded":"2023-11-28T17:41:19.632733+0000","last_fullsized":"2023-11-28T17:41:19.632733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:54:09.421121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568751+0000","last_change":"2023-11-28T17:41:17.241672+0000","last_active":"2023-11-28T17:41:18.568751+0000","last_peered":"2023-11-28T17:41:18.568751+0000","last_clean":"2023-11-28T17:41:18.568751+0000","last_became_active":"2023-11-28T17:41:17.241366+0000","last_became_peered":"2023-11-28T17:41:17.241366+0000","last_unstale":"2023-11-28T17:41:18.568751+0000","last_undegraded":"2023-11-28T17:41:18.568751+0000","last_fullsized":"2023-11-28T17:41:18.568751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:29:01.718362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650047+0000","last_change":"2023-11-28T17:41:17.229925+0000","last_active":"2023-11-28T17:41:18.650047+0000","last_peered":"2023-11-28T17:41:18.650047+0000","last_clean":"2023-11-28T17:41:18.650047+0000","last_became_active":"2023-11-28T17:41:17.229617+0000","last_became_peered":"2023-11-28T17:41:17.229617+0000","last_unstale":"2023-11-28T17:41:18.650047+0000","last_undegraded":"2023-11-28T17:41:18.650047+0000","last_fullsized":"2023-11-28T17:41:18.650047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:40:14.646558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206950+0000","last_change":"2023-11-28T17:41:17.241933+0000","last_active":"2023-11-28T17:41:18.206950+0000","last_peered":"2023-11-28T17:41:18.206950+0000","last_clean":"2023-11-28T17:41:18.206950+0000","last_became_active":"2023-11-28T17:41:17.241634+0000","last_became_peered":"2023-11-28T17:41:17.241634+0000","last_unstale":"2023-11-28T17:41:18.206950+0000","last_undegraded":"2023-11-28T17:41:18.206950+0000","last_fullsized":"2023-11-28T17:41:18.206950+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:44:01.525920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.202444+0000","last_change":"2023-11-28T17:41:17.233872+0000","last_active":"2023-11-28T17:41:18.202444+0000","last_peered":"2023-11-28T17:41:18.202444+0000","last_clean":"2023-11-28T17:41:18.202444+0000","last_became_active":"2023-11-28T17:41:17.233586+0000","last_became_peered":"2023-11-28T17:41:17.233586+0000","last_unstale":"2023-11-28T17:41:18.202444+0000","last_undegraded":"2023-11-28T17:41:18.202444+0000","last_fullsized":"2023-11-28T17:41:18.202444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:56.041470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633270+0000","last_change":"2023-11-28T17:41:17.236164+0000","last_active":"2023-11-28T17:41:19.633270+0000","last_peered":"2023-11-28T17:41:19.633270+0000","last_clean":"2023-11-28T17:41:19.633270+0000","last_became_active":"2023-11-28T17:41:17.235821+0000","last_became_peered":"2023-11-28T17:41:17.235821+0000","last_unstale":"2023-11-28T17:41:19.633270+0000","last_undegraded":"2023-11-28T17:41:19.633270+0000","last_fullsized":"2023-11-28T17:41:19.633270+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:54:16.568394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206920+0000","last_change":"2023-11-28T17:41:17.235816+0000","last_active":"2023-11-28T17:41:18.206920+0000","last_peered":"2023-11-28T17:41:18.206920+0000","last_clean":"2023-11-28T17:41:18.206920+0000","last_became_active":"2023-11-28T17:41:17.235506+0000","last_became_peered":"2023-11-28T17:41:17.235506+0000","last_unstale":"2023-11-28T17:41:18.206920+0000","last_undegraded":"2023-11-28T17:41:18.206920+0000","last_fullsized":"2023-11-28T17:41:18.206920+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:18:00.597727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:27.335928+0000","last_change":"2023-11-28T17:41:16.266076+0000","last_active":"2023-11-28T17:41:27.335928+0000","last_peered":"2023-11-28T17:41:27.335928+0000","last_clean":"2023-11-28T17:41:27.335928+0000","last_became_active":"2023-11-28T17:41:16.265837+0000","last_became_peered":"2023-11-28T17:41:16.265837+0000","last_unstale":"2023-11-28T17:41:27.335928+0000","last_undegraded":"2023-11-28T17:41:27.335928+0000","last_fullsized":"2023-11-28T17:41:27.335928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:01:51.477975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1062,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650250+0000","last_change":"2023-11-28T17:41:17.238915+0000","last_active":"2023-11-28T17:41:18.650250+0000","last_peered":"2023-11-28T17:41:18.650250+0000","last_clean":"2023-11-28T17:41:18.650250+0000","last_became_active":"2023-11-28T17:41:17.238581+0000","last_became_peered":"2023-11-28T17:41:17.238581+0000","last_unstale":"2023-11-28T17:41:18.650250+0000","last_undegraded":"2023-11-28T17:41:18.650250+0000","last_fullsized":"2023-11-28T17:41:18.650250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:52:31.640239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.242663+0000","last_change":"2023-11-28T17:41:16.261044+0000","last_active":"2023-11-28T17:41:39.242663+0000","last_peered":"2023-11-28T17:41:39.242663+0000","last_clean":"2023-11-28T17:41:39.242663+0000","last_became_active":"2023-11-28T17:41:16.260752+0000","last_became_peered":"2023-11-28T17:41:16.260752+0000","last_unstale":"2023-11-28T17:41:39.242663+0000","last_undegraded":"2023-11-28T17:41:39.242663+0000","last_fullsized":"2023-11-28T17:41:39.242663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:07:01.443140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:33.236512+0000","last_change":"2023-11-28T17:41:16.261095+0000","last_active":"2023-11-28T17:41:33.236512+0000","last_peered":"2023-11-28T17:41:33.236512+0000","last_clean":"2023-11-28T17:41:33.236512+0000","last_became_active":"2023-11-28T17:41:16.260793+0000","last_became_peered":"2023-11-28T17:41:16.260793+0000","last_unstale":"2023-11-28T17:41:33.236512+0000","last_undegraded":"2023-11-28T17:41:33.236512+0000","last_fullsized":"2023-11-28T17:41:33.236512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:54:16.821314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567897+0000","last_change":"2023-11-28T17:41:17.236517+0000","last_active":"2023-11-28T17:41:18.567897+0000","last_peered":"2023-11-28T17:41:18.567897+0000","last_clean":"2023-11-28T17:41:18.567897+0000","last_became_active":"2023-11-28T17:41:17.235925+0000","last_became_peered":"2023-11-28T17:41:17.235925+0000","last_unstale":"2023-11-28T17:41:18.567897+0000","last_undegraded":"2023-11-28T17:41:18.567897+0000","last_fullsized":"2023-11-28T17:41:18.567897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:45:09.817844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.242825+0000","last_change":"2023-11-28T17:41:16.258176+0000","last_active":"2023-11-28T17:41:39.242825+0000","last_peered":"2023-11-28T17:41:39.242825+0000","last_clean":"2023-11-28T17:41:39.242825+0000","last_became_active":"2023-11-28T17:41:16.257838+0000","last_became_peered":"2023-11-28T17:41:16.257838+0000","last_unstale":"2023-11-28T17:41:39.242825+0000","last_undegraded":"2023-11-28T17:41:39.242825+0000","last_fullsized":"2023-11-28T17:41:39.242825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:31:01.443661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.631980+0000","last_change":"2023-11-28T17:41:17.227823+0000","last_active":"2023-11-28T17:41:19.631980+0000","last_peered":"2023-11-28T17:41:19.631980+0000","last_clean":"2023-11-28T17:41:19.631980+0000","last_became_active":"2023-11-28T17:41:17.227545+0000","last_became_peered":"2023-11-28T17:41:17.227545+0000","last_unstale":"2023-11-28T17:41:19.631980+0000","last_undegraded":"2023-11-28T17:41:19.631980+0000","last_fullsized":"2023-11-28T17:41:19.631980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:20:33.253155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:55.076692+0000","last_change":"2023-11-28T17:41:16.262372+0000","last_active":"2023-11-28T17:41:55.076692+0000","last_peered":"2023-11-28T17:41:55.076692+0000","last_clean":"2023-11-28T17:41:55.076692+0000","last_became_active":"2023-11-28T17:41:16.262029+0000","last_became_peered":"2023-11-28T17:41:16.262029+0000","last_unstale":"2023-11-28T17:41:55.076692+0000","last_undegraded":"2023-11-28T17:41:55.076692+0000","last_fullsized":"2023-11-28T17:41:55.076692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:46:08.491380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207822+0000","last_change":"2023-11-28T17:41:17.235646+0000","last_active":"2023-11-28T17:41:18.207822+0000","last_peered":"2023-11-28T17:41:18.207822+0000","last_clean":"2023-11-28T17:41:18.207822+0000","last_became_active":"2023-11-28T17:41:17.235330+0000","last_became_peered":"2023-11-28T17:41:17.235330+0000","last_unstale":"2023-11-28T17:41:18.207822+0000","last_undegraded":"2023-11-28T17:41:18.207822+0000","last_fullsized":"2023-11-28T17:41:18.207822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:48:12.717031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.334160+0000","last_change":"2023-11-28T17:41:16.262716+0000","last_active":"2023-11-28T17:41:44.334160+0000","last_peered":"2023-11-28T17:41:44.334160+0000","last_clean":"2023-11-28T17:41:44.334160+0000","last_became_active":"2023-11-28T17:41:16.262347+0000","last_became_peered":"2023-11-28T17:41:16.262347+0000","last_unstale":"2023-11-28T17:41:44.334160+0000","last_undegraded":"2023-11-28T17:41:44.334160+0000","last_fullsized":"2023-11-28T17:41:44.334160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:50.685367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650261+0000","last_change":"2023-11-28T17:41:17.245852+0000","last_active":"2023-11-28T17:41:18.650261+0000","last_peered":"2023-11-28T17:41:18.650261+0000","last_clean":"2023-11-28T17:41:18.650261+0000","last_became_active":"2023-11-28T17:41:17.245596+0000","last_became_peered":"2023-11-28T17:41:17.245596+0000","last_unstale":"2023-11-28T17:41:18.650261+0000","last_undegraded":"2023-11-28T17:41:18.650261+0000","last_fullsized":"2023-11-28T17:41:18.650261+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:19:03.484052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:27.332030+0000","last_change":"2023-11-28T17:41:16.254855+0000","last_active":"2023-11-28T17:41:27.332030+0000","last_peered":"2023-11-28T17:41:27.332030+0000","last_clean":"2023-11-28T17:41:27.332030+0000","last_became_active":"2023-11-28T17:41:16.254557+0000","last_became_peered":"2023-11-28T17:41:16.254557+0000","last_unstale":"2023-11-28T17:41:27.332030+0000","last_undegraded":"2023-11-28T17:41:27.332030+0000","last_fullsized":"2023-11-28T17:41:27.332030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:55:38.565240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208613+0000","last_change":"2023-11-28T17:41:17.249020+0000","last_active":"2023-11-28T17:41:18.208613+0000","last_peered":"2023-11-28T17:41:18.208613+0000","last_clean":"2023-11-28T17:41:18.208613+0000","last_became_active":"2023-11-28T17:41:17.248345+0000","last_became_peered":"2023-11-28T17:41:17.248345+0000","last_unstale":"2023-11-28T17:41:18.208613+0000","last_undegraded":"2023-11-28T17:41:18.208613+0000","last_fullsized":"2023-11-28T17:41:18.208613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:05:09.688116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'12","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:55.092716+0000","last_change":"2023-11-28T17:41:16.256485+0000","last_active":"2023-11-28T17:41:55.092716+0000","last_peered":"2023-11-28T17:41:55.092716+0000","last_clean":"2023-11-28T17:41:55.092716+0000","last_became_active":"2023-11-28T17:41:16.256165+0000","last_became_peered":"2023-11-28T17:41:16.256165+0000","last_unstale":"2023-11-28T17:41:55.092716+0000","last_undegraded":"2023-11-28T17:41:55.092716+0000","last_fullsized":"2023-11-28T17:41:55.092716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:35:26.805435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2052,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338198+0000","last_change":"2023-11-28T17:41:17.230246+0000","last_active":"2023-11-28T17:41:18.338198+0000","last_peered":"2023-11-28T17:41:18.338198+0000","last_clean":"2023-11-28T17:41:18.338198+0000","last_became_active":"2023-11-28T17:41:17.229981+0000","last_became_peered":"2023-11-28T17:41:17.229981+0000","last_unstale":"2023-11-28T17:41:18.338198+0000","last_undegraded":"2023-11-28T17:41:18.338198+0000","last_fullsized":"2023-11-28T17:41:18.338198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:15:48.047641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.331053+0000","last_change":"2023-11-28T17:41:16.250894+0000","last_active":"2023-11-28T17:41:44.331053+0000","last_peered":"2023-11-28T17:41:44.331053+0000","last_clean":"2023-11-28T17:41:44.331053+0000","last_became_active":"2023-11-28T17:41:16.250547+0000","last_became_peered":"2023-11-28T17:41:16.250547+0000","last_unstale":"2023-11-28T17:41:44.331053+0000","last_undegraded":"2023-11-28T17:41:44.331053+0000","last_fullsized":"2023-11-28T17:41:44.331053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:58:47.412734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649511+0000","last_change":"2023-11-28T17:41:17.237500+0000","last_active":"2023-11-28T17:41:18.649511+0000","last_peered":"2023-11-28T17:41:18.649511+0000","last_clean":"2023-11-28T17:41:18.649511+0000","last_became_active":"2023-11-28T17:41:17.237211+0000","last_became_peered":"2023-11-28T17:41:17.237211+0000","last_unstale":"2023-11-28T17:41:18.649511+0000","last_undegraded":"2023-11-28T17:41:18.649511+0000","last_fullsized":"2023-11-28T17:41:18.649511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:13.675182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.238895+0000","last_change":"2023-11-28T17:41:16.258147+0000","last_active":"2023-11-28T17:41:39.238895+0000","last_peered":"2023-11-28T17:41:39.238895+0000","last_clean":"2023-11-28T17:41:39.238895+0000","last_became_active":"2023-11-28T17:41:16.257540+0000","last_became_peered":"2023-11-28T17:41:16.257540+0000","last_unstale":"2023-11-28T17:41:39.238895+0000","last_undegraded":"2023-11-28T17:41:39.238895+0000","last_fullsized":"2023-11-28T17:41:39.238895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:51:11.394507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338684+0000","last_change":"2023-11-28T17:41:17.247768+0000","last_active":"2023-11-28T17:41:18.338684+0000","last_peered":"2023-11-28T17:41:18.338684+0000","last_clean":"2023-11-28T17:41:18.338684+0000","last_became_active":"2023-11-28T17:41:17.247475+0000","last_became_peered":"2023-11-28T17:41:17.247475+0000","last_unstale":"2023-11-28T17:41:18.338684+0000","last_undegraded":"2023-11-28T17:41:18.338684+0000","last_fullsized":"2023-11-28T17:41:18.338684+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:18.555573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.244012+0000","last_change":"2023-11-28T17:41:16.262927+0000","last_active":"2023-11-28T17:41:39.244012+0000","last_peered":"2023-11-28T17:41:39.244012+0000","last_clean":"2023-11-28T17:41:39.244012+0000","last_became_active":"2023-11-28T17:41:16.262556+0000","last_became_peered":"2023-11-28T17:41:16.262556+0000","last_unstale":"2023-11-28T17:41:39.244012+0000","last_undegraded":"2023-11-28T17:41:39.244012+0000","last_fullsized":"2023-11-28T17:41:39.244012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:35:54.672978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1028,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632921+0000","last_change":"2023-11-28T17:41:17.239625+0000","last_active":"2023-11-28T17:41:19.632921+0000","last_peered":"2023-11-28T17:41:19.632921+0000","last_clean":"2023-11-28T17:41:19.632921+0000","last_became_active":"2023-11-28T17:41:17.238215+0000","last_became_peered":"2023-11-28T17:41:17.238215+0000","last_unstale":"2023-11-28T17:41:19.632921+0000","last_undegraded":"2023-11-28T17:41:19.632921+0000","last_fullsized":"2023-11-28T17:41:19.632921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:33:31.879592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'17","reported_seq":29,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.238291+0000","last_change":"2023-11-28T17:41:16.250674+0000","last_active":"2023-11-28T17:41:39.238291+0000","last_peered":"2023-11-28T17:41:39.238291+0000","last_clean":"2023-11-28T17:41:39.238291+0000","last_became_active":"2023-11-28T17:41:16.250312+0000","last_became_peered":"2023-11-28T17:41:16.250312+0000","last_unstale":"2023-11-28T17:41:39.238291+0000","last_undegraded":"2023-11-28T17:41:39.238291+0000","last_fullsized":"2023-11-28T17:41:39.238291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:04:05.342081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338315+0000","last_change":"2023-11-28T17:41:17.247675+0000","last_active":"2023-11-28T17:41:18.338315+0000","last_peered":"2023-11-28T17:41:18.338315+0000","last_clean":"2023-11-28T17:41:18.338315+0000","last_became_active":"2023-11-28T17:41:17.247346+0000","last_became_peered":"2023-11-28T17:41:17.247346+0000","last_unstale":"2023-11-28T17:41:18.338315+0000","last_undegraded":"2023-11-28T17:41:18.338315+0000","last_fullsized":"2023-11-28T17:41:18.338315+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:45:03.326583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.280691+0000","last_change":"2023-11-28T17:41:16.260893+0000","last_active":"2023-11-28T17:41:22.280691+0000","last_peered":"2023-11-28T17:41:22.280691+0000","last_clean":"2023-11-28T17:41:22.280691+0000","last_became_active":"2023-11-28T17:41:16.260637+0000","last_became_peered":"2023-11-28T17:41:16.260637+0000","last_unstale":"2023-11-28T17:41:22.280691+0000","last_undegraded":"2023-11-28T17:41:22.280691+0000","last_fullsized":"2023-11-28T17:41:22.280691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:07:24.032989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338811+0000","last_change":"2023-11-28T17:41:17.249202+0000","last_active":"2023-11-28T17:41:18.338811+0000","last_peered":"2023-11-28T17:41:18.338811+0000","last_clean":"2023-11-28T17:41:18.338811+0000","last_became_active":"2023-11-28T17:41:17.248187+0000","last_became_peered":"2023-11-28T17:41:17.248187+0000","last_unstale":"2023-11-28T17:41:18.338811+0000","last_undegraded":"2023-11-28T17:41:18.338811+0000","last_fullsized":"2023-11-28T17:41:18.338811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:23.973069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.328238+0000","last_change":"2023-11-28T17:41:16.244938+0000","last_active":"2023-11-28T17:41:44.328238+0000","last_peered":"2023-11-28T17:41:44.328238+0000","last_clean":"2023-11-28T17:41:44.328238+0000","last_became_active":"2023-11-28T17:41:16.244586+0000","last_became_peered":"2023-11-28T17:41:16.244586+0000","last_unstale":"2023-11-28T17:41:44.328238+0000","last_undegraded":"2023-11-28T17:41:44.328238+0000","last_fullsized":"2023-11-28T17:41:44.328238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:38.421966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339409+0000","last_change":"2023-11-28T17:41:17.248488+0000","last_active":"2023-11-28T17:41:18.339409+0000","last_peered":"2023-11-28T17:41:18.339409+0000","last_clean":"2023-11-28T17:41:18.339409+0000","last_became_active":"2023-11-28T17:41:17.248170+0000","last_became_peered":"2023-11-28T17:41:17.248170+0000","last_unstale":"2023-11-28T17:41:18.339409+0000","last_undegraded":"2023-11-28T17:41:18.339409+0000","last_fullsized":"2023-11-28T17:41:18.339409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:41.188370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.242265+0000","last_change":"2023-11-28T17:41:16.246192+0000","last_active":"2023-11-28T17:41:39.242265+0000","last_peered":"2023-11-28T17:41:39.242265+0000","last_clean":"2023-11-28T17:41:39.242265+0000","last_became_active":"2023-11-28T17:41:16.245379+0000","last_became_peered":"2023-11-28T17:41:16.245379+0000","last_unstale":"2023-11-28T17:41:39.242265+0000","last_undegraded":"2023-11-28T17:41:39.242265+0000","last_fullsized":"2023-11-28T17:41:39.242265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:23:08.664344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207475+0000","last_change":"2023-11-28T17:41:17.234273+0000","last_active":"2023-11-28T17:41:18.207475+0000","last_peered":"2023-11-28T17:41:18.207475+0000","last_clean":"2023-11-28T17:41:18.207475+0000","last_became_active":"2023-11-28T17:41:17.233756+0000","last_became_peered":"2023-11-28T17:41:17.233756+0000","last_unstale":"2023-11-28T17:41:18.207475+0000","last_undegraded":"2023-11-28T17:41:18.207475+0000","last_fullsized":"2023-11-28T17:41:18.207475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:18:40.539329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.242692+0000","last_change":"2023-11-28T17:41:16.257952+0000","last_active":"2023-11-28T17:41:39.242692+0000","last_peered":"2023-11-28T17:41:39.242692+0000","last_clean":"2023-11-28T17:41:39.242692+0000","last_became_active":"2023-11-28T17:41:16.257659+0000","last_became_peered":"2023-11-28T17:41:16.257659+0000","last_unstale":"2023-11-28T17:41:39.242692+0000","last_undegraded":"2023-11-28T17:41:39.242692+0000","last_fullsized":"2023-11-28T17:41:39.242692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:28:12.131592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1610,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338433+0000","last_change":"2023-11-28T17:41:17.239069+0000","last_active":"2023-11-28T17:41:18.338433+0000","last_peered":"2023-11-28T17:41:18.338433+0000","last_clean":"2023-11-28T17:41:18.338433+0000","last_became_active":"2023-11-28T17:41:17.238548+0000","last_became_peered":"2023-11-28T17:41:17.238548+0000","last_unstale":"2023-11-28T17:41:18.338433+0000","last_undegraded":"2023-11-28T17:41:18.338433+0000","last_fullsized":"2023-11-28T17:41:18.338433+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:12:56.686814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206759+0000","last_change":"2023-11-28T17:41:12.444311+0000","last_active":"2023-11-28T17:41:17.206759+0000","last_peered":"2023-11-28T17:41:17.206759+0000","last_clean":"2023-11-28T17:41:17.206759+0000","last_became_active":"2023-11-28T17:41:12.442193+0000","last_became_peered":"2023-11-28T17:41:12.442193+0000","last_unstale":"2023-11-28T17:41:17.206759+0000","last_undegraded":"2023-11-28T17:41:17.206759+0000","last_fullsized":"2023-11-28T17:41:17.206759+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:10.865936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:10.865936+0000","last_clean_scrub_stamp":"2023-11-28T17:41:10.865936+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:12:48.912392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.237404+0000","last_change":"2023-11-28T17:41:16.253023+0000","last_active":"2023-11-28T17:41:39.237404+0000","last_peered":"2023-11-28T17:41:39.237404+0000","last_clean":"2023-11-28T17:41:39.237404+0000","last_became_active":"2023-11-28T17:41:16.245730+0000","last_became_peered":"2023-11-28T17:41:16.245730+0000","last_unstale":"2023-11-28T17:41:39.237404+0000","last_undegraded":"2023-11-28T17:41:39.237404+0000","last_fullsized":"2023-11-28T17:41:39.237404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:05.970917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568226+0000","last_change":"2023-11-28T17:41:17.246677+0000","last_active":"2023-11-28T17:41:18.568226+0000","last_peered":"2023-11-28T17:41:18.568226+0000","last_clean":"2023-11-28T17:41:18.568226+0000","last_became_active":"2023-11-28T17:41:17.246373+0000","last_became_peered":"2023-11-28T17:41:17.246373+0000","last_unstale":"2023-11-28T17:41:18.568226+0000","last_undegraded":"2023-11-28T17:41:18.568226+0000","last_fullsized":"2023-11-28T17:41:18.568226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:22:18.623331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:33.235683+0000","last_change":"2023-11-28T17:41:16.272843+0000","last_active":"2023-11-28T17:41:33.235683+0000","last_peered":"2023-11-28T17:41:33.235683+0000","last_clean":"2023-11-28T17:41:33.235683+0000","last_became_active":"2023-11-28T17:41:16.272527+0000","last_became_peered":"2023-11-28T17:41:16.272527+0000","last_unstale":"2023-11-28T17:41:33.235683+0000","last_undegraded":"2023-11-28T17:41:33.235683+0000","last_fullsized":"2023-11-28T17:41:33.235683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:55:52.185953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649479+0000","last_change":"2023-11-28T17:41:17.240890+0000","last_active":"2023-11-28T17:41:18.649479+0000","last_peered":"2023-11-28T17:41:18.649479+0000","last_clean":"2023-11-28T17:41:18.649479+0000","last_became_active":"2023-11-28T17:41:17.240571+0000","last_became_peered":"2023-11-28T17:41:17.240571+0000","last_unstale":"2023-11-28T17:41:18.649479+0000","last_undegraded":"2023-11-28T17:41:18.649479+0000","last_fullsized":"2023-11-28T17:41:18.649479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:58.086322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.329294+0000","last_change":"2023-11-28T17:41:16.251925+0000","last_active":"2023-11-28T17:41:44.329294+0000","last_peered":"2023-11-28T17:41:44.329294+0000","last_clean":"2023-11-28T17:41:44.329294+0000","last_became_active":"2023-11-28T17:41:16.251482+0000","last_became_peered":"2023-11-28T17:41:16.251482+0000","last_unstale":"2023-11-28T17:41:44.329294+0000","last_undegraded":"2023-11-28T17:41:44.329294+0000","last_fullsized":"2023-11-28T17:41:44.329294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:07:09.590250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632279+0000","last_change":"2023-11-28T17:41:17.223810+0000","last_active":"2023-11-28T17:41:19.632279+0000","last_peered":"2023-11-28T17:41:19.632279+0000","last_clean":"2023-11-28T17:41:19.632279+0000","last_became_active":"2023-11-28T17:41:17.223244+0000","last_became_peered":"2023-11-28T17:41:17.223244+0000","last_unstale":"2023-11-28T17:41:19.632279+0000","last_undegraded":"2023-11-28T17:41:19.632279+0000","last_fullsized":"2023-11-28T17:41:19.632279+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:31:22.132844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.335847+0000","last_change":"2023-11-28T17:41:16.252402+0000","last_active":"2023-11-28T17:41:44.335847+0000","last_peered":"2023-11-28T17:41:44.335847+0000","last_clean":"2023-11-28T17:41:44.335847+0000","last_became_active":"2023-11-28T17:41:16.252061+0000","last_became_peered":"2023-11-28T17:41:16.252061+0000","last_unstale":"2023-11-28T17:41:44.335847+0000","last_undegraded":"2023-11-28T17:41:44.335847+0000","last_fullsized":"2023-11-28T17:41:44.335847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:20:11.890036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207754+0000","last_change":"2023-11-28T17:41:17.234320+0000","last_active":"2023-11-28T17:41:18.207754+0000","last_peered":"2023-11-28T17:41:18.207754+0000","last_clean":"2023-11-28T17:41:18.207754+0000","last_became_active":"2023-11-28T17:41:17.233738+0000","last_became_peered":"2023-11-28T17:41:17.233738+0000","last_unstale":"2023-11-28T17:41:18.207754+0000","last_undegraded":"2023-11-28T17:41:18.207754+0000","last_fullsized":"2023-11-28T17:41:18.207754+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:31:10.257906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632533+0000","last_change":"2023-11-28T17:41:16.245394+0000","last_active":"2023-11-28T17:41:19.632533+0000","last_peered":"2023-11-28T17:41:19.632533+0000","last_clean":"2023-11-28T17:41:19.632533+0000","last_became_active":"2023-11-28T17:41:16.242985+0000","last_became_peered":"2023-11-28T17:41:16.242985+0000","last_unstale":"2023-11-28T17:41:19.632533+0000","last_undegraded":"2023-11-28T17:41:19.632533+0000","last_fullsized":"2023-11-28T17:41:19.632533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:48.728960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207612+0000","last_change":"2023-11-28T17:41:17.233987+0000","last_active":"2023-11-28T17:41:18.207612+0000","last_peered":"2023-11-28T17:41:18.207612+0000","last_clean":"2023-11-28T17:41:18.207612+0000","last_became_active":"2023-11-28T17:41:17.233430+0000","last_became_peered":"2023-11-28T17:41:17.233430+0000","last_unstale":"2023-11-28T17:41:18.207612+0000","last_undegraded":"2023-11-28T17:41:18.207612+0000","last_fullsized":"2023-11-28T17:41:18.207612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:51:44.816033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.238818+0000","last_change":"2023-11-28T17:41:16.273047+0000","last_active":"2023-11-28T17:41:39.238818+0000","last_peered":"2023-11-28T17:41:39.238818+0000","last_clean":"2023-11-28T17:41:39.238818+0000","last_became_active":"2023-11-28T17:41:16.272667+0000","last_became_peered":"2023-11-28T17:41:16.272667+0000","last_unstale":"2023-11-28T17:41:39.238818+0000","last_undegraded":"2023-11-28T17:41:39.238818+0000","last_fullsized":"2023-11-28T17:41:39.238818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:19:37.472121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632439+0000","last_change":"2023-11-28T17:41:17.232465+0000","last_active":"2023-11-28T17:41:19.632439+0000","last_peered":"2023-11-28T17:41:19.632439+0000","last_clean":"2023-11-28T17:41:19.632439+0000","last_became_active":"2023-11-28T17:41:17.230496+0000","last_became_peered":"2023-11-28T17:41:17.230496+0000","last_unstale":"2023-11-28T17:41:19.632439+0000","last_undegraded":"2023-11-28T17:41:19.632439+0000","last_fullsized":"2023-11-28T17:41:19.632439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:52:08.590259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'7","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.243168+0000","last_change":"2023-11-28T17:41:16.258717+0000","last_active":"2023-11-28T17:41:39.243168+0000","last_peered":"2023-11-28T17:41:39.243168+0000","last_clean":"2023-11-28T17:41:39.243168+0000","last_became_active":"2023-11-28T17:41:16.258388+0000","last_became_peered":"2023-11-28T17:41:16.258388+0000","last_unstale":"2023-11-28T17:41:39.243168+0000","last_undegraded":"2023-11-28T17:41:39.243168+0000","last_fullsized":"2023-11-28T17:41:39.243168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:17.845137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568998+0000","last_change":"2023-11-28T17:41:17.239189+0000","last_active":"2023-11-28T17:41:18.568998+0000","last_peered":"2023-11-28T17:41:18.568998+0000","last_clean":"2023-11-28T17:41:18.568998+0000","last_became_active":"2023-11-28T17:41:17.238829+0000","last_became_peered":"2023-11-28T17:41:17.238829+0000","last_unstale":"2023-11-28T17:41:18.568998+0000","last_undegraded":"2023-11-28T17:41:18.568998+0000","last_fullsized":"2023-11-28T17:41:18.568998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:10.923572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.337125+0000","last_change":"2023-11-28T17:41:16.268334+0000","last_active":"2023-11-28T17:41:44.337125+0000","last_peered":"2023-11-28T17:41:44.337125+0000","last_clean":"2023-11-28T17:41:44.337125+0000","last_became_active":"2023-11-28T17:41:16.268015+0000","last_became_peered":"2023-11-28T17:41:16.268015+0000","last_unstale":"2023-11-28T17:41:44.337125+0000","last_undegraded":"2023-11-28T17:41:44.337125+0000","last_fullsized":"2023-11-28T17:41:44.337125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:57:22.031553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650340+0000","last_change":"2023-11-28T17:41:17.225161+0000","last_active":"2023-11-28T17:41:18.650340+0000","last_peered":"2023-11-28T17:41:18.650340+0000","last_clean":"2023-11-28T17:41:18.650340+0000","last_became_active":"2023-11-28T17:41:17.224420+0000","last_became_peered":"2023-11-28T17:41:17.224420+0000","last_unstale":"2023-11-28T17:41:18.650340+0000","last_undegraded":"2023-11-28T17:41:18.650340+0000","last_fullsized":"2023-11-28T17:41:18.650340+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:25:11.426980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:27.331262+0000","last_change":"2023-11-28T17:41:16.272884+0000","last_active":"2023-11-28T17:41:27.331262+0000","last_peered":"2023-11-28T17:41:27.331262+0000","last_clean":"2023-11-28T17:41:27.331262+0000","last_became_active":"2023-11-28T17:41:16.272557+0000","last_became_peered":"2023-11-28T17:41:16.272557+0000","last_unstale":"2023-11-28T17:41:27.331262+0000","last_undegraded":"2023-11-28T17:41:27.331262+0000","last_fullsized":"2023-11-28T17:41:27.331262+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:56.634559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206149+0000","last_change":"2023-11-28T17:41:17.238145+0000","last_active":"2023-11-28T17:41:18.206149+0000","last_peered":"2023-11-28T17:41:18.206149+0000","last_clean":"2023-11-28T17:41:18.206149+0000","last_became_active":"2023-11-28T17:41:17.237774+0000","last_became_peered":"2023-11-28T17:41:17.237774+0000","last_unstale":"2023-11-28T17:41:18.206149+0000","last_undegraded":"2023-11-28T17:41:18.206149+0000","last_fullsized":"2023-11-28T17:41:18.206149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:44.723180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.333438+0000","last_change":"2023-11-28T17:41:16.262318+0000","last_active":"2023-11-28T17:41:44.333438+0000","last_peered":"2023-11-28T17:41:44.333438+0000","last_clean":"2023-11-28T17:41:44.333438+0000","last_became_active":"2023-11-28T17:41:16.262014+0000","last_became_peered":"2023-11-28T17:41:16.262014+0000","last_unstale":"2023-11-28T17:41:44.333438+0000","last_undegraded":"2023-11-28T17:41:44.333438+0000","last_fullsized":"2023-11-28T17:41:44.333438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:17:15.470393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568031+0000","last_change":"2023-11-28T17:41:17.229233+0000","last_active":"2023-11-28T17:41:18.568031+0000","last_peered":"2023-11-28T17:41:18.568031+0000","last_clean":"2023-11-28T17:41:18.568031+0000","last_became_active":"2023-11-28T17:41:17.228885+0000","last_became_peered":"2023-11-28T17:41:17.228885+0000","last_unstale":"2023-11-28T17:41:18.568031+0000","last_undegraded":"2023-11-28T17:41:18.568031+0000","last_fullsized":"2023-11-28T17:41:18.568031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:10:24.247307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.236303+0000","last_change":"2023-11-28T17:41:16.246533+0000","last_active":"2023-11-28T17:41:39.236303+0000","last_peered":"2023-11-28T17:41:39.236303+0000","last_clean":"2023-11-28T17:41:39.236303+0000","last_became_active":"2023-11-28T17:41:16.244067+0000","last_became_peered":"2023-11-28T17:41:16.244067+0000","last_unstale":"2023-11-28T17:41:39.236303+0000","last_undegraded":"2023-11-28T17:41:39.236303+0000","last_fullsized":"2023-11-28T17:41:39.236303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:02:20.862297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568303+0000","last_change":"2023-11-28T17:41:17.236159+0000","last_active":"2023-11-28T17:41:18.568303+0000","last_peered":"2023-11-28T17:41:18.568303+0000","last_clean":"2023-11-28T17:41:18.568303+0000","last_became_active":"2023-11-28T17:41:17.235700+0000","last_became_peered":"2023-11-28T17:41:17.235700+0000","last_unstale":"2023-11-28T17:41:18.568303+0000","last_undegraded":"2023-11-28T17:41:18.568303+0000","last_fullsized":"2023-11-28T17:41:18.568303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:21.900805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:27.330503+0000","last_change":"2023-11-28T17:41:16.268969+0000","last_active":"2023-11-28T17:41:27.330503+0000","last_peered":"2023-11-28T17:41:27.330503+0000","last_clean":"2023-11-28T17:41:27.330503+0000","last_became_active":"2023-11-28T17:41:16.268717+0000","last_became_peered":"2023-11-28T17:41:16.268717+0000","last_unstale":"2023-11-28T17:41:27.330503+0000","last_undegraded":"2023-11-28T17:41:27.330503+0000","last_fullsized":"2023-11-28T17:41:27.330503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:03:29.120624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633192+0000","last_change":"2023-11-28T17:41:17.244376+0000","last_active":"2023-11-28T17:41:19.633192+0000","last_peered":"2023-11-28T17:41:19.633192+0000","last_clean":"2023-11-28T17:41:19.633192+0000","last_became_active":"2023-11-28T17:41:17.244034+0000","last_became_peered":"2023-11-28T17:41:17.244034+0000","last_unstale":"2023-11-28T17:41:19.633192+0000","last_undegraded":"2023-11-28T17:41:19.633192+0000","last_fullsized":"2023-11-28T17:41:19.633192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:15:56.198732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208198+0000","last_change":"2023-11-28T17:41:17.233749+0000","last_active":"2023-11-28T17:41:18.208198+0000","last_peered":"2023-11-28T17:41:18.208198+0000","last_clean":"2023-11-28T17:41:18.208198+0000","last_became_active":"2023-11-28T17:41:17.232992+0000","last_became_peered":"2023-11-28T17:41:17.232992+0000","last_unstale":"2023-11-28T17:41:18.208198+0000","last_undegraded":"2023-11-28T17:41:18.208198+0000","last_fullsized":"2023-11-28T17:41:18.208198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:00:43.698244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.338678+0000","last_change":"2023-11-28T17:41:16.269465+0000","last_active":"2023-11-28T17:41:44.338678+0000","last_peered":"2023-11-28T17:41:44.338678+0000","last_clean":"2023-11-28T17:41:44.338678+0000","last_became_active":"2023-11-28T17:41:16.269147+0000","last_became_peered":"2023-11-28T17:41:16.269147+0000","last_unstale":"2023-11-28T17:41:44.338678+0000","last_undegraded":"2023-11-28T17:41:44.338678+0000","last_fullsized":"2023-11-28T17:41:44.338678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:57:58.121232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207346+0000","last_change":"2023-11-28T17:41:17.243498+0000","last_active":"2023-11-28T17:41:18.207346+0000","last_peered":"2023-11-28T17:41:18.207346+0000","last_clean":"2023-11-28T17:41:18.207346+0000","last_became_active":"2023-11-28T17:41:17.243169+0000","last_became_peered":"2023-11-28T17:41:17.243169+0000","last_unstale":"2023-11-28T17:41:18.207346+0000","last_undegraded":"2023-11-28T17:41:18.207346+0000","last_fullsized":"2023-11-28T17:41:18.207346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:11:31.509921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'12","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.340143+0000","last_change":"2023-11-28T17:41:16.270231+0000","last_active":"2023-11-28T17:41:44.340143+0000","last_peered":"2023-11-28T17:41:44.340143+0000","last_clean":"2023-11-28T17:41:44.340143+0000","last_became_active":"2023-11-28T17:41:16.269938+0000","last_became_peered":"2023-11-28T17:41:16.269938+0000","last_unstale":"2023-11-28T17:41:44.340143+0000","last_undegraded":"2023-11-28T17:41:44.340143+0000","last_fullsized":"2023-11-28T17:41:44.340143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:45.654658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650357+0000","last_change":"2023-11-28T17:41:17.245856+0000","last_active":"2023-11-28T17:41:18.650357+0000","last_peered":"2023-11-28T17:41:18.650357+0000","last_clean":"2023-11-28T17:41:18.650357+0000","last_became_active":"2023-11-28T17:41:17.245624+0000","last_became_peered":"2023-11-28T17:41:17.245624+0000","last_unstale":"2023-11-28T17:41:18.650357+0000","last_undegraded":"2023-11-28T17:41:18.650357+0000","last_fullsized":"2023-11-28T17:41:18.650357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:03:10.206506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.279322+0000","last_change":"2023-11-28T17:41:16.260989+0000","last_active":"2023-11-28T17:41:22.279322+0000","last_peered":"2023-11-28T17:41:22.279322+0000","last_clean":"2023-11-28T17:41:22.279322+0000","last_became_active":"2023-11-28T17:41:16.260686+0000","last_became_peered":"2023-11-28T17:41:16.260686+0000","last_unstale":"2023-11-28T17:41:22.279322+0000","last_undegraded":"2023-11-28T17:41:22.279322+0000","last_fullsized":"2023-11-28T17:41:22.279322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:58:44.193791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568672+0000","last_change":"2023-11-28T17:41:17.240803+0000","last_active":"2023-11-28T17:41:18.568672+0000","last_peered":"2023-11-28T17:41:18.568672+0000","last_clean":"2023-11-28T17:41:18.568672+0000","last_became_active":"2023-11-28T17:41:17.240442+0000","last_became_peered":"2023-11-28T17:41:17.240442+0000","last_unstale":"2023-11-28T17:41:18.568672+0000","last_undegraded":"2023-11-28T17:41:18.568672+0000","last_fullsized":"2023-11-28T17:41:18.568672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:28:21.223788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.335954+0000","last_change":"2023-11-28T17:41:16.266520+0000","last_active":"2023-11-28T17:41:44.335954+0000","last_peered":"2023-11-28T17:41:44.335954+0000","last_clean":"2023-11-28T17:41:44.335954+0000","last_became_active":"2023-11-28T17:41:16.266245+0000","last_became_peered":"2023-11-28T17:41:16.266245+0000","last_unstale":"2023-11-28T17:41:44.335954+0000","last_undegraded":"2023-11-28T17:41:44.335954+0000","last_fullsized":"2023-11-28T17:41:44.335954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:18:28.732421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":9794,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":213,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":19588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":187,"ondisk_log_size":187,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65062,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509453,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":669440,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.49}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.595}]}]},{"osd":6,"up_from":15,"seq":64424509451,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":79814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3680000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.593}]}]},{"osd":5,"up_from":15,"seq":64424509452,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":80904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.214}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.694}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.643}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509452,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":245760,"data_stored":77862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3660000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.405}]}]},{"osd":3,"up_from":15,"seq":64424509451,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":78968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509452,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":78064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7390000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3170000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.49}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2229999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509453,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":315392,"data_stored":667774,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.175}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2090000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":78434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T17:41:59.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:41:59.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.471+0000 7f9846367640 1 Processor -- start 2023-11-28T17:41:59.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.471+0000 7f9846367640 1 -- start start 2023-11-28T17:41:59.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.473+0000 7f9846367640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9840150ac0 0x7f98400716e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:59.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.473+0000 7f9846367640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9840157300 0x7f9840071c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:59.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.473+0000 7f9846367640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9840150ea0 con 0x7f984006b1e0 2023-11-28T17:41:59.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.473+0000 7f9846367640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f984015ad90 con 0x7f9840157300 2023-11-28T17:41:59.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.473+0000 7f9846367640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98401567a0 con 0x7f9840150ac0 2023-11-28T17:41:59.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.473+0000 7f9845b66640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9840157300 0x7f9840071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:59.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.473+0000 7f9845b66640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9840157300 0x7f9840071c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57094/0 (socket says 172.21.15.98:57094) 2023-11-28T17:41:59.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.473+0000 7f9845b66640 1 -- 172.21.15.98:0/1012874797 learned_addr learned my addr 172.21.15.98:0/1012874797 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:59.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.473+0000 7f9845365640 1 --2- 172.21.15.98:0/1012874797 >> v2:172.21.15.113:3300/0 conn(0x7f9840150ac0 0x7f98400716e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:59.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.474+0000 7f9845b66640 1 -- 172.21.15.98:0/1012874797 >> v2:172.21.15.113:3300/0 conn(0x7f9840150ac0 msgr2=0x7f98400716e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:59.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.474+0000 7f9845b66640 1 --2- 172.21.15.98:0/1012874797 >> v2:172.21.15.113:3300/0 conn(0x7f9840150ac0 0x7f98400716e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.474+0000 7f9845b66640 1 -- 172.21.15.98:0/1012874797 >> v1:172.21.15.113:6789/0 conn(0x7f984006b1e0 legacy=0x7f9840156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:59.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.474+0000 7f9845b66640 1 -- 172.21.15.98:0/1012874797 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9840159f30 con 0x7f9840157300 2023-11-28T17:41:59.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.475+0000 7f9845b66640 1 --2- 172.21.15.98:0/1012874797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9840157300 0x7f9840071c20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9840066b40 tx=0x7f983802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91d97d8cf41fd2e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:59.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.475+0000 7f9837fff640 1 -- 172.21.15.98:0/1012874797 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983802cb20 con 0x7f9840157300 2023-11-28T17:41:59.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.475+0000 7f9837fff640 1 -- 172.21.15.98:0/1012874797 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983802cc80 con 0x7f9840157300 2023-11-28T17:41:59.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.475+0000 7f9837fff640 1 -- 172.21.15.98:0/1012874797 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98380343f0 con 0x7f9840157300 2023-11-28T17:41:59.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.476+0000 7f9846367640 1 -- 172.21.15.98:0/1012874797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9840157300 msgr2=0x7f9840071c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:59.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.476+0000 7f9846367640 1 --2- 172.21.15.98:0/1012874797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9840157300 0x7f9840071c20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9840066b40 tx=0x7f983802bf80 comp rx=0 tx=0).stop 2023-11-28T17:41:59.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.476+0000 7f9846367640 1 -- 172.21.15.98:0/1012874797 shutdown_connections 2023-11-28T17:41:59.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.476+0000 7f9846367640 1 --2- 172.21.15.98:0/1012874797 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9840157300 0x7f9840071c20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.476+0000 7f9846367640 1 --2- 172.21.15.98:0/1012874797 >> v2:172.21.15.113:3300/0 conn(0x7f9840150ac0 0x7f98400716e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.476+0000 7f9846367640 1 -- 172.21.15.98:0/1012874797 >> 172.21.15.98:0/1012874797 conn(0x7f9840066230 msgr2=0x7f984006a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:59.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.476+0000 7f9846367640 1 -- 172.21.15.98:0/1012874797 shutdown_connections 2023-11-28T17:41:59.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.476+0000 7f9846367640 1 -- 172.21.15.98:0/1012874797 wait complete. 2023-11-28T17:41:59.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.477+0000 7f9846367640 1 Processor -- start 2023-11-28T17:41:59.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.477+0000 7f9846367640 1 -- start start 2023-11-28T17:41:59.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.478+0000 7f9846367640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f984006b1e0 0x7f98401464b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.478+0000 7f9846367640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9840150ac0 0x7f9840163830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.478+0000 7f9846367640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9840157300 0x7f9840163d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.478+0000 7f9846367640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9840162980 con 0x7f9840157300 2023-11-28T17:41:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.478+0000 7f9846367640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98401633a0 con 0x7f9840150ac0 2023-11-28T17:41:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.478+0000 7f9846367640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98401f31a0 con 0x7f984006b1e0 2023-11-28T17:41:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.479+0000 7f9845b66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9840157300 0x7f9840163d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.479+0000 7f9845b66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9840157300 0x7f9840163d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57110/0 (socket says 172.21.15.98:57110) 2023-11-28T17:41:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.479+0000 7f9845b66640 1 -- 172.21.15.98:0/1614777493 learned_addr learned my addr 172.21.15.98:0/1614777493 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.479+0000 7f9845365640 1 --2- 172.21.15.98:0/1614777493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f984006b1e0 0x7f98401464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.479+0000 7f9845b66640 1 -- 172.21.15.98:0/1614777493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f984006b1e0 msgr2=0x7f98401464b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.479+0000 7f9845b66640 1 --2- 172.21.15.98:0/1614777493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f984006b1e0 0x7f98401464b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.479+0000 7f9845b66640 1 -- 172.21.15.98:0/1614777493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9840150ac0 msgr2=0x7f9840163830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.479+0000 7f9844b64640 1 --2- 172.21.15.98:0/1614777493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9840150ac0 0x7f9840163830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.479+0000 7f9845b66640 1 --2- 172.21.15.98:0/1614777493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9840150ac0 0x7f9840163830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.479+0000 7f9845b66640 1 -- 172.21.15.98:0/1614777493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9838009b30 con 0x7f9840157300 2023-11-28T17:41:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.479+0000 7f9845365640 1 --2- 172.21.15.98:0/1614777493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f984006b1e0 0x7f98401464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.479+0000 7f9845b66640 1 --2- 172.21.15.98:0/1614777493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9840157300 0x7f9840163d70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9838036040 tx=0x7f9838004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.480+0000 7f9835ffb640 1 -- 172.21.15.98:0/1614777493 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9838009510 con 0x7f9840157300 2023-11-28T17:41:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.480+0000 7f9835ffb640 1 -- 172.21.15.98:0/1614777493 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9838007e50 con 0x7f9840157300 2023-11-28T17:41:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.480+0000 7f9835ffb640 1 -- 172.21.15.98:0/1614777493 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983803e690 con 0x7f9840157300 2023-11-28T17:41:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.480+0000 7f9846367640 1 -- 172.21.15.98:0/1614777493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98401f3480 con 0x7f9840157300 2023-11-28T17:41:59.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.480+0000 7f9846367640 1 -- 172.21.15.98:0/1614777493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98401f3a40 con 0x7f9840157300 2023-11-28T17:41:59.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.481+0000 7f9846367640 1 -- 172.21.15.98:0/1614777493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9840163550 con 0x7f9840157300 2023-11-28T17:41:59.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.482+0000 7f9835ffb640 1 -- 172.21.15.98:0/1614777493 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f983803e7f0 con 0x7f9840157300 2023-11-28T17:41:59.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.483+0000 7f9835ffb640 1 --2- 172.21.15.98:0/1614777493 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f982406b2d0 0x7f982406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:59.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.483+0000 7f9835ffb640 1 -- 172.21.15.98:0/1614777493 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98380ac620 con 0x7f9840157300 2023-11-28T17:41:59.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.483+0000 7f9845365640 1 --2- 172.21.15.98:0/1614777493 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f982406b2d0 0x7f982406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:59.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.484+0000 7f9845365640 1 --2- 172.21.15.98:0/1614777493 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f982406b2d0 0x7f982406d790 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f983c008890 tx=0x7f983c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:59.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.488+0000 7f9835ffb640 1 -- 172.21.15.98:0/1614777493 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9840163550 con 0x7f9840157300 2023-11-28T17:41:59.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.673+0000 7f9846367640 1 -- 172.21.15.98:0/1614777493 --> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f984015a7d0 con 0x7f982406b2d0 2023-11-28T17:41:59.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.681+0000 7f9835ffb640 1 -- 172.21.15.98:0/1614777493 <== mgr.4098 v2:172.21.15.98:6832/2274392793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264615 (secure 0 0 0) 0x7f983c00c040 con 0x7f982406b2d0 2023-11-28T17:41:59.684 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:41:59.687 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-11-28T17:41:59.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.688+0000 7f9846367640 1 -- 172.21.15.98:0/1614777493 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f982406b2d0 msgr2=0x7f982406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:59.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.688+0000 7f9846367640 1 --2- 172.21.15.98:0/1614777493 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f982406b2d0 0x7f982406d790 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f983c008890 tx=0x7f983c008040 comp rx=0 tx=0).stop 2023-11-28T17:41:59.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.688+0000 7f9846367640 1 -- 172.21.15.98:0/1614777493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9840157300 msgr2=0x7f9840163d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:59.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.688+0000 7f9846367640 1 --2- 172.21.15.98:0/1614777493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9840157300 0x7f9840163d70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9838036040 tx=0x7f9838004a40 comp rx=0 tx=0).stop 2023-11-28T17:41:59.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.688+0000 7f9846367640 1 -- 172.21.15.98:0/1614777493 shutdown_connections 2023-11-28T17:41:59.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.688+0000 7f9846367640 1 --2- 172.21.15.98:0/1614777493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9840157300 0x7f9840163d70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.688+0000 7f9846367640 1 --2- 172.21.15.98:0/1614777493 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f982406b2d0 0x7f982406d790 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.688+0000 7f9846367640 1 --2- 172.21.15.98:0/1614777493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9840150ac0 0x7f9840163830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.688+0000 7f9846367640 1 --2- 172.21.15.98:0/1614777493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f984006b1e0 0x7f98401464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.688+0000 7f9846367640 1 -- 172.21.15.98:0/1614777493 >> 172.21.15.98:0/1614777493 conn(0x7f9840066230 msgr2=0x7f98400c71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:59.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.688+0000 7f9846367640 1 -- 172.21.15.98:0/1614777493 shutdown_connections 2023-11-28T17:41:59.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.688+0000 7f9846367640 1 -- 172.21.15.98:0/1614777493 wait complete. 2023-11-28T17:41:59.708 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2023-11-28T17:41:58.250229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600162,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":372,"num_write_kb":2486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":304,"ondisk_log_size":304,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218240,"kb_used_data":2112,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756480,"statfs":{"total":773094113280,"available":772870635520,"internally_reserved":0,"allocated":2162688,"data_stored":1811260,"data_compressed":65062,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1782,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004084"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339217+0000","last_change":"2023-11-28T17:41:17.238145+0000","last_active":"2023-11-28T17:41:18.339217+0000","last_peered":"2023-11-28T17:41:18.339217+0000","last_clean":"2023-11-28T17:41:18.339217+0000","last_became_active":"2023-11-28T17:41:17.237776+0000","last_became_peered":"2023-11-28T17:41:17.237776+0000","last_unstale":"2023-11-28T17:41:18.339217+0000","last_undegraded":"2023-11-28T17:41:18.339217+0000","last_fullsized":"2023-11-28T17:41:18.339217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:41:06.902665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632799+0000","last_change":"2023-11-28T17:41:17.231913+0000","last_active":"2023-11-28T17:41:19.632799+0000","last_peered":"2023-11-28T17:41:19.632799+0000","last_clean":"2023-11-28T17:41:19.632799+0000","last_became_active":"2023-11-28T17:41:17.231622+0000","last_became_peered":"2023-11-28T17:41:17.231622+0000","last_unstale":"2023-11-28T17:41:19.632799+0000","last_undegraded":"2023-11-28T17:41:19.632799+0000","last_fullsized":"2023-11-28T17:41:19.632799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:53:06.911223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649842+0000","last_change":"2023-11-28T17:41:17.230232+0000","last_active":"2023-11-28T17:41:18.649842+0000","last_peered":"2023-11-28T17:41:18.649842+0000","last_clean":"2023-11-28T17:41:18.649842+0000","last_became_active":"2023-11-28T17:41:17.229795+0000","last_became_peered":"2023-11-28T17:41:17.229795+0000","last_unstale":"2023-11-28T17:41:18.649842+0000","last_undegraded":"2023-11-28T17:41:18.649842+0000","last_fullsized":"2023-11-28T17:41:18.649842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:09:09.507547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207139+0000","last_change":"2023-11-28T17:41:17.234948+0000","last_active":"2023-11-28T17:41:18.207139+0000","last_peered":"2023-11-28T17:41:18.207139+0000","last_clean":"2023-11-28T17:41:18.207139+0000","last_became_active":"2023-11-28T17:41:17.233674+0000","last_became_peered":"2023-11-28T17:41:17.233674+0000","last_unstale":"2023-11-28T17:41:18.207139+0000","last_undegraded":"2023-11-28T17:41:18.207139+0000","last_fullsized":"2023-11-28T17:41:18.207139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:45:10.831719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338225+0000","last_change":"2023-11-28T17:41:17.233908+0000","last_active":"2023-11-28T17:41:18.338225+0000","last_peered":"2023-11-28T17:41:18.338225+0000","last_clean":"2023-11-28T17:41:18.338225+0000","last_became_active":"2023-11-28T17:41:17.233589+0000","last_became_peered":"2023-11-28T17:41:17.233589+0000","last_unstale":"2023-11-28T17:41:18.338225+0000","last_undegraded":"2023-11-28T17:41:18.338225+0000","last_fullsized":"2023-11-28T17:41:18.338225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:21:37.379103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206516+0000","last_change":"2023-11-28T17:41:17.239186+0000","last_active":"2023-11-28T17:41:18.206516+0000","last_peered":"2023-11-28T17:41:18.206516+0000","last_clean":"2023-11-28T17:41:18.206516+0000","last_became_active":"2023-11-28T17:41:17.233898+0000","last_became_peered":"2023-11-28T17:41:17.233898+0000","last_unstale":"2023-11-28T17:41:18.206516+0000","last_undegraded":"2023-11-28T17:41:18.206516+0000","last_fullsized":"2023-11-28T17:41:18.206516+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:08:54.918589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338198+0000","last_change":"2023-11-28T17:41:17.227923+0000","last_active":"2023-11-28T17:41:18.338198+0000","last_peered":"2023-11-28T17:41:18.338198+0000","last_clean":"2023-11-28T17:41:18.338198+0000","last_became_active":"2023-11-28T17:41:17.227642+0000","last_became_peered":"2023-11-28T17:41:17.227642+0000","last_unstale":"2023-11-28T17:41:18.338198+0000","last_undegraded":"2023-11-28T17:41:18.338198+0000","last_fullsized":"2023-11-28T17:41:18.338198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:33:29.400533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567710+0000","last_change":"2023-11-28T17:41:17.239818+0000","last_active":"2023-11-28T17:41:18.567710+0000","last_peered":"2023-11-28T17:41:18.567710+0000","last_clean":"2023-11-28T17:41:18.567710+0000","last_became_active":"2023-11-28T17:41:17.239594+0000","last_became_peered":"2023-11-28T17:41:17.239594+0000","last_unstale":"2023-11-28T17:41:18.567710+0000","last_undegraded":"2023-11-28T17:41:18.567710+0000","last_fullsized":"2023-11-28T17:41:18.567710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:23.969471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339318+0000","last_change":"2023-11-28T17:41:17.243214+0000","last_active":"2023-11-28T17:41:18.339318+0000","last_peered":"2023-11-28T17:41:18.339318+0000","last_clean":"2023-11-28T17:41:18.339318+0000","last_became_active":"2023-11-28T17:41:17.242933+0000","last_became_peered":"2023-11-28T17:41:17.242933+0000","last_unstale":"2023-11-28T17:41:18.339318+0000","last_undegraded":"2023-11-28T17:41:18.339318+0000","last_fullsized":"2023-11-28T17:41:18.339318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:47:47.697137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633703+0000","last_change":"2023-11-28T17:41:17.242507+0000","last_active":"2023-11-28T17:41:19.633703+0000","last_peered":"2023-11-28T17:41:19.633703+0000","last_clean":"2023-11-28T17:41:19.633703+0000","last_became_active":"2023-11-28T17:41:17.242196+0000","last_became_peered":"2023-11-28T17:41:17.242196+0000","last_unstale":"2023-11-28T17:41:19.633703+0000","last_undegraded":"2023-11-28T17:41:19.633703+0000","last_fullsized":"2023-11-28T17:41:19.633703+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:06.103318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633386+0000","last_change":"2023-11-28T17:41:17.244694+0000","last_active":"2023-11-28T17:41:19.633386+0000","last_peered":"2023-11-28T17:41:19.633386+0000","last_clean":"2023-11-28T17:41:19.633386+0000","last_became_active":"2023-11-28T17:41:17.244301+0000","last_became_peered":"2023-11-28T17:41:17.244301+0000","last_unstale":"2023-11-28T17:41:19.633386+0000","last_undegraded":"2023-11-28T17:41:19.633386+0000","last_fullsized":"2023-11-28T17:41:19.633386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:39:40.030685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208504+0000","last_change":"2023-11-28T17:41:17.242800+0000","last_active":"2023-11-28T17:41:18.208504+0000","last_peered":"2023-11-28T17:41:18.208504+0000","last_clean":"2023-11-28T17:41:18.208504+0000","last_became_active":"2023-11-28T17:41:17.242463+0000","last_became_peered":"2023-11-28T17:41:17.242463+0000","last_unstale":"2023-11-28T17:41:18.208504+0000","last_undegraded":"2023-11-28T17:41:18.208504+0000","last_fullsized":"2023-11-28T17:41:18.208504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:57:34.802348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568335+0000","last_change":"2023-11-28T17:41:17.232485+0000","last_active":"2023-11-28T17:41:18.568335+0000","last_peered":"2023-11-28T17:41:18.568335+0000","last_clean":"2023-11-28T17:41:18.568335+0000","last_became_active":"2023-11-28T17:41:17.232132+0000","last_became_peered":"2023-11-28T17:41:17.232132+0000","last_unstale":"2023-11-28T17:41:18.568335+0000","last_undegraded":"2023-11-28T17:41:18.568335+0000","last_fullsized":"2023-11-28T17:41:18.568335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:21:49.292565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649479+0000","last_change":"2023-11-28T17:41:17.234514+0000","last_active":"2023-11-28T17:41:18.649479+0000","last_peered":"2023-11-28T17:41:18.649479+0000","last_clean":"2023-11-28T17:41:18.649479+0000","last_became_active":"2023-11-28T17:41:17.234154+0000","last_became_peered":"2023-11-28T17:41:17.234154+0000","last_unstale":"2023-11-28T17:41:18.649479+0000","last_undegraded":"2023-11-28T17:41:18.649479+0000","last_fullsized":"2023-11-28T17:41:18.649479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:22.430159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632129+0000","last_change":"2023-11-28T17:41:17.225905+0000","last_active":"2023-11-28T17:41:19.632129+0000","last_peered":"2023-11-28T17:41:19.632129+0000","last_clean":"2023-11-28T17:41:19.632129+0000","last_became_active":"2023-11-28T17:41:17.223533+0000","last_became_peered":"2023-11-28T17:41:17.223533+0000","last_unstale":"2023-11-28T17:41:19.632129+0000","last_undegraded":"2023-11-28T17:41:19.632129+0000","last_fullsized":"2023-11-28T17:41:19.632129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:32.314586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.205612+0000","last_change":"2023-11-28T17:41:17.236242+0000","last_active":"2023-11-28T17:41:18.205612+0000","last_peered":"2023-11-28T17:41:18.205612+0000","last_clean":"2023-11-28T17:41:18.205612+0000","last_became_active":"2023-11-28T17:41:17.235890+0000","last_became_peered":"2023-11-28T17:41:17.235890+0000","last_unstale":"2023-11-28T17:41:18.205612+0000","last_undegraded":"2023-11-28T17:41:18.205612+0000","last_fullsized":"2023-11-28T17:41:18.205612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:20:09.004702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208885+0000","last_change":"2023-11-28T17:41:17.234610+0000","last_active":"2023-11-28T17:41:18.208885+0000","last_peered":"2023-11-28T17:41:18.208885+0000","last_clean":"2023-11-28T17:41:18.208885+0000","last_became_active":"2023-11-28T17:41:17.234255+0000","last_became_peered":"2023-11-28T17:41:17.234255+0000","last_unstale":"2023-11-28T17:41:18.208885+0000","last_undegraded":"2023-11-28T17:41:18.208885+0000","last_fullsized":"2023-11-28T17:41:18.208885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:25:51.137148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.202895+0000","last_change":"2023-11-28T17:41:17.231838+0000","last_active":"2023-11-28T17:41:18.202895+0000","last_peered":"2023-11-28T17:41:18.202895+0000","last_clean":"2023-11-28T17:41:18.202895+0000","last_became_active":"2023-11-28T17:41:17.231533+0000","last_became_peered":"2023-11-28T17:41:17.231533+0000","last_unstale":"2023-11-28T17:41:18.202895+0000","last_undegraded":"2023-11-28T17:41:18.202895+0000","last_fullsized":"2023-11-28T17:41:18.202895+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:00:56.554112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339038+0000","last_change":"2023-11-28T17:41:17.238888+0000","last_active":"2023-11-28T17:41:18.339038+0000","last_peered":"2023-11-28T17:41:18.339038+0000","last_clean":"2023-11-28T17:41:18.339038+0000","last_became_active":"2023-11-28T17:41:17.236140+0000","last_became_peered":"2023-11-28T17:41:17.236140+0000","last_unstale":"2023-11-28T17:41:18.339038+0000","last_undegraded":"2023-11-28T17:41:18.339038+0000","last_fullsized":"2023-11-28T17:41:18.339038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:20:57.936710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633201+0000","last_change":"2023-11-28T17:41:17.242150+0000","last_active":"2023-11-28T17:41:19.633201+0000","last_peered":"2023-11-28T17:41:19.633201+0000","last_clean":"2023-11-28T17:41:19.633201+0000","last_became_active":"2023-11-28T17:41:17.241832+0000","last_became_peered":"2023-11-28T17:41:17.241832+0000","last_unstale":"2023-11-28T17:41:19.633201+0000","last_undegraded":"2023-11-28T17:41:19.633201+0000","last_fullsized":"2023-11-28T17:41:19.633201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:30:37.837711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649928+0000","last_change":"2023-11-28T17:41:17.233651+0000","last_active":"2023-11-28T17:41:18.649928+0000","last_peered":"2023-11-28T17:41:18.649928+0000","last_clean":"2023-11-28T17:41:18.649928+0000","last_became_active":"2023-11-28T17:41:17.233393+0000","last_became_peered":"2023-11-28T17:41:17.233393+0000","last_unstale":"2023-11-28T17:41:18.649928+0000","last_undegraded":"2023-11-28T17:41:18.649928+0000","last_fullsized":"2023-11-28T17:41:18.649928+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:52:08.152046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206688+0000","last_change":"2023-11-28T17:41:17.239696+0000","last_active":"2023-11-28T17:41:18.206688+0000","last_peered":"2023-11-28T17:41:18.206688+0000","last_clean":"2023-11-28T17:41:18.206688+0000","last_became_active":"2023-11-28T17:41:17.239449+0000","last_became_peered":"2023-11-28T17:41:17.239449+0000","last_unstale":"2023-11-28T17:41:18.206688+0000","last_undegraded":"2023-11-28T17:41:18.206688+0000","last_fullsized":"2023-11-28T17:41:18.206688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:08:43.019689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567770+0000","last_change":"2023-11-28T17:41:17.244917+0000","last_active":"2023-11-28T17:41:18.567770+0000","last_peered":"2023-11-28T17:41:18.567770+0000","last_clean":"2023-11-28T17:41:18.567770+0000","last_became_active":"2023-11-28T17:41:17.244676+0000","last_became_peered":"2023-11-28T17:41:17.244676+0000","last_unstale":"2023-11-28T17:41:18.567770+0000","last_undegraded":"2023-11-28T17:41:18.567770+0000","last_fullsized":"2023-11-28T17:41:18.567770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:08:10.818556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206063+0000","last_change":"2023-11-28T17:41:17.236488+0000","last_active":"2023-11-28T17:41:18.206063+0000","last_peered":"2023-11-28T17:41:18.206063+0000","last_clean":"2023-11-28T17:41:18.206063+0000","last_became_active":"2023-11-28T17:41:17.235554+0000","last_became_peered":"2023-11-28T17:41:17.235554+0000","last_unstale":"2023-11-28T17:41:18.206063+0000","last_undegraded":"2023-11-28T17:41:18.206063+0000","last_fullsized":"2023-11-28T17:41:18.206063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:29:35.500772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568927+0000","last_change":"2023-11-28T17:41:17.245042+0000","last_active":"2023-11-28T17:41:18.568927+0000","last_peered":"2023-11-28T17:41:18.568927+0000","last_clean":"2023-11-28T17:41:18.568927+0000","last_became_active":"2023-11-28T17:41:17.244729+0000","last_became_peered":"2023-11-28T17:41:17.244729+0000","last_unstale":"2023-11-28T17:41:18.568927+0000","last_undegraded":"2023-11-28T17:41:18.568927+0000","last_fullsized":"2023-11-28T17:41:18.568927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:57:00.728653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633914+0000","last_change":"2023-11-28T17:41:17.249367+0000","last_active":"2023-11-28T17:41:19.633914+0000","last_peered":"2023-11-28T17:41:19.633914+0000","last_clean":"2023-11-28T17:41:19.633914+0000","last_became_active":"2023-11-28T17:41:17.249093+0000","last_became_peered":"2023-11-28T17:41:17.249093+0000","last_unstale":"2023-11-28T17:41:19.633914+0000","last_undegraded":"2023-11-28T17:41:19.633914+0000","last_fullsized":"2023-11-28T17:41:19.633914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:12:41.994354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632733+0000","last_change":"2023-11-28T17:41:17.230256+0000","last_active":"2023-11-28T17:41:19.632733+0000","last_peered":"2023-11-28T17:41:19.632733+0000","last_clean":"2023-11-28T17:41:19.632733+0000","last_became_active":"2023-11-28T17:41:17.229551+0000","last_became_peered":"2023-11-28T17:41:17.229551+0000","last_unstale":"2023-11-28T17:41:19.632733+0000","last_undegraded":"2023-11-28T17:41:19.632733+0000","last_fullsized":"2023-11-28T17:41:19.632733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:54:09.421121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568751+0000","last_change":"2023-11-28T17:41:17.241672+0000","last_active":"2023-11-28T17:41:18.568751+0000","last_peered":"2023-11-28T17:41:18.568751+0000","last_clean":"2023-11-28T17:41:18.568751+0000","last_became_active":"2023-11-28T17:41:17.241366+0000","last_became_peered":"2023-11-28T17:41:17.241366+0000","last_unstale":"2023-11-28T17:41:18.568751+0000","last_undegraded":"2023-11-28T17:41:18.568751+0000","last_fullsized":"2023-11-28T17:41:18.568751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:29:01.718362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650047+0000","last_change":"2023-11-28T17:41:17.229925+0000","last_active":"2023-11-28T17:41:18.650047+0000","last_peered":"2023-11-28T17:41:18.650047+0000","last_clean":"2023-11-28T17:41:18.650047+0000","last_became_active":"2023-11-28T17:41:17.229617+0000","last_became_peered":"2023-11-28T17:41:17.229617+0000","last_unstale":"2023-11-28T17:41:18.650047+0000","last_undegraded":"2023-11-28T17:41:18.650047+0000","last_fullsized":"2023-11-28T17:41:18.650047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:40:14.646558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206950+0000","last_change":"2023-11-28T17:41:17.241933+0000","last_active":"2023-11-28T17:41:18.206950+0000","last_peered":"2023-11-28T17:41:18.206950+0000","last_clean":"2023-11-28T17:41:18.206950+0000","last_became_active":"2023-11-28T17:41:17.241634+0000","last_became_peered":"2023-11-28T17:41:17.241634+0000","last_unstale":"2023-11-28T17:41:18.206950+0000","last_undegraded":"2023-11-28T17:41:18.206950+0000","last_fullsized":"2023-11-28T17:41:18.206950+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:44:01.525920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.202444+0000","last_change":"2023-11-28T17:41:17.233872+0000","last_active":"2023-11-28T17:41:18.202444+0000","last_peered":"2023-11-28T17:41:18.202444+0000","last_clean":"2023-11-28T17:41:18.202444+0000","last_became_active":"2023-11-28T17:41:17.233586+0000","last_became_peered":"2023-11-28T17:41:17.233586+0000","last_unstale":"2023-11-28T17:41:18.202444+0000","last_undegraded":"2023-11-28T17:41:18.202444+0000","last_fullsized":"2023-11-28T17:41:18.202444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:56.041470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633270+0000","last_change":"2023-11-28T17:41:17.236164+0000","last_active":"2023-11-28T17:41:19.633270+0000","last_peered":"2023-11-28T17:41:19.633270+0000","last_clean":"2023-11-28T17:41:19.633270+0000","last_became_active":"2023-11-28T17:41:17.235821+0000","last_became_peered":"2023-11-28T17:41:17.235821+0000","last_unstale":"2023-11-28T17:41:19.633270+0000","last_undegraded":"2023-11-28T17:41:19.633270+0000","last_fullsized":"2023-11-28T17:41:19.633270+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:54:16.568394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206920+0000","last_change":"2023-11-28T17:41:17.235816+0000","last_active":"2023-11-28T17:41:18.206920+0000","last_peered":"2023-11-28T17:41:18.206920+0000","last_clean":"2023-11-28T17:41:18.206920+0000","last_became_active":"2023-11-28T17:41:17.235506+0000","last_became_peered":"2023-11-28T17:41:17.235506+0000","last_unstale":"2023-11-28T17:41:18.206920+0000","last_undegraded":"2023-11-28T17:41:18.206920+0000","last_fullsized":"2023-11-28T17:41:18.206920+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:18:00.597727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:27.335928+0000","last_change":"2023-11-28T17:41:16.266076+0000","last_active":"2023-11-28T17:41:27.335928+0000","last_peered":"2023-11-28T17:41:27.335928+0000","last_clean":"2023-11-28T17:41:27.335928+0000","last_became_active":"2023-11-28T17:41:16.265837+0000","last_became_peered":"2023-11-28T17:41:16.265837+0000","last_unstale":"2023-11-28T17:41:27.335928+0000","last_undegraded":"2023-11-28T17:41:27.335928+0000","last_fullsized":"2023-11-28T17:41:27.335928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:01:51.477975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1062,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650250+0000","last_change":"2023-11-28T17:41:17.238915+0000","last_active":"2023-11-28T17:41:18.650250+0000","last_peered":"2023-11-28T17:41:18.650250+0000","last_clean":"2023-11-28T17:41:18.650250+0000","last_became_active":"2023-11-28T17:41:17.238581+0000","last_became_peered":"2023-11-28T17:41:17.238581+0000","last_unstale":"2023-11-28T17:41:18.650250+0000","last_undegraded":"2023-11-28T17:41:18.650250+0000","last_fullsized":"2023-11-28T17:41:18.650250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:52:31.640239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.242663+0000","last_change":"2023-11-28T17:41:16.261044+0000","last_active":"2023-11-28T17:41:39.242663+0000","last_peered":"2023-11-28T17:41:39.242663+0000","last_clean":"2023-11-28T17:41:39.242663+0000","last_became_active":"2023-11-28T17:41:16.260752+0000","last_became_peered":"2023-11-28T17:41:16.260752+0000","last_unstale":"2023-11-28T17:41:39.242663+0000","last_undegraded":"2023-11-28T17:41:39.242663+0000","last_fullsized":"2023-11-28T17:41:39.242663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:07:01.443140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:33.236512+0000","last_change":"2023-11-28T17:41:16.261095+0000","last_active":"2023-11-28T17:41:33.236512+0000","last_peered":"2023-11-28T17:41:33.236512+0000","last_clean":"2023-11-28T17:41:33.236512+0000","last_became_active":"2023-11-28T17:41:16.260793+0000","last_became_peered":"2023-11-28T17:41:16.260793+0000","last_unstale":"2023-11-28T17:41:33.236512+0000","last_undegraded":"2023-11-28T17:41:33.236512+0000","last_fullsized":"2023-11-28T17:41:33.236512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:54:16.821314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.567897+0000","last_change":"2023-11-28T17:41:17.236517+0000","last_active":"2023-11-28T17:41:18.567897+0000","last_peered":"2023-11-28T17:41:18.567897+0000","last_clean":"2023-11-28T17:41:18.567897+0000","last_became_active":"2023-11-28T17:41:17.235925+0000","last_became_peered":"2023-11-28T17:41:17.235925+0000","last_unstale":"2023-11-28T17:41:18.567897+0000","last_undegraded":"2023-11-28T17:41:18.567897+0000","last_fullsized":"2023-11-28T17:41:18.567897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:45:09.817844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.242825+0000","last_change":"2023-11-28T17:41:16.258176+0000","last_active":"2023-11-28T17:41:39.242825+0000","last_peered":"2023-11-28T17:41:39.242825+0000","last_clean":"2023-11-28T17:41:39.242825+0000","last_became_active":"2023-11-28T17:41:16.257838+0000","last_became_peered":"2023-11-28T17:41:16.257838+0000","last_unstale":"2023-11-28T17:41:39.242825+0000","last_undegraded":"2023-11-28T17:41:39.242825+0000","last_fullsized":"2023-11-28T17:41:39.242825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:31:01.443661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.631980+0000","last_change":"2023-11-28T17:41:17.227823+0000","last_active":"2023-11-28T17:41:19.631980+0000","last_peered":"2023-11-28T17:41:19.631980+0000","last_clean":"2023-11-28T17:41:19.631980+0000","last_became_active":"2023-11-28T17:41:17.227545+0000","last_became_peered":"2023-11-28T17:41:17.227545+0000","last_unstale":"2023-11-28T17:41:19.631980+0000","last_undegraded":"2023-11-28T17:41:19.631980+0000","last_fullsized":"2023-11-28T17:41:19.631980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:20:33.253155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:55.076692+0000","last_change":"2023-11-28T17:41:16.262372+0000","last_active":"2023-11-28T17:41:55.076692+0000","last_peered":"2023-11-28T17:41:55.076692+0000","last_clean":"2023-11-28T17:41:55.076692+0000","last_became_active":"2023-11-28T17:41:16.262029+0000","last_became_peered":"2023-11-28T17:41:16.262029+0000","last_unstale":"2023-11-28T17:41:55.076692+0000","last_undegraded":"2023-11-28T17:41:55.076692+0000","last_fullsized":"2023-11-28T17:41:55.076692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:46:08.491380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207822+0000","last_change":"2023-11-28T17:41:17.235646+0000","last_active":"2023-11-28T17:41:18.207822+0000","last_peered":"2023-11-28T17:41:18.207822+0000","last_clean":"2023-11-28T17:41:18.207822+0000","last_became_active":"2023-11-28T17:41:17.235330+0000","last_became_peered":"2023-11-28T17:41:17.235330+0000","last_unstale":"2023-11-28T17:41:18.207822+0000","last_undegraded":"2023-11-28T17:41:18.207822+0000","last_fullsized":"2023-11-28T17:41:18.207822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:48:12.717031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.334160+0000","last_change":"2023-11-28T17:41:16.262716+0000","last_active":"2023-11-28T17:41:44.334160+0000","last_peered":"2023-11-28T17:41:44.334160+0000","last_clean":"2023-11-28T17:41:44.334160+0000","last_became_active":"2023-11-28T17:41:16.262347+0000","last_became_peered":"2023-11-28T17:41:16.262347+0000","last_unstale":"2023-11-28T17:41:44.334160+0000","last_undegraded":"2023-11-28T17:41:44.334160+0000","last_fullsized":"2023-11-28T17:41:44.334160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:50.685367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650261+0000","last_change":"2023-11-28T17:41:17.245852+0000","last_active":"2023-11-28T17:41:18.650261+0000","last_peered":"2023-11-28T17:41:18.650261+0000","last_clean":"2023-11-28T17:41:18.650261+0000","last_became_active":"2023-11-28T17:41:17.245596+0000","last_became_peered":"2023-11-28T17:41:17.245596+0000","last_unstale":"2023-11-28T17:41:18.650261+0000","last_undegraded":"2023-11-28T17:41:18.650261+0000","last_fullsized":"2023-11-28T17:41:18.650261+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:19:03.484052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:27.332030+0000","last_change":"2023-11-28T17:41:16.254855+0000","last_active":"2023-11-28T17:41:27.332030+0000","last_peered":"2023-11-28T17:41:27.332030+0000","last_clean":"2023-11-28T17:41:27.332030+0000","last_became_active":"2023-11-28T17:41:16.254557+0000","last_became_peered":"2023-11-28T17:41:16.254557+0000","last_unstale":"2023-11-28T17:41:27.332030+0000","last_undegraded":"2023-11-28T17:41:27.332030+0000","last_fullsized":"2023-11-28T17:41:27.332030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:55:38.565240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208613+0000","last_change":"2023-11-28T17:41:17.249020+0000","last_active":"2023-11-28T17:41:18.208613+0000","last_peered":"2023-11-28T17:41:18.208613+0000","last_clean":"2023-11-28T17:41:18.208613+0000","last_became_active":"2023-11-28T17:41:17.248345+0000","last_became_peered":"2023-11-28T17:41:17.248345+0000","last_unstale":"2023-11-28T17:41:18.208613+0000","last_undegraded":"2023-11-28T17:41:18.208613+0000","last_fullsized":"2023-11-28T17:41:18.208613+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:05:09.688116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'12","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:55.092716+0000","last_change":"2023-11-28T17:41:16.256485+0000","last_active":"2023-11-28T17:41:55.092716+0000","last_peered":"2023-11-28T17:41:55.092716+0000","last_clean":"2023-11-28T17:41:55.092716+0000","last_became_active":"2023-11-28T17:41:16.256165+0000","last_became_peered":"2023-11-28T17:41:16.256165+0000","last_unstale":"2023-11-28T17:41:55.092716+0000","last_undegraded":"2023-11-28T17:41:55.092716+0000","last_fullsized":"2023-11-28T17:41:55.092716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:35:26.805435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2052,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338198+0000","last_change":"2023-11-28T17:41:17.230246+0000","last_active":"2023-11-28T17:41:18.338198+0000","last_peered":"2023-11-28T17:41:18.338198+0000","last_clean":"2023-11-28T17:41:18.338198+0000","last_became_active":"2023-11-28T17:41:17.229981+0000","last_became_peered":"2023-11-28T17:41:17.229981+0000","last_unstale":"2023-11-28T17:41:18.338198+0000","last_undegraded":"2023-11-28T17:41:18.338198+0000","last_fullsized":"2023-11-28T17:41:18.338198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:15:48.047641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.331053+0000","last_change":"2023-11-28T17:41:16.250894+0000","last_active":"2023-11-28T17:41:44.331053+0000","last_peered":"2023-11-28T17:41:44.331053+0000","last_clean":"2023-11-28T17:41:44.331053+0000","last_became_active":"2023-11-28T17:41:16.250547+0000","last_became_peered":"2023-11-28T17:41:16.250547+0000","last_unstale":"2023-11-28T17:41:44.331053+0000","last_undegraded":"2023-11-28T17:41:44.331053+0000","last_fullsized":"2023-11-28T17:41:44.331053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:58:47.412734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649511+0000","last_change":"2023-11-28T17:41:17.237500+0000","last_active":"2023-11-28T17:41:18.649511+0000","last_peered":"2023-11-28T17:41:18.649511+0000","last_clean":"2023-11-28T17:41:18.649511+0000","last_became_active":"2023-11-28T17:41:17.237211+0000","last_became_peered":"2023-11-28T17:41:17.237211+0000","last_unstale":"2023-11-28T17:41:18.649511+0000","last_undegraded":"2023-11-28T17:41:18.649511+0000","last_fullsized":"2023-11-28T17:41:18.649511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:13.675182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.238895+0000","last_change":"2023-11-28T17:41:16.258147+0000","last_active":"2023-11-28T17:41:39.238895+0000","last_peered":"2023-11-28T17:41:39.238895+0000","last_clean":"2023-11-28T17:41:39.238895+0000","last_became_active":"2023-11-28T17:41:16.257540+0000","last_became_peered":"2023-11-28T17:41:16.257540+0000","last_unstale":"2023-11-28T17:41:39.238895+0000","last_undegraded":"2023-11-28T17:41:39.238895+0000","last_fullsized":"2023-11-28T17:41:39.238895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:51:11.394507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338684+0000","last_change":"2023-11-28T17:41:17.247768+0000","last_active":"2023-11-28T17:41:18.338684+0000","last_peered":"2023-11-28T17:41:18.338684+0000","last_clean":"2023-11-28T17:41:18.338684+0000","last_became_active":"2023-11-28T17:41:17.247475+0000","last_became_peered":"2023-11-28T17:41:17.247475+0000","last_unstale":"2023-11-28T17:41:18.338684+0000","last_undegraded":"2023-11-28T17:41:18.338684+0000","last_fullsized":"2023-11-28T17:41:18.338684+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:40:18.555573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.244012+0000","last_change":"2023-11-28T17:41:16.262927+0000","last_active":"2023-11-28T17:41:39.244012+0000","last_peered":"2023-11-28T17:41:39.244012+0000","last_clean":"2023-11-28T17:41:39.244012+0000","last_became_active":"2023-11-28T17:41:16.262556+0000","last_became_peered":"2023-11-28T17:41:16.262556+0000","last_unstale":"2023-11-28T17:41:39.244012+0000","last_undegraded":"2023-11-28T17:41:39.244012+0000","last_fullsized":"2023-11-28T17:41:39.244012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:35:54.672978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1028,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632921+0000","last_change":"2023-11-28T17:41:17.239625+0000","last_active":"2023-11-28T17:41:19.632921+0000","last_peered":"2023-11-28T17:41:19.632921+0000","last_clean":"2023-11-28T17:41:19.632921+0000","last_became_active":"2023-11-28T17:41:17.238215+0000","last_became_peered":"2023-11-28T17:41:17.238215+0000","last_unstale":"2023-11-28T17:41:19.632921+0000","last_undegraded":"2023-11-28T17:41:19.632921+0000","last_fullsized":"2023-11-28T17:41:19.632921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:33:31.879592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'17","reported_seq":29,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.238291+0000","last_change":"2023-11-28T17:41:16.250674+0000","last_active":"2023-11-28T17:41:39.238291+0000","last_peered":"2023-11-28T17:41:39.238291+0000","last_clean":"2023-11-28T17:41:39.238291+0000","last_became_active":"2023-11-28T17:41:16.250312+0000","last_became_peered":"2023-11-28T17:41:16.250312+0000","last_unstale":"2023-11-28T17:41:39.238291+0000","last_undegraded":"2023-11-28T17:41:39.238291+0000","last_fullsized":"2023-11-28T17:41:39.238291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:04:05.342081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338315+0000","last_change":"2023-11-28T17:41:17.247675+0000","last_active":"2023-11-28T17:41:18.338315+0000","last_peered":"2023-11-28T17:41:18.338315+0000","last_clean":"2023-11-28T17:41:18.338315+0000","last_became_active":"2023-11-28T17:41:17.247346+0000","last_became_peered":"2023-11-28T17:41:17.247346+0000","last_unstale":"2023-11-28T17:41:18.338315+0000","last_undegraded":"2023-11-28T17:41:18.338315+0000","last_fullsized":"2023-11-28T17:41:18.338315+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:45:03.326583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.280691+0000","last_change":"2023-11-28T17:41:16.260893+0000","last_active":"2023-11-28T17:41:22.280691+0000","last_peered":"2023-11-28T17:41:22.280691+0000","last_clean":"2023-11-28T17:41:22.280691+0000","last_became_active":"2023-11-28T17:41:16.260637+0000","last_became_peered":"2023-11-28T17:41:16.260637+0000","last_unstale":"2023-11-28T17:41:22.280691+0000","last_undegraded":"2023-11-28T17:41:22.280691+0000","last_fullsized":"2023-11-28T17:41:22.280691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:07:24.032989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338811+0000","last_change":"2023-11-28T17:41:17.249202+0000","last_active":"2023-11-28T17:41:18.338811+0000","last_peered":"2023-11-28T17:41:18.338811+0000","last_clean":"2023-11-28T17:41:18.338811+0000","last_became_active":"2023-11-28T17:41:17.248187+0000","last_became_peered":"2023-11-28T17:41:17.248187+0000","last_unstale":"2023-11-28T17:41:18.338811+0000","last_undegraded":"2023-11-28T17:41:18.338811+0000","last_fullsized":"2023-11-28T17:41:18.338811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:23.973069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.328238+0000","last_change":"2023-11-28T17:41:16.244938+0000","last_active":"2023-11-28T17:41:44.328238+0000","last_peered":"2023-11-28T17:41:44.328238+0000","last_clean":"2023-11-28T17:41:44.328238+0000","last_became_active":"2023-11-28T17:41:16.244586+0000","last_became_peered":"2023-11-28T17:41:16.244586+0000","last_unstale":"2023-11-28T17:41:44.328238+0000","last_undegraded":"2023-11-28T17:41:44.328238+0000","last_fullsized":"2023-11-28T17:41:44.328238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:38.421966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.339409+0000","last_change":"2023-11-28T17:41:17.248488+0000","last_active":"2023-11-28T17:41:18.339409+0000","last_peered":"2023-11-28T17:41:18.339409+0000","last_clean":"2023-11-28T17:41:18.339409+0000","last_became_active":"2023-11-28T17:41:17.248170+0000","last_became_peered":"2023-11-28T17:41:17.248170+0000","last_unstale":"2023-11-28T17:41:18.339409+0000","last_undegraded":"2023-11-28T17:41:18.339409+0000","last_fullsized":"2023-11-28T17:41:18.339409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:41.188370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.242265+0000","last_change":"2023-11-28T17:41:16.246192+0000","last_active":"2023-11-28T17:41:39.242265+0000","last_peered":"2023-11-28T17:41:39.242265+0000","last_clean":"2023-11-28T17:41:39.242265+0000","last_became_active":"2023-11-28T17:41:16.245379+0000","last_became_peered":"2023-11-28T17:41:16.245379+0000","last_unstale":"2023-11-28T17:41:39.242265+0000","last_undegraded":"2023-11-28T17:41:39.242265+0000","last_fullsized":"2023-11-28T17:41:39.242265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:23:08.664344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207475+0000","last_change":"2023-11-28T17:41:17.234273+0000","last_active":"2023-11-28T17:41:18.207475+0000","last_peered":"2023-11-28T17:41:18.207475+0000","last_clean":"2023-11-28T17:41:18.207475+0000","last_became_active":"2023-11-28T17:41:17.233756+0000","last_became_peered":"2023-11-28T17:41:17.233756+0000","last_unstale":"2023-11-28T17:41:18.207475+0000","last_undegraded":"2023-11-28T17:41:18.207475+0000","last_fullsized":"2023-11-28T17:41:18.207475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:18:40.539329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.242692+0000","last_change":"2023-11-28T17:41:16.257952+0000","last_active":"2023-11-28T17:41:39.242692+0000","last_peered":"2023-11-28T17:41:39.242692+0000","last_clean":"2023-11-28T17:41:39.242692+0000","last_became_active":"2023-11-28T17:41:16.257659+0000","last_became_peered":"2023-11-28T17:41:16.257659+0000","last_unstale":"2023-11-28T17:41:39.242692+0000","last_undegraded":"2023-11-28T17:41:39.242692+0000","last_fullsized":"2023-11-28T17:41:39.242692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:28:12.131592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1610,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.338433+0000","last_change":"2023-11-28T17:41:17.239069+0000","last_active":"2023-11-28T17:41:18.338433+0000","last_peered":"2023-11-28T17:41:18.338433+0000","last_clean":"2023-11-28T17:41:18.338433+0000","last_became_active":"2023-11-28T17:41:17.238548+0000","last_became_peered":"2023-11-28T17:41:17.238548+0000","last_unstale":"2023-11-28T17:41:18.338433+0000","last_undegraded":"2023-11-28T17:41:18.338433+0000","last_fullsized":"2023-11-28T17:41:18.338433+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:12:56.686814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-11-28T17:41:17.206759+0000","last_change":"2023-11-28T17:41:12.444311+0000","last_active":"2023-11-28T17:41:17.206759+0000","last_peered":"2023-11-28T17:41:17.206759+0000","last_clean":"2023-11-28T17:41:17.206759+0000","last_became_active":"2023-11-28T17:41:12.442193+0000","last_became_peered":"2023-11-28T17:41:12.442193+0000","last_unstale":"2023-11-28T17:41:17.206759+0000","last_undegraded":"2023-11-28T17:41:17.206759+0000","last_fullsized":"2023-11-28T17:41:17.206759+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:10.865936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:10.865936+0000","last_clean_scrub_stamp":"2023-11-28T17:41:10.865936+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:12:48.912392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.237404+0000","last_change":"2023-11-28T17:41:16.253023+0000","last_active":"2023-11-28T17:41:39.237404+0000","last_peered":"2023-11-28T17:41:39.237404+0000","last_clean":"2023-11-28T17:41:39.237404+0000","last_became_active":"2023-11-28T17:41:16.245730+0000","last_became_peered":"2023-11-28T17:41:16.245730+0000","last_unstale":"2023-11-28T17:41:39.237404+0000","last_undegraded":"2023-11-28T17:41:39.237404+0000","last_fullsized":"2023-11-28T17:41:39.237404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:05.970917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568226+0000","last_change":"2023-11-28T17:41:17.246677+0000","last_active":"2023-11-28T17:41:18.568226+0000","last_peered":"2023-11-28T17:41:18.568226+0000","last_clean":"2023-11-28T17:41:18.568226+0000","last_became_active":"2023-11-28T17:41:17.246373+0000","last_became_peered":"2023-11-28T17:41:17.246373+0000","last_unstale":"2023-11-28T17:41:18.568226+0000","last_undegraded":"2023-11-28T17:41:18.568226+0000","last_fullsized":"2023-11-28T17:41:18.568226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:22:18.623331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:33.235683+0000","last_change":"2023-11-28T17:41:16.272843+0000","last_active":"2023-11-28T17:41:33.235683+0000","last_peered":"2023-11-28T17:41:33.235683+0000","last_clean":"2023-11-28T17:41:33.235683+0000","last_became_active":"2023-11-28T17:41:16.272527+0000","last_became_peered":"2023-11-28T17:41:16.272527+0000","last_unstale":"2023-11-28T17:41:33.235683+0000","last_undegraded":"2023-11-28T17:41:33.235683+0000","last_fullsized":"2023-11-28T17:41:33.235683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:55:52.185953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.649479+0000","last_change":"2023-11-28T17:41:17.240890+0000","last_active":"2023-11-28T17:41:18.649479+0000","last_peered":"2023-11-28T17:41:18.649479+0000","last_clean":"2023-11-28T17:41:18.649479+0000","last_became_active":"2023-11-28T17:41:17.240571+0000","last_became_peered":"2023-11-28T17:41:17.240571+0000","last_unstale":"2023-11-28T17:41:18.649479+0000","last_undegraded":"2023-11-28T17:41:18.649479+0000","last_fullsized":"2023-11-28T17:41:18.649479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:58.086322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.329294+0000","last_change":"2023-11-28T17:41:16.251925+0000","last_active":"2023-11-28T17:41:44.329294+0000","last_peered":"2023-11-28T17:41:44.329294+0000","last_clean":"2023-11-28T17:41:44.329294+0000","last_became_active":"2023-11-28T17:41:16.251482+0000","last_became_peered":"2023-11-28T17:41:16.251482+0000","last_unstale":"2023-11-28T17:41:44.329294+0000","last_undegraded":"2023-11-28T17:41:44.329294+0000","last_fullsized":"2023-11-28T17:41:44.329294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:07:09.590250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632279+0000","last_change":"2023-11-28T17:41:17.223810+0000","last_active":"2023-11-28T17:41:19.632279+0000","last_peered":"2023-11-28T17:41:19.632279+0000","last_clean":"2023-11-28T17:41:19.632279+0000","last_became_active":"2023-11-28T17:41:17.223244+0000","last_became_peered":"2023-11-28T17:41:17.223244+0000","last_unstale":"2023-11-28T17:41:19.632279+0000","last_undegraded":"2023-11-28T17:41:19.632279+0000","last_fullsized":"2023-11-28T17:41:19.632279+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:31:22.132844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.335847+0000","last_change":"2023-11-28T17:41:16.252402+0000","last_active":"2023-11-28T17:41:44.335847+0000","last_peered":"2023-11-28T17:41:44.335847+0000","last_clean":"2023-11-28T17:41:44.335847+0000","last_became_active":"2023-11-28T17:41:16.252061+0000","last_became_peered":"2023-11-28T17:41:16.252061+0000","last_unstale":"2023-11-28T17:41:44.335847+0000","last_undegraded":"2023-11-28T17:41:44.335847+0000","last_fullsized":"2023-11-28T17:41:44.335847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:20:11.890036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207754+0000","last_change":"2023-11-28T17:41:17.234320+0000","last_active":"2023-11-28T17:41:18.207754+0000","last_peered":"2023-11-28T17:41:18.207754+0000","last_clean":"2023-11-28T17:41:18.207754+0000","last_became_active":"2023-11-28T17:41:17.233738+0000","last_became_peered":"2023-11-28T17:41:17.233738+0000","last_unstale":"2023-11-28T17:41:18.207754+0000","last_undegraded":"2023-11-28T17:41:18.207754+0000","last_fullsized":"2023-11-28T17:41:18.207754+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:31:10.257906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632533+0000","last_change":"2023-11-28T17:41:16.245394+0000","last_active":"2023-11-28T17:41:19.632533+0000","last_peered":"2023-11-28T17:41:19.632533+0000","last_clean":"2023-11-28T17:41:19.632533+0000","last_became_active":"2023-11-28T17:41:16.242985+0000","last_became_peered":"2023-11-28T17:41:16.242985+0000","last_unstale":"2023-11-28T17:41:19.632533+0000","last_undegraded":"2023-11-28T17:41:19.632533+0000","last_fullsized":"2023-11-28T17:41:19.632533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:48.728960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207612+0000","last_change":"2023-11-28T17:41:17.233987+0000","last_active":"2023-11-28T17:41:18.207612+0000","last_peered":"2023-11-28T17:41:18.207612+0000","last_clean":"2023-11-28T17:41:18.207612+0000","last_became_active":"2023-11-28T17:41:17.233430+0000","last_became_peered":"2023-11-28T17:41:17.233430+0000","last_unstale":"2023-11-28T17:41:18.207612+0000","last_undegraded":"2023-11-28T17:41:18.207612+0000","last_fullsized":"2023-11-28T17:41:18.207612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:51:44.816033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.238818+0000","last_change":"2023-11-28T17:41:16.273047+0000","last_active":"2023-11-28T17:41:39.238818+0000","last_peered":"2023-11-28T17:41:39.238818+0000","last_clean":"2023-11-28T17:41:39.238818+0000","last_became_active":"2023-11-28T17:41:16.272667+0000","last_became_peered":"2023-11-28T17:41:16.272667+0000","last_unstale":"2023-11-28T17:41:39.238818+0000","last_undegraded":"2023-11-28T17:41:39.238818+0000","last_fullsized":"2023-11-28T17:41:39.238818+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:19:37.472121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.632439+0000","last_change":"2023-11-28T17:41:17.232465+0000","last_active":"2023-11-28T17:41:19.632439+0000","last_peered":"2023-11-28T17:41:19.632439+0000","last_clean":"2023-11-28T17:41:19.632439+0000","last_became_active":"2023-11-28T17:41:17.230496+0000","last_became_peered":"2023-11-28T17:41:17.230496+0000","last_unstale":"2023-11-28T17:41:19.632439+0000","last_undegraded":"2023-11-28T17:41:19.632439+0000","last_fullsized":"2023-11-28T17:41:19.632439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:52:08.590259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'7","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.243168+0000","last_change":"2023-11-28T17:41:16.258717+0000","last_active":"2023-11-28T17:41:39.243168+0000","last_peered":"2023-11-28T17:41:39.243168+0000","last_clean":"2023-11-28T17:41:39.243168+0000","last_became_active":"2023-11-28T17:41:16.258388+0000","last_became_peered":"2023-11-28T17:41:16.258388+0000","last_unstale":"2023-11-28T17:41:39.243168+0000","last_undegraded":"2023-11-28T17:41:39.243168+0000","last_fullsized":"2023-11-28T17:41:39.243168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:17.845137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568998+0000","last_change":"2023-11-28T17:41:17.239189+0000","last_active":"2023-11-28T17:41:18.568998+0000","last_peered":"2023-11-28T17:41:18.568998+0000","last_clean":"2023-11-28T17:41:18.568998+0000","last_became_active":"2023-11-28T17:41:17.238829+0000","last_became_peered":"2023-11-28T17:41:17.238829+0000","last_unstale":"2023-11-28T17:41:18.568998+0000","last_undegraded":"2023-11-28T17:41:18.568998+0000","last_fullsized":"2023-11-28T17:41:18.568998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:04:10.923572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.337125+0000","last_change":"2023-11-28T17:41:16.268334+0000","last_active":"2023-11-28T17:41:44.337125+0000","last_peered":"2023-11-28T17:41:44.337125+0000","last_clean":"2023-11-28T17:41:44.337125+0000","last_became_active":"2023-11-28T17:41:16.268015+0000","last_became_peered":"2023-11-28T17:41:16.268015+0000","last_unstale":"2023-11-28T17:41:44.337125+0000","last_undegraded":"2023-11-28T17:41:44.337125+0000","last_fullsized":"2023-11-28T17:41:44.337125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:57:22.031553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650340+0000","last_change":"2023-11-28T17:41:17.225161+0000","last_active":"2023-11-28T17:41:18.650340+0000","last_peered":"2023-11-28T17:41:18.650340+0000","last_clean":"2023-11-28T17:41:18.650340+0000","last_became_active":"2023-11-28T17:41:17.224420+0000","last_became_peered":"2023-11-28T17:41:17.224420+0000","last_unstale":"2023-11-28T17:41:18.650340+0000","last_undegraded":"2023-11-28T17:41:18.650340+0000","last_fullsized":"2023-11-28T17:41:18.650340+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:25:11.426980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:27.331262+0000","last_change":"2023-11-28T17:41:16.272884+0000","last_active":"2023-11-28T17:41:27.331262+0000","last_peered":"2023-11-28T17:41:27.331262+0000","last_clean":"2023-11-28T17:41:27.331262+0000","last_became_active":"2023-11-28T17:41:16.272557+0000","last_became_peered":"2023-11-28T17:41:16.272557+0000","last_unstale":"2023-11-28T17:41:27.331262+0000","last_undegraded":"2023-11-28T17:41:27.331262+0000","last_fullsized":"2023-11-28T17:41:27.331262+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:56.634559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.206149+0000","last_change":"2023-11-28T17:41:17.238145+0000","last_active":"2023-11-28T17:41:18.206149+0000","last_peered":"2023-11-28T17:41:18.206149+0000","last_clean":"2023-11-28T17:41:18.206149+0000","last_became_active":"2023-11-28T17:41:17.237774+0000","last_became_peered":"2023-11-28T17:41:17.237774+0000","last_unstale":"2023-11-28T17:41:18.206149+0000","last_undegraded":"2023-11-28T17:41:18.206149+0000","last_fullsized":"2023-11-28T17:41:18.206149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:44.723180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.333438+0000","last_change":"2023-11-28T17:41:16.262318+0000","last_active":"2023-11-28T17:41:44.333438+0000","last_peered":"2023-11-28T17:41:44.333438+0000","last_clean":"2023-11-28T17:41:44.333438+0000","last_became_active":"2023-11-28T17:41:16.262014+0000","last_became_peered":"2023-11-28T17:41:16.262014+0000","last_unstale":"2023-11-28T17:41:44.333438+0000","last_undegraded":"2023-11-28T17:41:44.333438+0000","last_fullsized":"2023-11-28T17:41:44.333438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:17:15.470393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568031+0000","last_change":"2023-11-28T17:41:17.229233+0000","last_active":"2023-11-28T17:41:18.568031+0000","last_peered":"2023-11-28T17:41:18.568031+0000","last_clean":"2023-11-28T17:41:18.568031+0000","last_became_active":"2023-11-28T17:41:17.228885+0000","last_became_peered":"2023-11-28T17:41:17.228885+0000","last_unstale":"2023-11-28T17:41:18.568031+0000","last_undegraded":"2023-11-28T17:41:18.568031+0000","last_fullsized":"2023-11-28T17:41:18.568031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:10:24.247307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:39.236303+0000","last_change":"2023-11-28T17:41:16.246533+0000","last_active":"2023-11-28T17:41:39.236303+0000","last_peered":"2023-11-28T17:41:39.236303+0000","last_clean":"2023-11-28T17:41:39.236303+0000","last_became_active":"2023-11-28T17:41:16.244067+0000","last_became_peered":"2023-11-28T17:41:16.244067+0000","last_unstale":"2023-11-28T17:41:39.236303+0000","last_undegraded":"2023-11-28T17:41:39.236303+0000","last_fullsized":"2023-11-28T17:41:39.236303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:02:20.862297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568303+0000","last_change":"2023-11-28T17:41:17.236159+0000","last_active":"2023-11-28T17:41:18.568303+0000","last_peered":"2023-11-28T17:41:18.568303+0000","last_clean":"2023-11-28T17:41:18.568303+0000","last_became_active":"2023-11-28T17:41:17.235700+0000","last_became_peered":"2023-11-28T17:41:17.235700+0000","last_unstale":"2023-11-28T17:41:18.568303+0000","last_undegraded":"2023-11-28T17:41:18.568303+0000","last_fullsized":"2023-11-28T17:41:18.568303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:21.900805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:27.330503+0000","last_change":"2023-11-28T17:41:16.268969+0000","last_active":"2023-11-28T17:41:27.330503+0000","last_peered":"2023-11-28T17:41:27.330503+0000","last_clean":"2023-11-28T17:41:27.330503+0000","last_became_active":"2023-11-28T17:41:16.268717+0000","last_became_peered":"2023-11-28T17:41:16.268717+0000","last_unstale":"2023-11-28T17:41:27.330503+0000","last_undegraded":"2023-11-28T17:41:27.330503+0000","last_fullsized":"2023-11-28T17:41:27.330503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:03:29.120624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:19.633192+0000","last_change":"2023-11-28T17:41:17.244376+0000","last_active":"2023-11-28T17:41:19.633192+0000","last_peered":"2023-11-28T17:41:19.633192+0000","last_clean":"2023-11-28T17:41:19.633192+0000","last_became_active":"2023-11-28T17:41:17.244034+0000","last_became_peered":"2023-11-28T17:41:17.244034+0000","last_unstale":"2023-11-28T17:41:19.633192+0000","last_undegraded":"2023-11-28T17:41:19.633192+0000","last_fullsized":"2023-11-28T17:41:19.633192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:15:56.198732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.208198+0000","last_change":"2023-11-28T17:41:17.233749+0000","last_active":"2023-11-28T17:41:18.208198+0000","last_peered":"2023-11-28T17:41:18.208198+0000","last_clean":"2023-11-28T17:41:18.208198+0000","last_became_active":"2023-11-28T17:41:17.232992+0000","last_became_peered":"2023-11-28T17:41:17.232992+0000","last_unstale":"2023-11-28T17:41:18.208198+0000","last_undegraded":"2023-11-28T17:41:18.208198+0000","last_fullsized":"2023-11-28T17:41:18.208198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:00:43.698244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.338678+0000","last_change":"2023-11-28T17:41:16.269465+0000","last_active":"2023-11-28T17:41:44.338678+0000","last_peered":"2023-11-28T17:41:44.338678+0000","last_clean":"2023-11-28T17:41:44.338678+0000","last_became_active":"2023-11-28T17:41:16.269147+0000","last_became_peered":"2023-11-28T17:41:16.269147+0000","last_unstale":"2023-11-28T17:41:44.338678+0000","last_undegraded":"2023-11-28T17:41:44.338678+0000","last_fullsized":"2023-11-28T17:41:44.338678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:57:58.121232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.207346+0000","last_change":"2023-11-28T17:41:17.243498+0000","last_active":"2023-11-28T17:41:18.207346+0000","last_peered":"2023-11-28T17:41:18.207346+0000","last_clean":"2023-11-28T17:41:18.207346+0000","last_became_active":"2023-11-28T17:41:17.243169+0000","last_became_peered":"2023-11-28T17:41:17.243169+0000","last_unstale":"2023-11-28T17:41:18.207346+0000","last_undegraded":"2023-11-28T17:41:18.207346+0000","last_fullsized":"2023-11-28T17:41:18.207346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:11:31.509921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'12","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.340143+0000","last_change":"2023-11-28T17:41:16.270231+0000","last_active":"2023-11-28T17:41:44.340143+0000","last_peered":"2023-11-28T17:41:44.340143+0000","last_clean":"2023-11-28T17:41:44.340143+0000","last_became_active":"2023-11-28T17:41:16.269938+0000","last_became_peered":"2023-11-28T17:41:16.269938+0000","last_unstale":"2023-11-28T17:41:44.340143+0000","last_undegraded":"2023-11-28T17:41:44.340143+0000","last_fullsized":"2023-11-28T17:41:44.340143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:48:45.654658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.650357+0000","last_change":"2023-11-28T17:41:17.245856+0000","last_active":"2023-11-28T17:41:18.650357+0000","last_peered":"2023-11-28T17:41:18.650357+0000","last_clean":"2023-11-28T17:41:18.650357+0000","last_became_active":"2023-11-28T17:41:17.245624+0000","last_became_peered":"2023-11-28T17:41:17.245624+0000","last_unstale":"2023-11-28T17:41:18.650357+0000","last_undegraded":"2023-11-28T17:41:18.650357+0000","last_fullsized":"2023-11-28T17:41:18.650357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:03:10.206506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:22.279322+0000","last_change":"2023-11-28T17:41:16.260989+0000","last_active":"2023-11-28T17:41:22.279322+0000","last_peered":"2023-11-28T17:41:22.279322+0000","last_clean":"2023-11-28T17:41:22.279322+0000","last_became_active":"2023-11-28T17:41:16.260686+0000","last_became_peered":"2023-11-28T17:41:16.260686+0000","last_unstale":"2023-11-28T17:41:22.279322+0000","last_undegraded":"2023-11-28T17:41:22.279322+0000","last_fullsized":"2023-11-28T17:41:22.279322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:58:44.193791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:18.568672+0000","last_change":"2023-11-28T17:41:17.240803+0000","last_active":"2023-11-28T17:41:18.568672+0000","last_peered":"2023-11-28T17:41:18.568672+0000","last_clean":"2023-11-28T17:41:18.568672+0000","last_became_active":"2023-11-28T17:41:17.240442+0000","last_became_peered":"2023-11-28T17:41:17.240442+0000","last_unstale":"2023-11-28T17:41:18.568672+0000","last_undegraded":"2023-11-28T17:41:18.568672+0000","last_fullsized":"2023-11-28T17:41:18.568672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:16.214502+0000","last_clean_scrub_stamp":"2023-11-28T17:41:16.214502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:28:21.223788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-11-28T17:41:44.335954+0000","last_change":"2023-11-28T17:41:16.266520+0000","last_active":"2023-11-28T17:41:44.335954+0000","last_peered":"2023-11-28T17:41:44.335954+0000","last_clean":"2023-11-28T17:41:44.335954+0000","last_became_active":"2023-11-28T17:41:16.266245+0000","last_became_peered":"2023-11-28T17:41:16.266245+0000","last_unstale":"2023-11-28T17:41:44.335954+0000","last_undegraded":"2023-11-28T17:41:44.335954+0000","last_fullsized":"2023-11-28T17:41:44.335954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:41:15.204463+0000","last_clean_scrub_stamp":"2023-11-28T17:41:15.204463+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:18:28.732421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":9794,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":213,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":19588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":187,"ondisk_log_size":187,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65062,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509453,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":669440,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.49}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.595}]}]},{"osd":6,"up_from":15,"seq":64424509451,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":79814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3680000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.593}]}]},{"osd":5,"up_from":15,"seq":64424509452,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":80904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.214}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.694}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.643}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509452,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":245760,"data_stored":77862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3660000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.405}]}]},{"osd":3,"up_from":15,"seq":64424509451,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":78968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509452,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":78064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7390000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3170000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.49}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2229999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509453,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":315392,"data_stored":667774,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.175}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2090000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":78434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32531,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T17:41:59.713 INFO:tasks.mon_thrash.ceph_manager:clean! 2023-11-28T17:41:59.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:41:59.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.962+0000 7fc4e4cd1640 1 Processor -- start 2023-11-28T17:41:59.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.962+0000 7fc4e4cd1640 1 -- start start 2023-11-28T17:41:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.963+0000 7fc4e4cd1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc4e01518d0 0x7fc4e0158240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.963+0000 7fc4e4cd1640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc4e00728a0 0x7fc4e0070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.963+0000 7fc4e4cd1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e015bc80 con 0x7fc4e006d950 2023-11-28T17:41:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.963+0000 7fc4e4cd1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e006ade0 con 0x7fc4e01518d0 2023-11-28T17:41:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.963+0000 7fc4e4cd1640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e0151cb0 con 0x7fc4e00728a0 2023-11-28T17:41:59.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.964+0000 7fc4df7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc4e01518d0 0x7fc4e0158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:59.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.964+0000 7fc4dffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc4e006d950 0x7fc4e006dda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46924/0 (socket says 172.21.15.98:46924) 2023-11-28T17:41:59.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.964+0000 7fc4dffff640 1 -- 172.21.15.98:0/3591896040 learned_addr learned my addr 172.21.15.98:0/3591896040 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:41:59.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.964+0000 7fc4deffd640 1 --2- 172.21.15.98:0/3591896040 >> v2:172.21.15.113:3300/0 conn(0x7fc4e00728a0 0x7fc4e0070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:59.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.964+0000 7fc4df7fe640 1 -- 172.21.15.98:0/3591896040 >> v2:172.21.15.113:3300/0 conn(0x7fc4e00728a0 msgr2=0x7fc4e0070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:41:59.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.964+0000 7fc4df7fe640 1 --2- 172.21.15.98:0/3591896040 >> v2:172.21.15.113:3300/0 conn(0x7fc4e00728a0 0x7fc4e0070bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.964+0000 7fc4df7fe640 1 -- 172.21.15.98:0/3591896040 >> v1:172.21.15.98:6789/0 conn(0x7fc4e006d950 legacy=0x7fc4e006dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:59.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.964+0000 7fc4df7fe640 1 -- 172.21.15.98:0/3591896040 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e006b030 con 0x7fc4e01518d0 2023-11-28T17:41:59.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.964+0000 7fc4de7fc640 1 -- 172.21.15.98:0/3591896040 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680971784 0 0) 0x7fc4e015bc80 con 0x7fc4e006d950 2023-11-28T17:41:59.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.965+0000 7fc4df7fe640 1 --2- 172.21.15.98:0/3591896040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e01518d0 0x7fc4e0158240 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc4e006eb20 tx=0x7fc4d002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dcb0c99842597ad server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:59.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.966+0000 7fc4de7fc640 1 -- 172.21.15.98:0/3591896040 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d002e070 con 0x7fc4e01518d0 2023-11-28T17:41:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.966+0000 7fc4de7fc640 1 -- 172.21.15.98:0/3591896040 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d00315a0 con 0x7fc4e01518d0 2023-11-28T17:41:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.966+0000 7fc4de7fc640 1 -- 172.21.15.98:0/3591896040 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0031930 con 0x7fc4e01518d0 2023-11-28T17:41:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.966+0000 7fc4e4cd1640 1 -- 172.21.15.98:0/3591896040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e01518d0 msgr2=0x7fc4e0158240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.966+0000 7fc4e4cd1640 1 --2- 172.21.15.98:0/3591896040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e01518d0 0x7fc4e0158240 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc4e006eb20 tx=0x7fc4d002b840 comp rx=0 tx=0).stop 2023-11-28T17:41:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.966+0000 7fc4e4cd1640 1 -- 172.21.15.98:0/3591896040 shutdown_connections 2023-11-28T17:41:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.966+0000 7fc4e4cd1640 1 --2- 172.21.15.98:0/3591896040 >> v2:172.21.15.113:3300/0 conn(0x7fc4e00728a0 0x7fc4e0070bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.966+0000 7fc4e4cd1640 1 --2- 172.21.15.98:0/3591896040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e01518d0 0x7fc4e0158240 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.966+0000 7fc4e4cd1640 1 -- 172.21.15.98:0/3591896040 >> 172.21.15.98:0/3591896040 conn(0x7fc4e0066230 msgr2=0x7fc4e006a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:41:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.967+0000 7fc4e4cd1640 1 -- 172.21.15.98:0/3591896040 shutdown_connections 2023-11-28T17:41:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.967+0000 7fc4e4cd1640 1 -- 172.21.15.98:0/3591896040 wait complete. 2023-11-28T17:41:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.968+0000 7fc4e4cd1640 1 Processor -- start 2023-11-28T17:41:59.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.968+0000 7fc4e4cd1640 1 -- start start 2023-11-28T17:41:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.969+0000 7fc4e4cd1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e006d950 0x7fc4e01ef7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.969+0000 7fc4e4cd1640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4e00728a0 0x7fc4e00c1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.969+0000 7fc4e4cd1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4e01518d0 0x7fc4e00adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.969+0000 7fc4e4cd1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e00c1670 con 0x7fc4e006d950 2023-11-28T17:41:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.969+0000 7fc4e4cd1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e01ef340 con 0x7fc4e01518d0 2023-11-28T17:41:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.969+0000 7fc4e4cd1640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e00c5a50 con 0x7fc4e00728a0 2023-11-28T17:41:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.970+0000 7fc4df7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e006d950 0x7fc4e01ef7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.970+0000 7fc4df7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e006d950 0x7fc4e01ef7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57128/0 (socket says 172.21.15.98:57128) 2023-11-28T17:41:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.970+0000 7fc4df7fe640 1 -- 172.21.15.98:0/2815355169 learned_addr learned my addr 172.21.15.98:0/2815355169 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:41:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.970+0000 7fc4deffd640 1 --2- 172.21.15.98:0/2815355169 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4e00728a0 0x7fc4e00c1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.970+0000 7fc4dffff640 1 --2- 172.21.15.98:0/2815355169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4e01518d0 0x7fc4e00adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.970+0000 7fc4df7fe640 1 -- 172.21.15.98:0/2815355169 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4e00728a0 msgr2=0x7fc4e00c1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.970+0000 7fc4df7fe640 1 --2- 172.21.15.98:0/2815355169 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4e00728a0 0x7fc4e00c1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.970+0000 7fc4df7fe640 1 -- 172.21.15.98:0/2815355169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4e01518d0 msgr2=0x7fc4e00adbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:41:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.970+0000 7fc4df7fe640 1 --2- 172.21.15.98:0/2815355169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4e01518d0 0x7fc4e00adbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:41:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.970+0000 7fc4df7fe640 1 -- 172.21.15.98:0/2815355169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d0009400 con 0x7fc4e006d950 2023-11-28T17:41:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.970+0000 7fc4dffff640 1 --2- 172.21.15.98:0/2815355169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4e01518d0 0x7fc4e00adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:41:59.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.970+0000 7fc4df7fe640 1 --2- 172.21.15.98:0/2815355169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e006d950 0x7fc4e01ef7d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009890 tx=0x7fc4d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:59.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.971+0000 7fc4c7fff640 1 -- 172.21.15.98:0/2815355169 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0032840 con 0x7fc4e006d950 2023-11-28T17:41:59.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.971+0000 7fc4c7fff640 1 -- 172.21.15.98:0/2815355169 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d00329a0 con 0x7fc4e006d950 2023-11-28T17:41:59.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.971+0000 7fc4c7fff640 1 -- 172.21.15.98:0/2815355169 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0032d50 con 0x7fc4e006d950 2023-11-28T17:41:59.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.971+0000 7fc4e4cd1640 1 -- 172.21.15.98:0/2815355169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e00c1890 con 0x7fc4e006d950 2023-11-28T17:41:59.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.971+0000 7fc4e4cd1640 1 -- 172.21.15.98:0/2815355169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e00c1e50 con 0x7fc4e006d950 2023-11-28T17:41:59.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.972+0000 7fc4e4cd1640 1 -- 172.21.15.98:0/2815355169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e0074db0 con 0x7fc4e006d950 2023-11-28T17:41:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.973+0000 7fc4c7fff640 1 -- 172.21.15.98:0/2815355169 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fc4d00453f0 con 0x7fc4e006d950 2023-11-28T17:41:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.974+0000 7fc4c7fff640 1 --2- 172.21.15.98:0/2815355169 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc4cc06b2a0 0x7fc4cc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:41:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.974+0000 7fc4c7fff640 1 -- 172.21.15.98:0/2815355169 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4d00abc00 con 0x7fc4e006d950 2023-11-28T17:41:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.974+0000 7fc4deffd640 1 --2- 172.21.15.98:0/2815355169 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc4cc06b2a0 0x7fc4cc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:41:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.975+0000 7fc4deffd640 1 --2- 172.21.15.98:0/2815355169 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc4cc06b2a0 0x7fc4cc06d760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4006fd0 tx=0x7fc4d4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:41:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:41:59.978+0000 7fc4c7fff640 1 -- 172.21.15.98:0/2815355169 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4d00b2080 con 0x7fc4e006d950 2023-11-28T17:42:00.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.169+0000 7fc4e4cd1640 1 -- 172.21.15.98:0/2815355169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4e00c4330 con 0x7fc4e006d950 2023-11-28T17:42:00.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.171+0000 7fc4c7fff640 1 -- 172.21.15.98:0/2815355169 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16517 (secure 0 0 0) 0x7fc4d0076b00 con 0x7fc4e006d950 2023-11-28T17:42:00.172 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:42:00.172 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:41:18.195933+0000","last_up_change":"2023-11-28T17:41:09.556993+0000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:41:10.303280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:41:14.890871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:41:15.704103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6810","nonce":2188704385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6813","nonce":2188704385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6819","nonce":2188704385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6816","nonce":2188704385}]},"public_addr":"172.21.15.98:6810/2188704385","cluster_addr":"172.21.15.98:6813/2188704385","heartbeat_back_addr":"172.21.15.98:6819/2188704385","heartbeat_front_addr":"172.21.15.98:6816/2188704385","state":["exists","up"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6801","nonce":3918226860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6803","nonce":3918226860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6807","nonce":3918226860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6805","nonce":3918226860}]},"public_addr":"172.21.15.98:6801/3918226860","cluster_addr":"172.21.15.98:6803/3918226860","heartbeat_back_addr":"172.21.15.98:6807/3918226860","heartbeat_front_addr":"172.21.15.98:6805/3918226860","state":["exists","up"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6812","nonce":2691532400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6815","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6818","nonce":2691532400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6823","nonce":2691532400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6821","nonce":2691532400}]},"public_addr":"172.21.15.98:6812/2691532400","cluster_addr":"172.21.15.98:6818/2691532400","heartbeat_back_addr":"172.21.15.98:6823/2691532400","heartbeat_front_addr":"172.21.15.98:6821/2691532400","state":["exists","up"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6825","nonce":1905487008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6827","nonce":1905487008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6831","nonce":1905487008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6829","nonce":1905487008}]},"public_addr":"172.21.15.98:6825/1905487008","cluster_addr":"172.21.15.98:6827/1905487008","heartbeat_back_addr":"172.21.15.98:6831/1905487008","heartbeat_front_addr":"172.21.15.98:6829/1905487008","state":["exists","up"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6801","nonce":2974677215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6803","nonce":2974677215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6807","nonce":2974677215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6805","nonce":2974677215}]},"public_addr":"172.21.15.113:6801/2974677215","cluster_addr":"172.21.15.113:6803/2974677215","heartbeat_back_addr":"172.21.15.113:6807/2974677215","heartbeat_front_addr":"172.21.15.113:6805/2974677215","state":["exists","up"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6817","nonce":2685000355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6821","nonce":2685000355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6829","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6831","nonce":2685000355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6823","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6826","nonce":2685000355}]},"public_addr":"172.21.15.113:6817/2685000355","cluster_addr":"172.21.15.113:6821/2685000355","heartbeat_back_addr":"172.21.15.113:6831/2685000355","heartbeat_front_addr":"172.21.15.113:6826/2685000355","state":["exists","up"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6809","nonce":2764300831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6811","nonce":2764300831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6815","nonce":2764300831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6813","nonce":2764300831}]},"public_addr":"172.21.15.113:6809/2764300831","cluster_addr":"172.21.15.113:6811/2764300831","heartbeat_back_addr":"172.21.15.113:6815/2764300831","heartbeat_front_addr":"172.21.15.113:6813/2764300831","state":["exists","up"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6819","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6820","nonce":1656248400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6824","nonce":1656248400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6830","nonce":1656248400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6825","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6827","nonce":1656248400}]},"public_addr":"172.21.15.113:6820/1656248400","cluster_addr":"172.21.15.113:6824/1656248400","heartbeat_back_addr":"172.21.15.113:6830/1656248400","heartbeat_front_addr":"172.21.15.113:6827/1656248400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.379549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.312196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.117494+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.388128+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.360608+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.411101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.355995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.390362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:42:00.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.174+0000 7fc4e4cd1640 1 -- 172.21.15.98:0/2815355169 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc4cc06b2a0 msgr2=0x7fc4cc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:00.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.174+0000 7fc4e4cd1640 1 --2- 172.21.15.98:0/2815355169 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc4cc06b2a0 0x7fc4cc06d760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4006fd0 tx=0x7fc4d4008040 comp rx=0 tx=0).stop 2023-11-28T17:42:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.174+0000 7fc4e4cd1640 1 -- 172.21.15.98:0/2815355169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e006d950 msgr2=0x7fc4e01ef7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.174+0000 7fc4e4cd1640 1 --2- 172.21.15.98:0/2815355169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e006d950 0x7fc4e01ef7d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009890 tx=0x7fc4d00046c0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.174+0000 7fc4e4cd1640 1 -- 172.21.15.98:0/2815355169 shutdown_connections 2023-11-28T17:42:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.174+0000 7fc4e4cd1640 1 --2- 172.21.15.98:0/2815355169 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc4cc06b2a0 0x7fc4cc06d760 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.174+0000 7fc4e4cd1640 1 --2- 172.21.15.98:0/2815355169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4e01518d0 0x7fc4e00adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.174+0000 7fc4e4cd1640 1 --2- 172.21.15.98:0/2815355169 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc4e00728a0 0x7fc4e00c1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.174+0000 7fc4e4cd1640 1 --2- 172.21.15.98:0/2815355169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4e006d950 0x7fc4e01ef7d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.174+0000 7fc4e4cd1640 1 -- 172.21.15.98:0/2815355169 >> 172.21.15.98:0/2815355169 conn(0x7fc4e0066230 msgr2=0x7fc4e0159d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.174+0000 7fc4e4cd1640 1 -- 172.21.15.98:0/2815355169 shutdown_connections 2023-11-28T17:42:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.174+0000 7fc4e4cd1640 1 -- 172.21.15.98:0/2815355169 wait complete. 2023-11-28T17:42:00.193 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:41:10.303280+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:41:14.890871+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:41:15.704103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:42:00.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:42:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.412+0000 7fb42d059640 1 Processor -- start 2023-11-28T17:42:00.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.412+0000 7fb42d059640 1 -- start start 2023-11-28T17:42:00.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.414+0000 7fb42d059640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb4280cba90 0x7fb42815a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:00.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.414+0000 7fb42d059640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4280cabf0 0x7fb4280cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:00.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.414+0000 7fb42d059640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4280c9ea0 con 0x7fb4280cc3b0 2023-11-28T17:42:00.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.414+0000 7fb42d059640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4280ca180 con 0x7fb4280cba90 2023-11-28T17:42:00.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.414+0000 7fb42d059640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42806bfb0 con 0x7fb4280cabf0 2023-11-28T17:42:00.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.414+0000 7fb427fff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb4280cba90 0x7fb42815a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:00.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.414+0000 7fb427fff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb4280cba90 0x7fb42815a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:52398/0 (socket says 172.21.15.98:52398) 2023-11-28T17:42:00.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.414+0000 7fb42c858640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fb4280cc3b0 0x7fb428153940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.98:34256/0 (socket says 172.21.15.98:34256) 2023-11-28T17:42:00.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.414+0000 7fb427fff640 1 -- 172.21.15.98:0/860672716 learned_addr learned my addr 172.21.15.98:0/860672716 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:00.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.414+0000 7fb4277fe640 1 --2- 172.21.15.98:0/860672716 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4280cabf0 0x7fb4280cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:00.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.415+0000 7fb427fff640 1 -- 172.21.15.98:0/860672716 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4280cabf0 msgr2=0x7fb4280cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:00.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.415+0000 7fb427fff640 1 --2- 172.21.15.98:0/860672716 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4280cabf0 0x7fb4280cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.415+0000 7fb427fff640 1 -- 172.21.15.98:0/860672716 >> v1:172.21.15.113:6789/0 conn(0x7fb4280cc3b0 legacy=0x7fb428153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:00.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.415+0000 7fb427fff640 1 -- 172.21.15.98:0/860672716 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42806c290 con 0x7fb4280cba90 2023-11-28T17:42:00.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.416+0000 7fb427fff640 1 --2- 172.21.15.98:0/860672716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4280cba90 0x7fb42815a4f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb418009870 tx=0x7fb41802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=248cb924936bdad7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:00.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.417+0000 7fb426ffd640 1 -- 172.21.15.98:0/860672716 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb418031640 con 0x7fb4280cba90 2023-11-28T17:42:00.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.417+0000 7fb426ffd640 1 -- 172.21.15.98:0/860672716 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4180317a0 con 0x7fb4280cba90 2023-11-28T17:42:00.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.417+0000 7fb426ffd640 1 -- 172.21.15.98:0/860672716 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb418031b30 con 0x7fb4280cba90 2023-11-28T17:42:00.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.417+0000 7fb42d059640 1 -- 172.21.15.98:0/860672716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4280cba90 msgr2=0x7fb42815a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:00.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.417+0000 7fb42d059640 1 --2- 172.21.15.98:0/860672716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4280cba90 0x7fb42815a4f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb418009870 tx=0x7fb41802b8f0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.417+0000 7fb42d059640 1 -- 172.21.15.98:0/860672716 shutdown_connections 2023-11-28T17:42:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.417+0000 7fb42d059640 1 --2- 172.21.15.98:0/860672716 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4280cabf0 0x7fb4280cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.417+0000 7fb42d059640 1 --2- 172.21.15.98:0/860672716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4280cba90 0x7fb42815a4f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.417+0000 7fb42d059640 1 -- 172.21.15.98:0/860672716 >> 172.21.15.98:0/860672716 conn(0x7fb42806cec0 msgr2=0x7fb428066230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.418+0000 7fb42d059640 1 -- 172.21.15.98:0/860672716 shutdown_connections 2023-11-28T17:42:00.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.418+0000 7fb42d059640 1 -- 172.21.15.98:0/860672716 wait complete. 2023-11-28T17:42:00.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.419+0000 7fb42d059640 1 Processor -- start 2023-11-28T17:42:00.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.419+0000 7fb42d059640 1 -- start start 2023-11-28T17:42:00.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.420+0000 7fb42d059640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4280cabf0 0x7fb4281f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:00.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.420+0000 7fb42d059640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4280cba90 0x7fb42814a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:00.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.420+0000 7fb42d059640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4280cc3b0 0x7fb428150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:00.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.420+0000 7fb42d059640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4280c71d0 con 0x7fb4280cba90 2023-11-28T17:42:00.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.420+0000 7fb42d059640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4281543e0 con 0x7fb4280cabf0 2023-11-28T17:42:00.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.420+0000 7fb42d059640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb428158a60 con 0x7fb4280cc3b0 2023-11-28T17:42:00.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.421+0000 7fb4277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4280cba90 0x7fb42814a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:00.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.421+0000 7fb4277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4280cba90 0x7fb42814a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50342/0 (socket says 172.21.15.98:50342) 2023-11-28T17:42:00.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.421+0000 7fb4277fe640 1 -- 172.21.15.98:0/2789967418 learned_addr learned my addr 172.21.15.98:0/2789967418 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:00.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.421+0000 7fb427fff640 1 --2- 172.21.15.98:0/2789967418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4280cabf0 0x7fb4281f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:00.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.421+0000 7fb42c858640 1 --2- 172.21.15.98:0/2789967418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4280cc3b0 0x7fb428150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.421+0000 7fb4277fe640 1 -- 172.21.15.98:0/2789967418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4280cc3b0 msgr2=0x7fb428150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.421+0000 7fb4277fe640 1 --2- 172.21.15.98:0/2789967418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4280cc3b0 0x7fb428150580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.421+0000 7fb4277fe640 1 -- 172.21.15.98:0/2789967418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4280cabf0 msgr2=0x7fb4281f32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.421+0000 7fb4277fe640 1 --2- 172.21.15.98:0/2789967418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4280cabf0 0x7fb4281f32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.421+0000 7fb4277fe640 1 -- 172.21.15.98:0/2789967418 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4180094d0 con 0x7fb4280cba90 2023-11-28T17:42:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.421+0000 7fb42c858640 1 --2- 172.21.15.98:0/2789967418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4280cc3b0 0x7fb428150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:42:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.421+0000 7fb4277fe640 1 --2- 172.21.15.98:0/2789967418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4280cba90 0x7fb42814a0f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb41c00e9b0 tx=0x7fb41c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.422+0000 7fb424ff9640 1 -- 172.21.15.98:0/2789967418 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c009670 con 0x7fb4280cba90 2023-11-28T17:42:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.422+0000 7fb424ff9640 1 -- 172.21.15.98:0/2789967418 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41c00c9a0 con 0x7fb4280cba90 2023-11-28T17:42:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.422+0000 7fb424ff9640 1 -- 172.21.15.98:0/2789967418 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c00cd50 con 0x7fb4280cba90 2023-11-28T17:42:00.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.422+0000 7fb42d059640 1 -- 172.21.15.98:0/2789967418 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4281f37e0 con 0x7fb4280cba90 2023-11-28T17:42:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.422+0000 7fb42d059640 1 -- 172.21.15.98:0/2789967418 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb428159d30 con 0x7fb4280cba90 2023-11-28T17:42:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.423+0000 7fb42d059640 1 -- 172.21.15.98:0/2789967418 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb428158bd0 con 0x7fb4280cba90 2023-11-28T17:42:00.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.424+0000 7fb424ff9640 1 -- 172.21.15.98:0/2789967418 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb41c013470 con 0x7fb4280cba90 2023-11-28T17:42:00.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.425+0000 7fb424ff9640 1 --2- 172.21.15.98:0/2789967418 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb41406b2a0 0x7fb41406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:00.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.425+0000 7fb424ff9640 1 -- 172.21.15.98:0/2789967418 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb41c089f20 con 0x7fb4280cba90 2023-11-28T17:42:00.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.425+0000 7fb427fff640 1 --2- 172.21.15.98:0/2789967418 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb41406b2a0 0x7fb41406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:00.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.426+0000 7fb427fff640 1 --2- 172.21.15.98:0/2789967418 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb41406b2a0 0x7fb41406d760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb418036000 tx=0x7fb41803c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:00.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.429+0000 7fb424ff9640 1 -- 172.21.15.98:0/2789967418 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb41c054cf0 con 0x7fb4280cba90 2023-11-28T17:42:00.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.617+0000 7fb42d059640 1 -- 172.21.15.98:0/2789967418 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb428074db0 con 0x7fb4280cba90 2023-11-28T17:42:00.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.618+0000 7fb424ff9640 1 -- 172.21.15.98:0/2789967418 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb41c0170a0 con 0x7fb4280cba90 2023-11-28T17:42:00.619 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-11-28T17:42:00.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.621+0000 7fb42d059640 1 -- 172.21.15.98:0/2789967418 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb41406b2a0 msgr2=0x7fb41406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:00.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.621+0000 7fb42d059640 1 --2- 172.21.15.98:0/2789967418 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb41406b2a0 0x7fb41406d760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb418036000 tx=0x7fb41803c040 comp rx=0 tx=0).stop 2023-11-28T17:42:00.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.621+0000 7fb42d059640 1 -- 172.21.15.98:0/2789967418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4280cba90 msgr2=0x7fb42814a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:00.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.621+0000 7fb42d059640 1 --2- 172.21.15.98:0/2789967418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4280cba90 0x7fb42814a0f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb41c00e9b0 tx=0x7fb41c00ee70 comp rx=0 tx=0).stop 2023-11-28T17:42:00.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.621+0000 7fb42d059640 1 -- 172.21.15.98:0/2789967418 shutdown_connections 2023-11-28T17:42:00.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.621+0000 7fb42d059640 1 --2- 172.21.15.98:0/2789967418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb4280cc3b0 0x7fb428150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.621+0000 7fb42d059640 1 --2- 172.21.15.98:0/2789967418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4280cba90 0x7fb42814a0f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.621+0000 7fb42d059640 1 --2- 172.21.15.98:0/2789967418 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb41406b2a0 0x7fb41406d760 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.621+0000 7fb42d059640 1 --2- 172.21.15.98:0/2789967418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb4280cabf0 0x7fb4281f32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.621+0000 7fb42d059640 1 -- 172.21.15.98:0/2789967418 >> 172.21.15.98:0/2789967418 conn(0x7fb42806cec0 msgr2=0x7fb4281513d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.621+0000 7fb42d059640 1 -- 172.21.15.98:0/2789967418 shutdown_connections 2023-11-28T17:42:00.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.621+0000 7fb42d059640 1 -- 172.21.15.98:0/2789967418 wait complete. 2023-11-28T17:42:00.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:42:00.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.850+0000 7fb3ad968640 1 Processor -- start 2023-11-28T17:42:00.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.850+0000 7fb3ad968640 1 -- start start 2023-11-28T17:42:00.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.852+0000 7fb3ad968640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3a8071380 0x7fb3a8070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:00.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.852+0000 7fb3ad968640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a80c3920 con 0x7fb3a81500f0 2023-11-28T17:42:00.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.852+0000 7fb3ad968640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a80cb560 con 0x7fb3a80caff0 2023-11-28T17:42:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.852+0000 7fb3ad968640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a80cb820 con 0x7fb3a8071380 2023-11-28T17:42:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.853+0000 7fb3a6ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb3a81500f0 0x7fb3a8156b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48106/0 (socket says 172.21.15.98:48106) 2023-11-28T17:42:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.853+0000 7fb3a6ffd640 1 -- 172.21.15.98:0/20734090 learned_addr learned my addr 172.21.15.98:0/20734090 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:42:00.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.853+0000 7fb39e7fc640 1 --2- 172.21.15.98:0/20734090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3a8071380 0x7fb3a8070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:00.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.853+0000 7fb3a67fc640 1 -- 172.21.15.98:0/20734090 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406411644 0 0) 0x7fb3a80c3920 con 0x7fb3a81500f0 2023-11-28T17:42:00.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.853+0000 7fb3a67fc640 1 -- 172.21.15.98:0/20734090 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb394003520 con 0x7fb3a81500f0 2023-11-28T17:42:00.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.853+0000 7fb3a67fc640 1 -- 172.21.15.98:0/20734090 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1211368016 0 0) 0x7fb394003520 con 0x7fb3a81500f0 2023-11-28T17:42:00.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.853+0000 7fb3a67fc640 1 -- 172.21.15.98:0/20734090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3a8071380 msgr2=0x7fb3a8070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:00.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.853+0000 7fb3a67fc640 1 --2- 172.21.15.98:0/20734090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3a8071380 0x7fb3a8070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.853+0000 7fb3a67fc640 1 -- 172.21.15.98:0/20734090 >> v1:172.21.15.113:6789/0 conn(0x7fb3a80caff0 legacy=0x7fb3a80c4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:00.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.853+0000 7fb3a67fc640 1 -- 172.21.15.98:0/20734090 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a80c3b90 con 0x7fb3a81500f0 2023-11-28T17:42:00.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.853+0000 7fb3a67fc640 1 -- 172.21.15.98:0/20734090 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7fb398001670 con 0x7fb3a81500f0 2023-11-28T17:42:00.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb3a67fc640 1 -- 172.21.15.98:0/20734090 >> v1:172.21.15.98:6789/0 conn(0x7fb3a81500f0 legacy=0x7fb3a8156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:00.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb3a67fc640 1 --2- 172.21.15.98:0/20734090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3940039a0 0x7fb394003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:00.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb3a67fc640 1 --2- 172.21.15.98:0/20734090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3a8071380 0x7fb3940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb3a67fc640 1 --2- 172.21.15.98:0/20734090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb394004d00 0x7fb394005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb3a67fc640 1 -- 172.21.15.98:0/20734090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a80cb820 con 0x7fb3940039a0 2023-11-28T17:42:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb3a67fc640 1 -- 172.21.15.98:0/20734090 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39400cbd0 con 0x7fb394004d00 2023-11-28T17:42:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb3a67fc640 1 -- 172.21.15.98:0/20734090 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39400cd90 con 0x7fb3a8071380 2023-11-28T17:42:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb3a6ffd640 1 --2- 172.21.15.98:0/20734090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3940039a0 0x7fb394003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:00.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb39e7fc640 1 --2- 172.21.15.98:0/20734090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3a8071380 0x7fb3940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:00.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb3a77fe640 1 --2- 172.21.15.98:0/20734090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb394004d00 0x7fb394005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:00.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb3a6ffd640 1 -- 172.21.15.98:0/20734090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3a8071380 msgr2=0x7fb3940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:00.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb3a6ffd640 1 --2- 172.21.15.98:0/20734090 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3a8071380 0x7fb3940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb3a6ffd640 1 -- 172.21.15.98:0/20734090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb394004d00 msgr2=0x7fb394005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:00.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb3a6ffd640 1 --2- 172.21.15.98:0/20734090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb394004d00 0x7fb394005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb3a6ffd640 1 -- 172.21.15.98:0/20734090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb394003520 con 0x7fb3940039a0 2023-11-28T17:42:00.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.854+0000 7fb3a77fe640 1 --2- 172.21.15.98:0/20734090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb394004d00 0x7fb394005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:42:00.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.855+0000 7fb3a6ffd640 1 --2- 172.21.15.98:0/20734090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3940039a0 0x7fb394003dc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb398007fd0 tx=0x7fb3940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed29be24ea6aa4d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:00.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.856+0000 7fb3a67fc640 1 -- 172.21.15.98:0/20734090 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398005ad0 con 0x7fb3940039a0 2023-11-28T17:42:00.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.856+0000 7fb3a67fc640 1 -- 172.21.15.98:0/20734090 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb398005c30 con 0x7fb3940039a0 2023-11-28T17:42:00.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.856+0000 7fb3a67fc640 1 -- 172.21.15.98:0/20734090 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39802c870 con 0x7fb3940039a0 2023-11-28T17:42:00.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.856+0000 7fb3ad968640 1 -- 172.21.15.98:0/20734090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3940039a0 msgr2=0x7fb394003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:00.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.856+0000 7fb3ad968640 1 --2- 172.21.15.98:0/20734090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3940039a0 0x7fb394003dc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb398007fd0 tx=0x7fb3940057f0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.856+0000 7fb3a77fe640 1 -- 172.21.15.98:0/20734090 reap_dead start 2023-11-28T17:42:00.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.856+0000 7fb3ad968640 1 -- 172.21.15.98:0/20734090 shutdown_connections 2023-11-28T17:42:00.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.856+0000 7fb3ad968640 1 -- 172.21.15.98:0/20734090 >> 172.21.15.98:0/20734090 conn(0x7fb3a806c0f0 msgr2=0x7fb3a806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:00.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.856+0000 7fb3ad968640 1 -- 172.21.15.98:0/20734090 shutdown_connections 2023-11-28T17:42:00.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.856+0000 7fb3ad968640 1 -- 172.21.15.98:0/20734090 wait complete. 2023-11-28T17:42:00.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.858+0000 7fb3ad968640 1 Processor -- start 2023-11-28T17:42:00.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.858+0000 7fb3ad968640 1 -- start start 2023-11-28T17:42:00.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.859+0000 7fb3ad968640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3940039a0 0x7fb3a8152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:00.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.859+0000 7fb3ad968640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3a81510b0 0x7fb3a81572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:00.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.859+0000 7fb3ad968640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3a81577e0 0x7fb3a80c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:00.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.859+0000 7fb3ad968640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a8147bb0 con 0x7fb3a81577e0 2023-11-28T17:42:00.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.859+0000 7fb3ad968640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a8152470 con 0x7fb3a81510b0 2023-11-28T17:42:00.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.859+0000 7fb3ad968640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a8157c70 con 0x7fb3940039a0 2023-11-28T17:42:00.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.859+0000 7fb3a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3a81577e0 0x7fb3a80c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:00.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.859+0000 7fb3a77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3a81577e0 0x7fb3a80c75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50358/0 (socket says 172.21.15.98:50358) 2023-11-28T17:42:00.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.859+0000 7fb3a77fe640 1 -- 172.21.15.98:0/3831155521 learned_addr learned my addr 172.21.15.98:0/3831155521 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:00.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.860+0000 7fb3a6ffd640 1 --2- 172.21.15.98:0/3831155521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3940039a0 0x7fb3a8152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:00.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.860+0000 7fb39e7fc640 1 --2- 172.21.15.98:0/3831155521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3a81510b0 0x7fb3a81572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:00.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.860+0000 7fb3a77fe640 1 -- 172.21.15.98:0/3831155521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3940039a0 msgr2=0x7fb3a8152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:00.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.860+0000 7fb3a77fe640 1 --2- 172.21.15.98:0/3831155521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3940039a0 0x7fb3a8152720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.860+0000 7fb3a77fe640 1 -- 172.21.15.98:0/3831155521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3a81510b0 msgr2=0x7fb3a81572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:00.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.860+0000 7fb3a77fe640 1 --2- 172.21.15.98:0/3831155521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3a81510b0 0x7fb3a81572a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:00.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.860+0000 7fb3a77fe640 1 -- 172.21.15.98:0/3831155521 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a8157f50 con 0x7fb3a81577e0 2023-11-28T17:42:00.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.860+0000 7fb3a6ffd640 1 --2- 172.21.15.98:0/3831155521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3940039a0 0x7fb3a8152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:42:00.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.860+0000 7fb3a77fe640 1 --2- 172.21.15.98:0/3831155521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3a81577e0 0x7fb3a80c75f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb3a80cbe60 tx=0x7fb3a000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:00.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.860+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/3831155521 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0006d50 con 0x7fb3a81577e0 2023-11-28T17:42:00.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.860+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/3831155521 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a0003970 con 0x7fb3a81577e0 2023-11-28T17:42:00.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.860+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/3831155521 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0003d60 con 0x7fb3a81577e0 2023-11-28T17:42:00.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.860+0000 7fb3ad968640 1 -- 172.21.15.98:0/3831155521 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a81efd90 con 0x7fb3a81577e0 2023-11-28T17:42:00.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.861+0000 7fb3ad968640 1 -- 172.21.15.98:0/3831155521 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a81f0350 con 0x7fb3a81577e0 2023-11-28T17:42:00.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.862+0000 7fb3ad968640 1 -- 172.21.15.98:0/3831155521 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a8074f60 con 0x7fb3a81577e0 2023-11-28T17:42:00.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.863+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/3831155521 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb3a0005070 con 0x7fb3a81577e0 2023-11-28T17:42:00.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.863+0000 7fb3a4ff9640 1 --2- 172.21.15.98:0/3831155521 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb38806b290 0x7fb38806d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:00.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.863+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/3831155521 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3a0089b50 con 0x7fb3a81577e0 2023-11-28T17:42:00.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.864+0000 7fb3a6ffd640 1 --2- 172.21.15.98:0/3831155521 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb38806b290 0x7fb38806d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:00.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.864+0000 7fb3a6ffd640 1 --2- 172.21.15.98:0/3831155521 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb38806b290 0x7fb38806d750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb398034000 tx=0x7fb398009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:00.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:00.868+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/3831155521 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3a0054930 con 0x7fb3a81577e0 2023-11-28T17:42:01.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.051+0000 7fb3ad968640 1 -- 172.21.15.98:0/3831155521 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb3a814f8f0 con 0x7fb3a81577e0 2023-11-28T17:42:01.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.052+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/3831155521 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb3a0057f50 con 0x7fb3a81577e0 2023-11-28T17:42:01.053 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-11-28T17:42:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.054+0000 7fb3ad968640 1 -- 172.21.15.98:0/3831155521 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb38806b290 msgr2=0x7fb38806d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.055+0000 7fb3ad968640 1 --2- 172.21.15.98:0/3831155521 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb38806b290 0x7fb38806d750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb398034000 tx=0x7fb398009030 comp rx=0 tx=0).stop 2023-11-28T17:42:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.055+0000 7fb3ad968640 1 -- 172.21.15.98:0/3831155521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3a81577e0 msgr2=0x7fb3a80c75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.055+0000 7fb3ad968640 1 --2- 172.21.15.98:0/3831155521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3a81577e0 0x7fb3a80c75f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb3a80cbe60 tx=0x7fb3a000dc00 comp rx=0 tx=0).stop 2023-11-28T17:42:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.055+0000 7fb3ad968640 1 -- 172.21.15.98:0/3831155521 shutdown_connections 2023-11-28T17:42:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.055+0000 7fb3ad968640 1 --2- 172.21.15.98:0/3831155521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3a81577e0 0x7fb3a80c75f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.055+0000 7fb3ad968640 1 --2- 172.21.15.98:0/3831155521 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb38806b290 0x7fb38806d750 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.055+0000 7fb3ad968640 1 --2- 172.21.15.98:0/3831155521 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3a81510b0 0x7fb3a81572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.055+0000 7fb3ad968640 1 --2- 172.21.15.98:0/3831155521 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb3940039a0 0x7fb3a8152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.055+0000 7fb3ad968640 1 -- 172.21.15.98:0/3831155521 >> 172.21.15.98:0/3831155521 conn(0x7fb3a806c0f0 msgr2=0x7fb3a806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.055+0000 7fb3ad968640 1 -- 172.21.15.98:0/3831155521 shutdown_connections 2023-11-28T17:42:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.055+0000 7fb3ad968640 1 -- 172.21.15.98:0/3831155521 wait complete. 2023-11-28T17:42:01.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:42:01.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.283+0000 7fa4c359e640 1 Processor -- start 2023-11-28T17:42:01.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.283+0000 7fa4c359e640 1 -- start start 2023-11-28T17:42:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.284+0000 7fa4c359e640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa4c4150df0 0x7fa4c406a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.284+0000 7fa4c359e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4c4070ce0 0x7fa4c406aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.284+0000 7fa4c359e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa4c406b210 0x7fa4c4071b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.284+0000 7fa4c359e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c41511d0 con 0x7fa4c406b210 2023-11-28T17:42:01.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.284+0000 7fa4c359e640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c40711a0 con 0x7fa4c4150df0 2023-11-28T17:42:01.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.284+0000 7fa4c359e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c4157480 con 0x7fa4c4070ce0 2023-11-28T17:42:01.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.285+0000 7fa4c2d9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa4c406b210 0x7fa4c4071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:01.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.285+0000 7fa4c2d9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa4c406b210 0x7fa4c4071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50360/0 (socket says 172.21.15.98:50360) 2023-11-28T17:42:01.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.285+0000 7fa4c2d9d640 1 -- 172.21.15.98:0/1207129397 learned_addr learned my addr 172.21.15.98:0/1207129397 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:01.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.285+0000 7fa4c259c640 1 --2- 172.21.15.98:0/1207129397 >> v2:172.21.15.113:3300/0 conn(0x7fa4c4150df0 0x7fa4c406a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:01.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.285+0000 7fa4bbfff640 1 --2- 172.21.15.98:0/1207129397 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4c4070ce0 0x7fa4c406aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:01.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.285+0000 7fa4c2d9d640 1 -- 172.21.15.98:0/1207129397 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4c4070ce0 msgr2=0x7fa4c406aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:01.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.285+0000 7fa4c2d9d640 1 --2- 172.21.15.98:0/1207129397 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4c4070ce0 0x7fa4c406aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.285+0000 7fa4c2d9d640 1 -- 172.21.15.98:0/1207129397 >> v2:172.21.15.113:3300/0 conn(0x7fa4c4150df0 msgr2=0x7fa4c406a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:01.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.285+0000 7fa4c2d9d640 1 --2- 172.21.15.98:0/1207129397 >> v2:172.21.15.113:3300/0 conn(0x7fa4c4150df0 0x7fa4c406a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.285+0000 7fa4c2d9d640 1 -- 172.21.15.98:0/1207129397 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c4070a40 con 0x7fa4c406b210 2023-11-28T17:42:01.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.286+0000 7fa4c2d9d640 1 --2- 172.21.15.98:0/1207129397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c406b210 0x7fa4c4071b10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40cc020 tx=0x7fa4bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a711841e4445ee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:01.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.286+0000 7fa4c1d9b640 1 -- 172.21.15.98:0/1207129397 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc02ca70 con 0x7fa4c406b210 2023-11-28T17:42:01.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.287+0000 7fa4c1d9b640 1 -- 172.21.15.98:0/1207129397 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4bc02cbd0 con 0x7fa4c406b210 2023-11-28T17:42:01.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.287+0000 7fa4c1d9b640 1 -- 172.21.15.98:0/1207129397 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc0343f0 con 0x7fa4c406b210 2023-11-28T17:42:01.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.287+0000 7fa4c359e640 1 -- 172.21.15.98:0/1207129397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c406b210 msgr2=0x7fa4c4071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:01.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.287+0000 7fa4c359e640 1 --2- 172.21.15.98:0/1207129397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c406b210 0x7fa4c4071b10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40cc020 tx=0x7fa4bc02bed0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.287+0000 7fa4c359e640 1 -- 172.21.15.98:0/1207129397 shutdown_connections 2023-11-28T17:42:01.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.287+0000 7fa4c359e640 1 --2- 172.21.15.98:0/1207129397 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4c4070ce0 0x7fa4c406aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.287+0000 7fa4c359e640 1 --2- 172.21.15.98:0/1207129397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c406b210 0x7fa4c4071b10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.287+0000 7fa4c359e640 1 --2- 172.21.15.98:0/1207129397 >> v2:172.21.15.113:3300/0 conn(0x7fa4c4150df0 0x7fa4c406a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.287+0000 7fa4c359e640 1 -- 172.21.15.98:0/1207129397 >> 172.21.15.98:0/1207129397 conn(0x7fa4c40cafe0 msgr2=0x7fa4c40cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:01.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.287+0000 7fa4c359e640 1 -- 172.21.15.98:0/1207129397 shutdown_connections 2023-11-28T17:42:01.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.287+0000 7fa4c359e640 1 -- 172.21.15.98:0/1207129397 wait complete. 2023-11-28T17:42:01.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.289+0000 7fa4c359e640 1 Processor -- start 2023-11-28T17:42:01.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.289+0000 7fa4c359e640 1 -- start start 2023-11-28T17:42:01.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.290+0000 7fa4c359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c406b210 0x7fa4c41ed440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.290+0000 7fa4c359e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4c4070ce0 0x7fa4c4158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.290+0000 7fa4c359e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4c4150df0 0x7fa4c41e10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.290+0000 7fa4c359e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c4157bc0 con 0x7fa4c406b210 2023-11-28T17:42:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.290+0000 7fa4c359e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c4157e80 con 0x7fa4c4150df0 2023-11-28T17:42:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.290+0000 7fa4c359e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c41f13c0 con 0x7fa4c4070ce0 2023-11-28T17:42:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.290+0000 7fa4c259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c406b210 0x7fa4c41ed440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:01.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.290+0000 7fa4c259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c406b210 0x7fa4c41ed440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50362/0 (socket says 172.21.15.98:50362) 2023-11-28T17:42:01.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.290+0000 7fa4c259c640 1 -- 172.21.15.98:0/1469877849 learned_addr learned my addr 172.21.15.98:0/1469877849 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:01.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.291+0000 7fa4c259c640 1 -- 172.21.15.98:0/1469877849 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4c4070ce0 msgr2=0x7fa4c4158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:01.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.291+0000 7fa4c259c640 1 --2- 172.21.15.98:0/1469877849 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4c4070ce0 0x7fa4c4158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.291+0000 7fa4c259c640 1 -- 172.21.15.98:0/1469877849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4c4150df0 msgr2=0x7fa4c41e10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:01.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.291+0000 7fa4c259c640 1 --2- 172.21.15.98:0/1469877849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4c4150df0 0x7fa4c41e10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.291+0000 7fa4c259c640 1 -- 172.21.15.98:0/1469877849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4bc009b30 con 0x7fa4c406b210 2023-11-28T17:42:01.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.291+0000 7fa4c259c640 1 --2- 172.21.15.98:0/1469877849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c406b210 0x7fa4c41ed440 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400d5f0 tx=0x7fa4b400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.291+0000 7fa4baffd640 1 -- 172.21.15.98:0/1469877849 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b40125f0 con 0x7fa4c406b210 2023-11-28T17:42:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.291+0000 7fa4baffd640 1 -- 172.21.15.98:0/1469877849 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b4012750 con 0x7fa4c406b210 2023-11-28T17:42:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.291+0000 7fa4baffd640 1 -- 172.21.15.98:0/1469877849 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4012b00 con 0x7fa4c406b210 2023-11-28T17:42:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.291+0000 7fa4c359e640 1 -- 172.21.15.98:0/1469877849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c41f16a0 con 0x7fa4c406b210 2023-11-28T17:42:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.292+0000 7fa4c359e640 1 -- 172.21.15.98:0/1469877849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c40c20a0 con 0x7fa4c406b210 2023-11-28T17:42:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.293+0000 7fa4c359e640 1 -- 172.21.15.98:0/1469877849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c4067de0 con 0x7fa4c406b210 2023-11-28T17:42:01.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.293+0000 7fa4baffd640 1 -- 172.21.15.98:0/1469877849 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa4b4012c60 con 0x7fa4c406b210 2023-11-28T17:42:01.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.294+0000 7fa4baffd640 1 --2- 172.21.15.98:0/1469877849 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa4a406b2a0 0x7fa4a406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.294+0000 7fa4baffd640 1 -- 172.21.15.98:0/1469877849 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4b408a1d0 con 0x7fa4c406b210 2023-11-28T17:42:01.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.295+0000 7fa4bbfff640 1 --2- 172.21.15.98:0/1469877849 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa4a406b2a0 0x7fa4a406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:01.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.295+0000 7fa4bbfff640 1 --2- 172.21.15.98:0/1469877849 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa4a406b2a0 0x7fa4a406d760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4150b50 tx=0x7fa4ac008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:01.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.298+0000 7fa4baffd640 1 -- 172.21.15.98:0/1469877849 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4b4054f10 con 0x7fa4c406b210 2023-11-28T17:42:01.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.476+0000 7fa4c359e640 1 -- 172.21.15.98:0/1469877849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa4c41f1890 con 0x7fa4c406b210 2023-11-28T17:42:01.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.476+0000 7fa4baffd640 1 -- 172.21.15.98:0/1469877849 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa4b4058530 con 0x7fa4c406b210 2023-11-28T17:42:01.477 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-11-28T17:42:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.479+0000 7fa4c359e640 1 -- 172.21.15.98:0/1469877849 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa4a406b2a0 msgr2=0x7fa4a406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.479+0000 7fa4c359e640 1 --2- 172.21.15.98:0/1469877849 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa4a406b2a0 0x7fa4a406d760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4150b50 tx=0x7fa4ac008040 comp rx=0 tx=0).stop 2023-11-28T17:42:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.479+0000 7fa4c359e640 1 -- 172.21.15.98:0/1469877849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c406b210 msgr2=0x7fa4c41ed440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.479+0000 7fa4c359e640 1 --2- 172.21.15.98:0/1469877849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c406b210 0x7fa4c41ed440 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400d5f0 tx=0x7fa4b400dab0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.479+0000 7fa4c359e640 1 -- 172.21.15.98:0/1469877849 shutdown_connections 2023-11-28T17:42:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.479+0000 7fa4c359e640 1 --2- 172.21.15.98:0/1469877849 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa4a406b2a0 0x7fa4a406d760 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.479+0000 7fa4c359e640 1 --2- 172.21.15.98:0/1469877849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4c4150df0 0x7fa4c41e10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.479+0000 7fa4c359e640 1 --2- 172.21.15.98:0/1469877849 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa4c4070ce0 0x7fa4c4158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.479+0000 7fa4c359e640 1 --2- 172.21.15.98:0/1469877849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4c406b210 0x7fa4c41ed440 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.479+0000 7fa4c359e640 1 -- 172.21.15.98:0/1469877849 >> 172.21.15.98:0/1469877849 conn(0x7fa4c40cafe0 msgr2=0x7fa4c414f440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:01.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.479+0000 7fa4c359e640 1 -- 172.21.15.98:0/1469877849 shutdown_connections 2023-11-28T17:42:01.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.479+0000 7fa4c359e640 1 -- 172.21.15.98:0/1469877849 wait complete. 2023-11-28T17:42:01.496 DEBUG:tasks.mon_thrash:Yielding 2023-11-28T17:42:01.496 INFO:teuthology.run_tasks:Running task workunit... 2023-11-28T17:42:01.509 INFO:tasks.workunit:Pulling workunits from ref 47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T17:42:01.509 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-11-28T17:42:01.509 INFO:tasks.workunit:timeout=3h 2023-11-28T17:42:01.509 INFO:tasks.workunit:cleanup=True 2023-11-28T17:42:01.510 DEBUG:teuthology.orchestra.run.smithi098:> stat -- /home/ubuntu/cephtest/mnt.0 2023-11-28T17:42:01.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:42:01.557 INFO:teuthology.orchestra.run.smithi098.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-11-28T17:42:01.557 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2023-11-28T17:42:01.557 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 27h/39d Inode: 1 Links: 2 2023-11-28T17:42:01.558 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:42:01.558 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:fusefs_t:s0 2023-11-28T17:42:01.558 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-11-28T17:42:01.558 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-11-28 17:41:22.254409681 +0000 2023-11-28T17:42:01.558 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-11-28 17:41:57.008188895 +0000 2023-11-28T17:42:01.558 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2023-11-28T17:42:01.558 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-11-28T17:42:01.558 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-11-28T17:42:01.704 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T17:42:01.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.705+0000 7facd841e640 1 Processor -- start 2023-11-28T17:42:01.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.705+0000 7facd841e640 1 -- start start 2023-11-28T17:42:01.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.706+0000 7facd841e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd00cf6d0 0x7facd00c7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.706+0000 7facd841e640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7facd00cfab0 0x7facd0159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.706+0000 7facd841e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd015aa10 con 0x7facd00cb780 2023-11-28T17:42:01.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.707+0000 7facd841e640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd015acf0 con 0x7facd00cfab0 2023-11-28T17:42:01.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.707+0000 7facd841e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd015d430 con 0x7facd00cf6d0 2023-11-28T17:42:01.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.707+0000 7facd69bb640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7facd00cb780 0x7facd0153cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48114/0 (socket says 172.21.15.98:48114) 2023-11-28T17:42:01.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.707+0000 7facd69bb640 1 -- 172.21.15.98:0/667297796 learned_addr learned my addr 172.21.15.98:0/667297796 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:42:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.707+0000 7facd61ba640 1 --2- 172.21.15.98:0/667297796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd00cf6d0 0x7facd00c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.707+0000 7facd59b9640 1 --2- 172.21.15.98:0/667297796 >> v2:172.21.15.113:3300/0 conn(0x7facd00cfab0 0x7facd0159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.707+0000 7facd51b8640 1 -- 172.21.15.98:0/667297796 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528034038 0 0) 0x7facd015aa10 con 0x7facd00cb780 2023-11-28T17:42:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.707+0000 7facd51b8640 1 -- 172.21.15.98:0/667297796 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facbc003520 con 0x7facd00cb780 2023-11-28T17:42:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd51b8640 1 -- 172.21.15.98:0/667297796 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2193073959 0 0) 0x7facbc003520 con 0x7facd00cb780 2023-11-28T17:42:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd51b8640 1 -- 172.21.15.98:0/667297796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd00cf6d0 msgr2=0x7facd00c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd51b8640 1 --2- 172.21.15.98:0/667297796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd00cf6d0 0x7facd00c7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd51b8640 1 -- 172.21.15.98:0/667297796 >> v2:172.21.15.113:3300/0 conn(0x7facd00cfab0 msgr2=0x7facd0159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd51b8640 1 --2- 172.21.15.98:0/667297796 >> v2:172.21.15.113:3300/0 conn(0x7facd00cfab0 0x7facd0159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd51b8640 1 -- 172.21.15.98:0/667297796 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd015d710 con 0x7facd00cb780 2023-11-28T17:42:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd51b8640 1 -- 172.21.15.98:0/667297796 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7facc80019f0 con 0x7facd00cb780 2023-11-28T17:42:01.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd51b8640 1 -- 172.21.15.98:0/667297796 >> v1:172.21.15.98:6789/0 conn(0x7facd00cb780 legacy=0x7facd0153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:01.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd51b8640 1 --2- 172.21.15.98:0/667297796 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facbc003970 0x7facbc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd51b8640 1 --2- 172.21.15.98:0/667297796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facbc0047c0 0x7facbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd51b8640 1 --2- 172.21.15.98:0/667297796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd00cf6d0 0x7facbc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd51b8640 1 -- 172.21.15.98:0/667297796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd015acf0 con 0x7facbc0047c0 2023-11-28T17:42:01.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd51b8640 1 -- 172.21.15.98:0/667297796 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd015d430 con 0x7facbc003970 2023-11-28T17:42:01.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd51b8640 1 -- 172.21.15.98:0/667297796 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facbc00b980 con 0x7facd00cf6d0 2023-11-28T17:42:01.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd59b9640 1 --2- 172.21.15.98:0/667297796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facbc0047c0 0x7facbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:01.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd61ba640 1 --2- 172.21.15.98:0/667297796 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facbc003970 0x7facbc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:01.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.708+0000 7facd69bb640 1 --2- 172.21.15.98:0/667297796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd00cf6d0 0x7facbc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:01.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.709+0000 7facd59b9640 1 -- 172.21.15.98:0/667297796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd00cf6d0 msgr2=0x7facbc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:01.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.709+0000 7facd59b9640 1 --2- 172.21.15.98:0/667297796 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd00cf6d0 0x7facbc005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.709+0000 7facd59b9640 1 -- 172.21.15.98:0/667297796 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facbc003970 msgr2=0x7facbc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:01.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.709+0000 7facd59b9640 1 --2- 172.21.15.98:0/667297796 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facbc003970 0x7facbc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.709+0000 7facd59b9640 1 -- 172.21.15.98:0/667297796 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc003520 con 0x7facbc0047c0 2023-11-28T17:42:01.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.710+0000 7facd59b9640 1 --2- 172.21.15.98:0/667297796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facbc0047c0 0x7facbc004c40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7facc0009ac0 tx=0x7facbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3498574728df645 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:01.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.710+0000 7facd51b8640 1 -- 172.21.15.98:0/667297796 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc002c6d0 con 0x7facbc0047c0 2023-11-28T17:42:01.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.710+0000 7facd51b8640 1 -- 172.21.15.98:0/667297796 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc002c830 con 0x7facbc0047c0 2023-11-28T17:42:01.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.710+0000 7facd51b8640 1 -- 172.21.15.98:0/667297796 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc002cbc0 con 0x7facbc0047c0 2023-11-28T17:42:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.710+0000 7facd841e640 1 -- 172.21.15.98:0/667297796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facbc0047c0 msgr2=0x7facbc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.710+0000 7facd841e640 1 --2- 172.21.15.98:0/667297796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facbc0047c0 0x7facbc004c40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7facc0009ac0 tx=0x7facbc0057f0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.710+0000 7facd69bb640 1 -- 172.21.15.98:0/667297796 reap_dead start 2023-11-28T17:42:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.710+0000 7facd841e640 1 -- 172.21.15.98:0/667297796 shutdown_connections 2023-11-28T17:42:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.711+0000 7facd841e640 1 -- 172.21.15.98:0/667297796 >> 172.21.15.98:0/667297796 conn(0x7facd006af10 msgr2=0x7facd006b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.711+0000 7facd841e640 1 -- 172.21.15.98:0/667297796 shutdown_connections 2023-11-28T17:42:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.711+0000 7facd841e640 1 -- 172.21.15.98:0/667297796 wait complete. 2023-11-28T17:42:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.712+0000 7facd841e640 1 Processor -- start 2023-11-28T17:42:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.712+0000 7facd841e640 1 -- start start 2023-11-28T17:42:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.713+0000 7facd841e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facbc0047c0 0x7facd01f0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.713+0000 7facd841e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd0157fd0 0x7facd00ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.713+0000 7facd841e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facd0159b00 0x7facd015c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.713+0000 7facd841e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd014d510 con 0x7facd0157fd0 2023-11-28T17:42:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.713+0000 7facd841e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd00c5380 con 0x7facd0159b00 2023-11-28T17:42:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.713+0000 7facd841e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd01efa30 con 0x7facbc0047c0 2023-11-28T17:42:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.714+0000 7facd59b9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd0157fd0 0x7facd00ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:01.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.714+0000 7facd59b9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd0157fd0 0x7facd00ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50374/0 (socket says 172.21.15.98:50374) 2023-11-28T17:42:01.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.714+0000 7facd59b9640 1 -- 172.21.15.98:0/4248746784 learned_addr learned my addr 172.21.15.98:0/4248746784 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:01.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.714+0000 7facd61ba640 1 --2- 172.21.15.98:0/4248746784 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facbc0047c0 0x7facd01f0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:01.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.714+0000 7facd69bb640 1 --2- 172.21.15.98:0/4248746784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facd0159b00 0x7facd015c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:01.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.714+0000 7facd59b9640 1 -- 172.21.15.98:0/4248746784 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facbc0047c0 msgr2=0x7facd01f0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:01.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.714+0000 7facd59b9640 1 --2- 172.21.15.98:0/4248746784 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facbc0047c0 0x7facd01f0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.714+0000 7facd59b9640 1 -- 172.21.15.98:0/4248746784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facd0159b00 msgr2=0x7facd015c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:01.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.714+0000 7facd59b9640 1 --2- 172.21.15.98:0/4248746784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facd0159b00 0x7facd015c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.714+0000 7facd59b9640 1 -- 172.21.15.98:0/4248746784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd015e370 con 0x7facd0157fd0 2023-11-28T17:42:01.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.714+0000 7facd59b9640 1 --2- 172.21.15.98:0/4248746784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd0157fd0 0x7facd00ce6e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7facc002c2d0 tx=0x7facc0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:01.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.715+0000 7facc6ffd640 1 -- 172.21.15.98:0/4248746784 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc0003810 con 0x7facd0157fd0 2023-11-28T17:42:01.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.715+0000 7facc6ffd640 1 -- 172.21.15.98:0/4248746784 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc0003970 con 0x7facd0157fd0 2023-11-28T17:42:01.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.715+0000 7facc6ffd640 1 -- 172.21.15.98:0/4248746784 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc0003ce0 con 0x7facd0157fd0 2023-11-28T17:42:01.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.715+0000 7facd841e640 1 -- 172.21.15.98:0/4248746784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd01f0980 con 0x7facd0157fd0 2023-11-28T17:42:01.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.715+0000 7facd841e640 1 -- 172.21.15.98:0/4248746784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd014cf50 con 0x7facd0157fd0 2023-11-28T17:42:01.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.716+0000 7facd841e640 1 -- 172.21.15.98:0/4248746784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd0074f60 con 0x7facd0157fd0 2023-11-28T17:42:01.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.717+0000 7facc6ffd640 1 -- 172.21.15.98:0/4248746784 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7facc0040660 con 0x7facd0157fd0 2023-11-28T17:42:01.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.718+0000 7facc6ffd640 1 --2- 172.21.15.98:0/4248746784 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7facb406b2d0 0x7facb406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:01.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.718+0000 7facc6ffd640 1 -- 172.21.15.98:0/4248746784 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7facc00abce0 con 0x7facd0157fd0 2023-11-28T17:42:01.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.718+0000 7facd61ba640 1 --2- 172.21.15.98:0/4248746784 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7facb406b2d0 0x7facb406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:01.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.719+0000 7facd61ba640 1 --2- 172.21.15.98:0/4248746784 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7facb406b2d0 0x7facb406d790 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faccc009930 tx=0x7faccc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:01.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.722+0000 7facc6ffd640 1 -- 172.21.15.98:0/4248746784 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7facc007a5e0 con 0x7facd0157fd0 2023-11-28T17:42:01.748 INFO:tasks.workunit.client.0.smithi098.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-11-28T17:42:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.957+0000 7facd841e640 1 -- 172.21.15.98:0/4248746784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facd0150130 con 0x7facd0157fd0 2023-11-28T17:42:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.958+0000 7facc6ffd640 1 -- 172.21.15.98:0/4248746784 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6559 (secure 0 0 0) 0x7facc00044e0 con 0x7facd0157fd0 2023-11-28T17:42:01.959 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:42:01.959 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/2429440814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":2429440814},{"type":"v1","addr":"172.21.15.98:6836","nonce":2429440814}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:41:16.691765+0000","modified":"2023-11-28T17:41:44.341058+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4273,"mds_2":4272,"mds_3":4270,"mds_4":4269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"d","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.113:6835/1057450253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6833","nonce":1057450253},{"type":"v1","addr":"172.21.15.113:6835","nonce":1057450253}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"c","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.98:6837/1925178583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1925178583},{"type":"v1","addr":"172.21.15.98:6837","nonce":1925178583}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4272":{"gid":4272,"name":"f","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.113:6837/1732613907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":1732613907},{"type":"v1","addr":"172.21.15.113:6837","nonce":1732613907}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4273":{"gid":4273,"name":"e","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.98:6839/3585869153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3585869153},{"type":"v1","addr":"172.21.15.98:6839","nonce":3585869153}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4291":{"gid":4291,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.113:6834/2183585971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2183585971},{"type":"v1","addr":"172.21.15.113:6834","nonce":2183585971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T17:42:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 16 2023-11-28T17:42:01.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.960+0000 7facd841e640 1 -- 172.21.15.98:0/4248746784 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7facb406b2d0 msgr2=0x7facb406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.960+0000 7facd841e640 1 --2- 172.21.15.98:0/4248746784 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7facb406b2d0 0x7facb406d790 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faccc009930 tx=0x7faccc008040 comp rx=0 tx=0).stop 2023-11-28T17:42:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.960+0000 7facd841e640 1 -- 172.21.15.98:0/4248746784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd0157fd0 msgr2=0x7facd00ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.960+0000 7facd841e640 1 --2- 172.21.15.98:0/4248746784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd0157fd0 0x7facd00ce6e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7facc002c2d0 tx=0x7facc0004730 comp rx=0 tx=0).stop 2023-11-28T17:42:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.960+0000 7facd841e640 1 -- 172.21.15.98:0/4248746784 shutdown_connections 2023-11-28T17:42:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.960+0000 7facd841e640 1 --2- 172.21.15.98:0/4248746784 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7facb406b2d0 0x7facb406d790 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.960+0000 7facd841e640 1 --2- 172.21.15.98:0/4248746784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facd0159b00 0x7facd015c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.960+0000 7facd841e640 1 --2- 172.21.15.98:0/4248746784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd0157fd0 0x7facd00ce6e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.960+0000 7facd841e640 1 --2- 172.21.15.98:0/4248746784 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facbc0047c0 0x7facd01f0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.960+0000 7facd841e640 1 -- 172.21.15.98:0/4248746784 >> 172.21.15.98:0/4248746784 conn(0x7facd006af10 msgr2=0x7facd00cc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.961+0000 7facd841e640 1 -- 172.21.15.98:0/4248746784 shutdown_connections 2023-11-28T17:42:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:01.961+0000 7facd841e640 1 -- 172.21.15.98:0/4248746784 wait complete. 2023-11-28T17:42:01.979 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2023-11-28T17:42:01.979 INFO:tasks.mon_thrash.mon_thrasher:seed: 1701193319, revive delay: 20.0, thrash delay: 10.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2023-11-28T17:42:01.980 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:42:01.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:42:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.221+0000 7f7ed7ee7640 1 Processor -- start 2023-11-28T17:42:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.222+0000 7f7ed7ee7640 1 -- start start 2023-11-28T17:42:02.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.223+0000 7f7ed7ee7640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7ed00cf6d0 0x7f7ed00c7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.223+0000 7f7ed7ee7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7ed00cfab0 0x7f7ed0159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.223+0000 7f7ed7ee7640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed015aa10 con 0x7f7ed00cb780 2023-11-28T17:42:02.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.223+0000 7f7ed7ee7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ed015acf0 con 0x7f7ed00cfab0 2023-11-28T17:42:02.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.223+0000 7f7ed7ee7640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ed015d430 con 0x7f7ed00cf6d0 2023-11-28T17:42:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.224+0000 7f7ed5482640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7ed00cfab0 0x7f7ed0159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.224+0000 7f7ed5482640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7ed00cfab0 0x7f7ed0159e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50390/0 (socket says 172.21.15.98:50390) 2023-11-28T17:42:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.224+0000 7f7ed5482640 1 -- 172.21.15.98:0/810455533 learned_addr learned my addr 172.21.15.98:0/810455533 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.224+0000 7f7ed5c83640 1 --2- 172.21.15.98:0/810455533 >> v2:172.21.15.113:3300/0 conn(0x7f7ed00cf6d0 0x7f7ed00c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.224+0000 7f7ed5482640 1 -- 172.21.15.98:0/810455533 >> v2:172.21.15.113:3300/0 conn(0x7f7ed00cf6d0 msgr2=0x7f7ed00c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.224+0000 7f7ed5482640 1 --2- 172.21.15.98:0/810455533 >> v2:172.21.15.113:3300/0 conn(0x7f7ed00cf6d0 0x7f7ed00c7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.224+0000 7f7ed5482640 1 -- 172.21.15.98:0/810455533 >> v1:172.21.15.113:6789/0 conn(0x7f7ed00cb780 legacy=0x7f7ed0153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.224+0000 7f7ed5482640 1 -- 172.21.15.98:0/810455533 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed015d710 con 0x7f7ed00cfab0 2023-11-28T17:42:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.224+0000 7f7ed5c83640 1 --2- 172.21.15.98:0/810455533 >> v2:172.21.15.113:3300/0 conn(0x7f7ed00cf6d0 0x7f7ed00c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:42:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.225+0000 7f7ed5482640 1 --2- 172.21.15.98:0/810455533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed00cfab0 0x7f7ed0159e80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00c9b10 tx=0x7f7ec002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d9dafd8e09191d1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:02.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.225+0000 7f7ed4c81640 1 -- 172.21.15.98:0/810455533 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0031870 con 0x7f7ed00cfab0 2023-11-28T17:42:02.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.225+0000 7f7ed4c81640 1 -- 172.21.15.98:0/810455533 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec00319d0 con 0x7f7ed00cfab0 2023-11-28T17:42:02.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.225+0000 7f7ed4c81640 1 -- 172.21.15.98:0/810455533 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0031d60 con 0x7f7ed00cfab0 2023-11-28T17:42:02.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.226+0000 7f7ed7ee7640 1 -- 172.21.15.98:0/810455533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed00cfab0 msgr2=0x7f7ed0159e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.226+0000 7f7ed7ee7640 1 --2- 172.21.15.98:0/810455533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed00cfab0 0x7f7ed0159e80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00c9b10 tx=0x7f7ec002b910 comp rx=0 tx=0).stop 2023-11-28T17:42:02.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.226+0000 7f7ed7ee7640 1 -- 172.21.15.98:0/810455533 shutdown_connections 2023-11-28T17:42:02.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.226+0000 7f7ed7ee7640 1 --2- 172.21.15.98:0/810455533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed00cfab0 0x7f7ed0159e80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.226+0000 7f7ed7ee7640 1 --2- 172.21.15.98:0/810455533 >> v2:172.21.15.113:3300/0 conn(0x7f7ed00cf6d0 0x7f7ed00c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.226+0000 7f7ed7ee7640 1 -- 172.21.15.98:0/810455533 >> 172.21.15.98:0/810455533 conn(0x7f7ed006af10 msgr2=0x7f7ed006b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:02.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.226+0000 7f7ed7ee7640 1 -- 172.21.15.98:0/810455533 shutdown_connections 2023-11-28T17:42:02.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.226+0000 7f7ed7ee7640 1 -- 172.21.15.98:0/810455533 wait complete. 2023-11-28T17:42:02.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.228+0000 7f7ed7ee7640 1 Processor -- start 2023-11-28T17:42:02.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.228+0000 7f7ed7ee7640 1 -- start start 2023-11-28T17:42:02.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.229+0000 7f7ed7ee7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed00cb780 0x7f7ed0157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.229+0000 7f7ed7ee7640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7ed00cf6d0 0x7f7ed0158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.229+0000 7f7ed7ee7640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed00cfab0 0x7f7ed01593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.229+0000 7f7ed7ee7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed00c48f0 con 0x7f7ed00cb780 2023-11-28T17:42:02.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.229+0000 7f7ed7ee7640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed00c5e60 con 0x7f7ed00cfab0 2023-11-28T17:42:02.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.229+0000 7f7ed7ee7640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed00c3ed0 con 0x7f7ed00cf6d0 2023-11-28T17:42:02.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.229+0000 7f7ed5c83640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed00cb780 0x7f7ed0157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.229+0000 7f7ed5c83640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed00cb780 0x7f7ed0157d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50406/0 (socket says 172.21.15.98:50406) 2023-11-28T17:42:02.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.229+0000 7f7ed5c83640 1 -- 172.21.15.98:0/1939340575 learned_addr learned my addr 172.21.15.98:0/1939340575 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:02.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.229+0000 7f7ed5482640 1 --2- 172.21.15.98:0/1939340575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7ed00cf6d0 0x7f7ed0158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.229+0000 7f7ed6484640 1 --2- 172.21.15.98:0/1939340575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed00cfab0 0x7f7ed01593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.230+0000 7f7ed5c83640 1 -- 172.21.15.98:0/1939340575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7ed00cf6d0 msgr2=0x7f7ed0158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.230+0000 7f7ed5c83640 1 --2- 172.21.15.98:0/1939340575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7ed00cf6d0 0x7f7ed0158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.230+0000 7f7ed5c83640 1 -- 172.21.15.98:0/1939340575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed00cfab0 msgr2=0x7f7ed01593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.230+0000 7f7ed5c83640 1 --2- 172.21.15.98:0/1939340575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed00cfab0 0x7f7ed01593b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.230+0000 7f7ed5c83640 1 -- 172.21.15.98:0/1939340575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec00094d0 con 0x7f7ed00cb780 2023-11-28T17:42:02.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.230+0000 7f7ed5c83640 1 --2- 172.21.15.98:0/1939340575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed00cb780 0x7f7ed0157d00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc00c7f0 tx=0x7f7ecc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:02.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.230+0000 7f7ec67fc640 1 -- 172.21.15.98:0/1939340575 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc007a30 con 0x7f7ed00cb780 2023-11-28T17:42:02.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.230+0000 7f7ec67fc640 1 -- 172.21.15.98:0/1939340575 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ecc007b90 con 0x7f7ed00cb780 2023-11-28T17:42:02.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.230+0000 7f7ed7ee7640 1 -- 172.21.15.98:0/1939340575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed015e370 con 0x7f7ed00cb780 2023-11-28T17:42:02.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.230+0000 7f7ec67fc640 1 -- 172.21.15.98:0/1939340575 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc00ce40 con 0x7f7ed00cb780 2023-11-28T17:42:02.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.231+0000 7f7ed7ee7640 1 -- 172.21.15.98:0/1939340575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed00c5b80 con 0x7f7ed00cb780 2023-11-28T17:42:02.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.232+0000 7f7ed7ee7640 1 -- 172.21.15.98:0/1939340575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed0074f60 con 0x7f7ed00cb780 2023-11-28T17:42:02.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.232+0000 7f7ec67fc640 1 -- 172.21.15.98:0/1939340575 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f7ecc0113f0 con 0x7f7ed00cb780 2023-11-28T17:42:02.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.233+0000 7f7ec67fc640 1 --2- 172.21.15.98:0/1939340575 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7eb406b2a0 0x7f7eb406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.233+0000 7f7ec67fc640 1 -- 172.21.15.98:0/1939340575 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ecc089d10 con 0x7f7ed00cb780 2023-11-28T17:42:02.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.233+0000 7f7ed5482640 1 --2- 172.21.15.98:0/1939340575 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7eb406b2a0 0x7f7eb406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.234+0000 7f7ed5482640 1 --2- 172.21.15.98:0/1939340575 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7eb406b2a0 0x7f7eb406d760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0036000 tx=0x7f7ec003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:02.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.238+0000 7f7ec67fc640 1 -- 172.21.15.98:0/1939340575 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ecc054c10 con 0x7f7ed00cb780 2023-11-28T17:42:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.501+0000 7f7ed7ee7640 1 -- 172.21.15.98:0/1939340575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7ed00738c0 con 0x7f7ed00cb780 2023-11-28T17:42:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.502+0000 7f7ec67fc640 1 -- 172.21.15.98:0/1939340575 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7f7ecc01f080 con 0x7f7ed00cb780 2023-11-28T17:42:02.503 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":65,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:42:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.504+0000 7f7ed7ee7640 1 -- 172.21.15.98:0/1939340575 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7eb406b2a0 msgr2=0x7f7eb406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.504+0000 7f7ed7ee7640 1 --2- 172.21.15.98:0/1939340575 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7eb406b2a0 0x7f7eb406d760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0036000 tx=0x7f7ec003c040 comp rx=0 tx=0).stop 2023-11-28T17:42:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.504+0000 7f7ed7ee7640 1 -- 172.21.15.98:0/1939340575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed00cb780 msgr2=0x7f7ed0157d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.504+0000 7f7ed7ee7640 1 --2- 172.21.15.98:0/1939340575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed00cb780 0x7f7ed0157d00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc00c7f0 tx=0x7f7ecc00cbf0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.504+0000 7f7ed7ee7640 1 -- 172.21.15.98:0/1939340575 shutdown_connections 2023-11-28T17:42:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.505+0000 7f7ed7ee7640 1 --2- 172.21.15.98:0/1939340575 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7eb406b2a0 0x7f7eb406d760 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.505+0000 7f7ed7ee7640 1 --2- 172.21.15.98:0/1939340575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed00cfab0 0x7f7ed01593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.505+0000 7f7ed7ee7640 1 --2- 172.21.15.98:0/1939340575 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7ed00cf6d0 0x7f7ed0158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.505+0000 7f7ed7ee7640 1 --2- 172.21.15.98:0/1939340575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ed00cb780 0x7f7ed0157d00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.505+0000 7f7ed7ee7640 1 -- 172.21.15.98:0/1939340575 >> 172.21.15.98:0/1939340575 conn(0x7f7ed006af10 msgr2=0x7f7ed0152a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.505+0000 7f7ed7ee7640 1 -- 172.21.15.98:0/1939340575 shutdown_connections 2023-11-28T17:42:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.505+0000 7f7ed7ee7640 1 -- 172.21.15.98:0/1939340575 wait complete. 2023-11-28T17:42:02.522 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:42:02.522 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-11-28T17:42:02.523 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:42:02.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:42:02.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.729+0000 7f5394f97640 1 Processor -- start 2023-11-28T17:42:02.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.729+0000 7f5394f97640 1 -- start start 2023-11-28T17:42:02.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.730+0000 7f5394f97640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f5390151be0 0x7f53900c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.730+0000 7f5394f97640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53900c5a60 0x7f53901584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.730+0000 7f5394f97640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5390158a10 con 0x7f53900c4730 2023-11-28T17:42:02.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.730+0000 7f5394f97640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53900c8150 con 0x7f5390151be0 2023-11-28T17:42:02.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.730+0000 7f5394f97640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53900c8850 con 0x7f53900c5a60 2023-11-28T17:42:02.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.731+0000 7f538ed9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f53900c4730 0x7f539006dd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48124/0 (socket says 172.21.15.98:48124) 2023-11-28T17:42:02.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.731+0000 7f538ed9d640 1 -- 172.21.15.98:0/716260164 learned_addr learned my addr 172.21.15.98:0/716260164 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:42:02.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.731+0000 7f5387fff640 1 --2- 172.21.15.98:0/716260164 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53900c5a60 0x7f53901584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.731+0000 7f538e59c640 1 --2- 172.21.15.98:0/716260164 >> v2:172.21.15.113:3300/0 conn(0x7f5390151be0 0x7f53900c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.731+0000 7f538dd9b640 1 -- 172.21.15.98:0/716260164 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232260452 0 0) 0x7f5388001490 con 0x7f53900c4730 2023-11-28T17:42:02.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.731+0000 7f538dd9b640 1 -- 172.21.15.98:0/716260164 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f537c003520 con 0x7f53900c4730 2023-11-28T17:42:02.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.731+0000 7f538dd9b640 1 -- 172.21.15.98:0/716260164 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2004259631 0 0) 0x7f537c003520 con 0x7f53900c4730 2023-11-28T17:42:02.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.731+0000 7f538dd9b640 1 -- 172.21.15.98:0/716260164 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53900c5a60 msgr2=0x7f53901584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:02.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.731+0000 7f538dd9b640 1 --2- 172.21.15.98:0/716260164 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53900c5a60 0x7f53901584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.731+0000 7f538dd9b640 1 -- 172.21.15.98:0/716260164 >> v2:172.21.15.113:3300/0 conn(0x7f5390151be0 msgr2=0x7f53900c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:02.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.731+0000 7f538dd9b640 1 --2- 172.21.15.98:0/716260164 >> v2:172.21.15.113:3300/0 conn(0x7f5390151be0 0x7f53900c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.731+0000 7f538dd9b640 1 -- 172.21.15.98:0/716260164 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53900c5f50 con 0x7f53900c4730 2023-11-28T17:42:02.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.731+0000 7f538dd9b640 1 -- 172.21.15.98:0/716260164 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f5388001b80 con 0x7f53900c4730 2023-11-28T17:42:02.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f5387fff640 1 --2- 172.21.15.98:0/716260164 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53900c5a60 0x7f53901584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:42:02.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f538dd9b640 1 -- 172.21.15.98:0/716260164 >> v1:172.21.15.98:6789/0 conn(0x7f53900c4730 legacy=0x7f539006dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f538dd9b640 1 --2- 172.21.15.98:0/716260164 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53900c5a60 0x7f537c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f538dd9b640 1 --2- 172.21.15.98:0/716260164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f537c003f00 0x7f537c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f538dd9b640 1 --2- 172.21.15.98:0/716260164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f537c004d00 0x7f537c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f538dd9b640 1 -- 172.21.15.98:0/716260164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53900c8150 con 0x7f537c003f00 2023-11-28T17:42:02.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f538dd9b640 1 -- 172.21.15.98:0/716260164 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53900c8850 con 0x7f537c004d00 2023-11-28T17:42:02.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f538dd9b640 1 -- 172.21.15.98:0/716260164 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537c00ba00 con 0x7f53900c5a60 2023-11-28T17:42:02.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f5387fff640 1 --2- 172.21.15.98:0/716260164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f537c003f00 0x7f537c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f538e59c640 1 --2- 172.21.15.98:0/716260164 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53900c5a60 0x7f537c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f538ed9d640 1 --2- 172.21.15.98:0/716260164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f537c004d00 0x7f537c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f5387fff640 1 -- 172.21.15.98:0/716260164 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53900c5a60 msgr2=0x7f537c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:02.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f5387fff640 1 --2- 172.21.15.98:0/716260164 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53900c5a60 0x7f537c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f5387fff640 1 -- 172.21.15.98:0/716260164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f537c004d00 msgr2=0x7f537c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:02.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f5387fff640 1 --2- 172.21.15.98:0/716260164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f537c004d00 0x7f537c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.732+0000 7f5387fff640 1 -- 172.21.15.98:0/716260164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537c003520 con 0x7f537c003f00 2023-11-28T17:42:02.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.733+0000 7f5387fff640 1 --2- 172.21.15.98:0/716260164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f537c003f00 0x7f537c0047c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f53900c5e80 tx=0x7f537c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86d45be9a46d63d7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:02.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.734+0000 7f538dd9b640 1 -- 172.21.15.98:0/716260164 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378004030 con 0x7f537c003f00 2023-11-28T17:42:02.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.734+0000 7f538dd9b640 1 -- 172.21.15.98:0/716260164 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5378004190 con 0x7f537c003f00 2023-11-28T17:42:02.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.734+0000 7f538dd9b640 1 -- 172.21.15.98:0/716260164 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378003610 con 0x7f537c003f00 2023-11-28T17:42:02.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.734+0000 7f5394f97640 1 -- 172.21.15.98:0/716260164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f537c003f00 msgr2=0x7f537c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.734+0000 7f5394f97640 1 --2- 172.21.15.98:0/716260164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f537c003f00 0x7f537c0047c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f53900c5e80 tx=0x7f537c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.734+0000 7f538ed9d640 1 -- 172.21.15.98:0/716260164 reap_dead start 2023-11-28T17:42:02.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.734+0000 7f5394f97640 1 -- 172.21.15.98:0/716260164 shutdown_connections 2023-11-28T17:42:02.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.734+0000 7f5394f97640 1 -- 172.21.15.98:0/716260164 >> 172.21.15.98:0/716260164 conn(0x7f539006c0f0 msgr2=0x7f539006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:02.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.734+0000 7f5394f97640 1 -- 172.21.15.98:0/716260164 shutdown_connections 2023-11-28T17:42:02.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.734+0000 7f5394f97640 1 -- 172.21.15.98:0/716260164 wait complete. 2023-11-28T17:42:02.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.736+0000 7f5394f97640 1 Processor -- start 2023-11-28T17:42:02.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.736+0000 7f5394f97640 1 -- start start 2023-11-28T17:42:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.737+0000 7f5394f97640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f537c003f00 0x7f5390159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.737+0000 7f5394f97640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5390150740 0x7f5390154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.737+0000 7f5394f97640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53901523d0 0x7f539014f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.737+0000 7f5394f97640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539014d810 con 0x7f53901523d0 2023-11-28T17:42:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.737+0000 7f5394f97640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539014e020 con 0x7f537c003f00 2023-11-28T17:42:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.737+0000 7f5394f97640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53901ee1c0 con 0x7f5390150740 2023-11-28T17:42:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.737+0000 7f538ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53901523d0 0x7f539014f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.737+0000 7f538ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53901523d0 0x7f539014f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50418/0 (socket says 172.21.15.98:50418) 2023-11-28T17:42:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.737+0000 7f538ed9d640 1 -- 172.21.15.98:0/513424727 learned_addr learned my addr 172.21.15.98:0/513424727 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.738+0000 7f538e59c640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f537c003f00 0x7f5390159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.738+0000 7f5387fff640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5390150740 0x7f5390154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.738+0000 7f538ed9d640 1 -- 172.21.15.98:0/513424727 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5390150740 msgr2=0x7f5390154fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.738+0000 7f538ed9d640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f5390150740 0x7f5390154fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.738+0000 7f538ed9d640 1 -- 172.21.15.98:0/513424727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f537c003f00 msgr2=0x7f5390159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.738+0000 7f538ed9d640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f537c003f00 0x7f5390159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.738+0000 7f538ed9d640 1 -- 172.21.15.98:0/513424727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53901ee4a0 con 0x7f53901523d0 2023-11-28T17:42:02.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.738+0000 7f538e59c640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f537c003f00 0x7f5390159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:42:02.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.738+0000 7f538ed9d640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53901523d0 0x7f539014f820 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f53880013d0 tx=0x7f538800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:02.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.738+0000 7f5386ffd640 1 -- 172.21.15.98:0/513424727 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538800ec10 con 0x7f53901523d0 2023-11-28T17:42:02.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.738+0000 7f5386ffd640 1 -- 172.21.15.98:0/513424727 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538800ed90 con 0x7f53901523d0 2023-11-28T17:42:02.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.738+0000 7f5386ffd640 1 -- 172.21.15.98:0/513424727 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388013660 con 0x7f53901523d0 2023-11-28T17:42:02.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.738+0000 7f5394f97640 1 -- 172.21.15.98:0/513424727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f539014da60 con 0x7f53901523d0 2023-11-28T17:42:02.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.738+0000 7f5394f97640 1 -- 172.21.15.98:0/513424727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53900ca920 con 0x7f53901523d0 2023-11-28T17:42:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.739+0000 7f5394f97640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5390154590 0x7f53901468a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.739+0000 7f5394f97640 1 -- 172.21.15.98:0/513424727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53900caab0 con 0x7f5390154590 2023-11-28T17:42:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.740+0000 7f538e59c640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5390154590 0x7f53901468a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.740+0000 7f538e59c640 1 -- 172.21.15.98:0/513424727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5390155c00 con 0x7f5390154590 2023-11-28T17:42:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.740+0000 7f538e59c640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5390154590 0x7f53901468a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f53800128a0 tx=0x7f5380012d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.740+0000 7f5386ffd640 1 -- 172.21.15.98:0/513424727 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380010cb0 con 0x7f5390154590 2023-11-28T17:42:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.741+0000 7f5386ffd640 1 -- 172.21.15.98:0/513424727 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f53880137c0 con 0x7f53901523d0 2023-11-28T17:42:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.741+0000 7f5386ffd640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f537006b370 0x7f537006d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.742+0000 7f5386ffd640 1 -- 172.21.15.98:0/513424727 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5388089f20 con 0x7f53901523d0 2023-11-28T17:42:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.742+0000 7f5387fff640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f537006b370 0x7f537006d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.742+0000 7f5386ffd640 1 -- 172.21.15.98:0/513424727 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f5380010e10 con 0x7f5390154590 2023-11-28T17:42:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.742+0000 7f5386ffd640 1 -- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5390154590 msgr2=0x7f53901468a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.742+0000 7f5386ffd640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5390154590 0x7f53901468a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f53800128a0 tx=0x7f5380012d60 comp rx=0 tx=0).stop 2023-11-28T17:42:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.742+0000 7f5386ffd640 1 -- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5390154590 msgr2=0x7f53901468a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.742+0000 7f5386ffd640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5390154590 0x7f53901468a0 secure :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f53800128a0 tx=0x7f5380012d60 comp rx=0 tx=0).stop 2023-11-28T17:42:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.742+0000 7f5387fff640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f537006b370 0x7f537006d830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5378035000 tx=0x7f537803a040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:02.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.749+0000 7f5394f97640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5390146de0 0x7f53901f0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.749+0000 7f5394f97640 1 -- 172.21.15.98:0/513424727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53900744d0 con 0x7f5390146de0 2023-11-28T17:42:02.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.749+0000 7f538e59c640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5390146de0 0x7f53901f0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.749+0000 7f538e59c640 1 -- 172.21.15.98:0/513424727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f53900c84e0 con 0x7f5390146de0 2023-11-28T17:42:02.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.750+0000 7f538e59c640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5390146de0 0x7f53901f0c10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f53800128a0 tx=0x7f5380004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:02.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.750+0000 7f5386ffd640 1 -- 172.21.15.98:0/513424727 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380022050 con 0x7f5390146de0 2023-11-28T17:42:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.750+0000 7f5386ffd640 1 -- 172.21.15.98:0/513424727 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4977 (secure 0 0 0) 0x7f5380023040 con 0x7f5390146de0 2023-11-28T17:42:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.750+0000 7f5386ffd640 1 -- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5390146de0 msgr2=0x7f53901f0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.750+0000 7f5386ffd640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5390146de0 0x7f53901f0c10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f53800128a0 tx=0x7f5380004730 comp rx=0 tx=0).stop 2023-11-28T17:42:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.750+0000 7f5386ffd640 1 -- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5390146de0 msgr2=0x7f53901f0c10 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.750+0000 7f5386ffd640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5390146de0 0x7f53901f0c10 secure :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f53800128a0 tx=0x7f5380004730 comp rx=0 tx=0).stop 2023-11-28T17:42:02.754 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:42:02.754 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:42:02.754 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:42:02.755 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:42:02.755 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 8, 2023-11-28T17:42:02.755 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:42:02.755 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:42:02.755 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:42:02.755 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:42:02.755 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:02.755 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 65, 2023-11-28T17:42:02.755 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:02.755 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:42:02.755 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:42:02.755 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:02.756 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:02.756 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:02.756 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:02.756 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:02.756 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:02.756 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:02.756 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:02.756 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:02.756 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:02.756 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:02.756 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:42:02.756 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:42:02.756 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:02.757 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:02.757 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:02.757 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:02.757 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:02.757 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:02.757 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:02.757 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:02.757 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:02.757 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:02.757 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:02.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:02.757 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:42:02.758 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:42:02.758 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:42:02.758 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:42:02.758 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:42:02.758 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:42:02.758 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:02.758 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:02.758 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:42:02.758 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:42:02.758 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:42:02.758 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:42:02.758 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:42:02.759 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:42:02.759 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:42:02.759 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:02.759 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:42:02.759 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:02.759 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:02.759 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:02.759 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:02.759 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:02.759 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:02.759 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:02.759 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:02.760 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:02.760 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:02.760 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:02.760 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:42:02.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:02.760 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:42:02.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:02.760 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:42:02.760 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:42:02.760 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:02.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:02.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:02.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:02.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:42:02.761 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:02.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:02.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:02.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:02.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:42:02.761 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:02.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:02.761 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:02.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:02.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:42:02.761 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:42:02.762 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:02.762 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:02.762 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:02.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:02.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:02.762 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:42:02.762 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:42:02.762 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:02.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:02.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:02.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:02.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:42:02.763 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:02.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:02.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:02.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:02.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:42:02.763 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:02.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:02.763 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:02.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:02.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:42:02.763 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:42:02.763 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:02.763 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:02.764 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:02.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:02.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:02.764 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:42:02.764 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:42:02.764 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:02.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:02.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:02.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:02.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:42:02.765 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:02.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:02.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:02.765 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:02.765 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:42:02.765 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:02.765 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:02.765 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:02.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:02.765 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:42:02.765 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:42:02.765 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:02.765 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:02.766 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:02.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:02.766 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:02.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:02.766 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:42:02.766 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:42:02.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:02.766 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:02.766 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:02.766 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:02.767 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:02.767 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:02.767 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:42:02.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:02.767 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:02.767 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:02.767 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:42:02.767 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:02.767 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:02.767 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:42:02.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:02.767 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:02.768 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:02.768 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:42:02.768 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:02.768 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:02.768 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:42:02.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:02.768 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:02.768 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:02.768 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 7 2023-11-28T17:42:02.769 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:02.769 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:02.769 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:42:02.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:02.769 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:02.769 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:02.769 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:02.769 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:02.769 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:02.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:02.769 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:42:02.769 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:42:02.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.751+0000 7f5394f97640 1 -- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f537006b370 msgr2=0x7f537006d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.751+0000 7f5394f97640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f537006b370 0x7f537006d830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5378035000 tx=0x7f537803a040 comp rx=0 tx=0).stop 2023-11-28T17:42:02.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.751+0000 7f5394f97640 1 -- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53901523d0 msgr2=0x7f539014f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.751+0000 7f5394f97640 1 --2- 172.21.15.98:0/513424727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53901523d0 0x7f539014f820 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f53880013d0 tx=0x7f538800e530 comp rx=0 tx=0).stop 2023-11-28T17:42:02.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.751+0000 7f538ed9d640 1 -- 172.21.15.98:0/513424727 reap_dead start 2023-11-28T17:42:02.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.751+0000 7f538ed9d640 1 -- 172.21.15.98:0/513424727 reap_dead start 2023-11-28T17:42:02.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.751+0000 7f5394f97640 1 -- 172.21.15.98:0/513424727 shutdown_connections 2023-11-28T17:42:02.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.751+0000 7f5394f97640 1 -- 172.21.15.98:0/513424727 >> 172.21.15.98:0/513424727 conn(0x7f539006c0f0 msgr2=0x7f539006e650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:02.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.751+0000 7f5394f97640 1 -- 172.21.15.98:0/513424727 shutdown_connections 2023-11-28T17:42:02.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.751+0000 7f5394f97640 1 -- 172.21.15.98:0/513424727 wait complete. 2023-11-28T17:42:02.772 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:42:02.772 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:42:02.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:42:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.974+0000 7f44805de640 1 Processor -- start 2023-11-28T17:42:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.974+0000 7f44805de640 1 -- start start 2023-11-28T17:42:02.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.976+0000 7f44805de640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f44780c4760 0x7f44780c2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.976+0000 7f44805de640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f447815a0f0 0x7f447815a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.976+0000 7f44805de640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4478156980 con 0x7f44780c4380 2023-11-28T17:42:02.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.976+0000 7f44805de640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44780c2a60 con 0x7f447815a0f0 2023-11-28T17:42:02.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.976+0000 7f44805de640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f447815aa60 con 0x7f44780c4760 2023-11-28T17:42:02.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.976+0000 7f447eb7b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f447815a0f0 0x7f447815a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.976+0000 7f447e37a640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f44780c4380 0x7f44780c6020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48140/0 (socket says 172.21.15.98:48140) 2023-11-28T17:42:02.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.976+0000 7f447eb7b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f447815a0f0 0x7f447815a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50456/0 (socket says 172.21.15.98:50456) 2023-11-28T17:42:02.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.976+0000 7f447eb7b640 1 -- 172.21.15.98:0/1207601064 learned_addr learned my addr 172.21.15.98:0/1207601064 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:02.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.976+0000 7f447db79640 1 --2- 172.21.15.98:0/1207601064 >> v2:172.21.15.113:3300/0 conn(0x7f44780c4760 0x7f44780c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.976+0000 7f447eb7b640 1 -- 172.21.15.98:0/1207601064 >> v2:172.21.15.113:3300/0 conn(0x7f44780c4760 msgr2=0x7f44780c2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:02.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.976+0000 7f447eb7b640 1 --2- 172.21.15.98:0/1207601064 >> v2:172.21.15.113:3300/0 conn(0x7f44780c4760 0x7f44780c2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.976+0000 7f447eb7b640 1 -- 172.21.15.98:0/1207601064 >> v1:172.21.15.98:6789/0 conn(0x7f44780c4380 legacy=0x7f44780c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.976+0000 7f447eb7b640 1 -- 172.21.15.98:0/1207601064 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4478156c40 con 0x7f447815a0f0 2023-11-28T17:42:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.978+0000 7f447eb7b640 1 --2- 172.21.15.98:0/1207601064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f447815a0f0 0x7f447815a520 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f447806b9f0 tx=0x7f447002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581f4b597c9e444 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.978+0000 7f447d378640 1 -- 172.21.15.98:0/1207601064 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447002f850 con 0x7f447815a0f0 2023-11-28T17:42:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.978+0000 7f447d378640 1 -- 172.21.15.98:0/1207601064 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447002f9b0 con 0x7f447815a0f0 2023-11-28T17:42:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.978+0000 7f447d378640 1 -- 172.21.15.98:0/1207601064 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447002fd40 con 0x7f447815a0f0 2023-11-28T17:42:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.978+0000 7f44805de640 1 -- 172.21.15.98:0/1207601064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f447815a0f0 msgr2=0x7f447815a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.978+0000 7f44805de640 1 --2- 172.21.15.98:0/1207601064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f447815a0f0 0x7f447815a520 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f447806b9f0 tx=0x7f447002ba30 comp rx=0 tx=0).stop 2023-11-28T17:42:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.979+0000 7f44805de640 1 -- 172.21.15.98:0/1207601064 shutdown_connections 2023-11-28T17:42:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.979+0000 7f44805de640 1 --2- 172.21.15.98:0/1207601064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f447815a0f0 0x7f447815a520 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.979+0000 7f44805de640 1 --2- 172.21.15.98:0/1207601064 >> v2:172.21.15.113:3300/0 conn(0x7f44780c4760 0x7f44780c2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.979+0000 7f44805de640 1 -- 172.21.15.98:0/1207601064 >> 172.21.15.98:0/1207601064 conn(0x7f447806c0f0 msgr2=0x7f447806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.979+0000 7f44805de640 1 -- 172.21.15.98:0/1207601064 shutdown_connections 2023-11-28T17:42:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.979+0000 7f44805de640 1 -- 172.21.15.98:0/1207601064 wait complete. 2023-11-28T17:42:02.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.980+0000 7f44805de640 1 Processor -- start 2023-11-28T17:42:02.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.981+0000 7f44805de640 1 -- start start 2023-11-28T17:42:02.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.981+0000 7f44805de640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f44780c4380 0x7f447814bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f44805de640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44780c4760 0x7f44781efb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f44805de640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f447815a0f0 0x7f447814cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f44805de640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44781566e0 con 0x7f447815a0f0 2023-11-28T17:42:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f44805de640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447814d850 con 0x7f44780c4760 2023-11-28T17:42:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f44805de640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4478156120 con 0x7f44780c4380 2023-11-28T17:42:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f447eb7b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f447815a0f0 0x7f447814cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f447eb7b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f447815a0f0 0x7f447814cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50472/0 (socket says 172.21.15.98:50472) 2023-11-28T17:42:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f447eb7b640 1 -- 172.21.15.98:0/1101987694 learned_addr learned my addr 172.21.15.98:0/1101987694 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f447e37a640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f44780c4380 0x7f447814bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f447db79640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44780c4760 0x7f44781efb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f447eb7b640 1 -- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f44780c4380 msgr2=0x7f447814bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f447eb7b640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f44780c4380 0x7f447814bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f447eb7b640 1 -- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44780c4760 msgr2=0x7f44781efb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f447eb7b640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44780c4760 0x7f44781efb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f447eb7b640 1 -- 172.21.15.98:0/1101987694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44700094d0 con 0x7f447815a0f0 2023-11-28T17:42:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.982+0000 7f447db79640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44780c4760 0x7f44781efb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:42:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.983+0000 7f447eb7b640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f447815a0f0 0x7f447814cd10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f44700090b0 tx=0x7f4470004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.983+0000 7f4476ffd640 1 -- 172.21.15.98:0/1101987694 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4470037bc0 con 0x7f447815a0f0 2023-11-28T17:42:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.983+0000 7f4476ffd640 1 -- 172.21.15.98:0/1101987694 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4470037d20 con 0x7f447815a0f0 2023-11-28T17:42:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.983+0000 7f4476ffd640 1 -- 172.21.15.98:0/1101987694 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4470041640 con 0x7f447815a0f0 2023-11-28T17:42:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.983+0000 7f44805de640 1 -- 172.21.15.98:0/1101987694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4478156400 con 0x7f447815a0f0 2023-11-28T17:42:02.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.983+0000 7f44805de640 1 -- 172.21.15.98:0/1101987694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44781ef600 con 0x7f447815a0f0 2023-11-28T17:42:02.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.984+0000 7f44805de640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44781441d0 0x7f44781f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.984+0000 7f44805de640 1 -- 172.21.15.98:0/1101987694 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44781445b0 con 0x7f44781441d0 2023-11-28T17:42:02.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.985+0000 7f447e37a640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44781441d0 0x7f44781f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.985+0000 7f4476ffd640 1 -- 172.21.15.98:0/1101987694 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f44700417a0 con 0x7f447815a0f0 2023-11-28T17:42:02.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.985+0000 7f447e37a640 1 -- 172.21.15.98:0/1101987694 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f44781ef1a0 con 0x7f44781441d0 2023-11-28T17:42:02.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.986+0000 7f447e37a640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44781441d0 0x7f44781f2710 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4468001ab0 tx=0x7f4468001f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:02.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.986+0000 7f4476ffd640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f446406b2a0 0x7f446406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.986+0000 7f4476ffd640 1 -- 172.21.15.98:0/1101987694 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f447003c070 con 0x7f447815a0f0 2023-11-28T17:42:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.986+0000 7f447db79640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f446406b2a0 0x7f446406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.986+0000 7f4476ffd640 1 -- 172.21.15.98:0/1101987694 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446800e970 con 0x7f44781441d0 2023-11-28T17:42:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.987+0000 7f447db79640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f446406b2a0 0x7f446406d760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f44780cb990 tx=0x7f446c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.987+0000 7f4476ffd640 1 -- 172.21.15.98:0/1101987694 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f446800ead0 con 0x7f44781441d0 2023-11-28T17:42:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.987+0000 7f4476ffd640 1 -- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44781441d0 msgr2=0x7f44781f2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.987+0000 7f4476ffd640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44781441d0 0x7f44781f2710 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4468001ab0 tx=0x7f4468001f70 comp rx=0 tx=0).stop 2023-11-28T17:42:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.987+0000 7f4476ffd640 1 -- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44781441d0 msgr2=0x7f44781f2710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.987+0000 7f4476ffd640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44781441d0 0x7f44781f2710 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.994+0000 7f44805de640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44781f2c50 0x7f44781f3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:02.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.994+0000 7f44805de640 1 -- 172.21.15.98:0/1101987694 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4478156590 con 0x7f44781f2c50 2023-11-28T17:42:02.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.995+0000 7f447e37a640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44781f2c50 0x7f44781f3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:02.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.996+0000 7f447e37a640 1 -- 172.21.15.98:0/1101987694 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f44781574e0 con 0x7f44781f2c50 2023-11-28T17:42:02.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.996+0000 7f447e37a640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44781f2c50 0x7f44781f3060 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4468001ab0 tx=0x7f44680014a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:02.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.997+0000 7f4476ffd640 1 -- 172.21.15.98:0/1101987694 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446801d9c0 con 0x7f44781f2c50 2023-11-28T17:42:02.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.997+0000 7f4476ffd640 1 -- 172.21.15.98:0/1101987694 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4805 (secure 0 0 0) 0x7f4468021070 con 0x7f44781f2c50 2023-11-28T17:42:02.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.997+0000 7f4476ffd640 1 -- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44781f2c50 msgr2=0x7f44781f3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:02.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.998+0000 7f4476ffd640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44781f2c50 0x7f44781f3060 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4468001ab0 tx=0x7f44680014a0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.998+0000 7f4476ffd640 1 -- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44781f2c50 msgr2=0x7f44781f3060 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:02.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.998+0000 7f4476ffd640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44781f2c50 0x7f44781f3060 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4468001ab0 tx=0x7f44680014a0 comp rx=0 tx=0).stop 2023-11-28T17:42:02.999 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:42:02.999 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:42:03.000 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:42:03.000 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:42:03.000 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 8, 2023-11-28T17:42:03.000 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:42:03.000 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:42:03.000 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:42:03.001 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:42:03.001 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:03.001 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 65, 2023-11-28T17:42:03.001 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:03.001 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:42:03.001 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:42:03.001 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:03.001 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:03.001 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:03.001 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:03.001 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:03.001 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:03.002 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:03.002 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:03.002 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:03.002 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:03.002 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:03.002 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:42:03.002 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:42:03.002 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:03.002 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:03.002 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:03.002 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:03.002 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:03.003 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:03.003 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:03.003 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:03.003 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:03.003 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:03.003 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:03.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.003 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:42:03.003 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:42:03.003 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:42:03.003 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:42:03.003 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:42:03.003 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:42:03.004 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:03.004 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:03.004 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:42:03.004 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:42:03.004 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:42:03.004 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:42:03.004 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:42:03.004 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:42:03.004 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:42:03.004 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:03.004 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:42:03.004 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:03.005 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:03.005 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:03.005 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:03.005 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:03.005 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:03.005 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:03.005 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:03.006 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:03.006 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:03.006 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:03.006 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:42:03.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.007 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:42:03.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.007 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:42:03.007 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:42:03.007 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:03.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:03.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:03.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:42:03.007 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:03.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.008 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:03.008 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:42:03.008 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:03.008 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.008 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:03.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.008 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:42:03.008 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:42:03.008 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:03.008 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:03.008 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:03.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.009 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:42:03.009 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:42:03.009 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:03.009 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:03.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:03.009 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:42:03.009 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:03.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.010 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:03.010 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:42:03.010 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:03.010 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.010 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:03.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.011 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:42:03.011 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:42:03.011 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:03.011 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:03.011 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:03.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.011 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:42:03.011 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:42:03.011 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:03.011 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:03.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.012 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:03.012 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:42:03.012 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:03.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.012 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:03.012 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:42:03.012 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:03.012 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.012 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:03.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.012 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:42:03.013 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:42:03.013 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:03.013 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:03.013 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:03.013 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.013 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:03.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.013 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:42:03.013 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:42:03.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.013 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:03.013 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:03.013 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:03.014 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.014 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:03.014 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:42:03.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.014 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:03.014 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:03.014 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:42:03.014 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.014 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:03.014 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:42:03.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.014 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:03.015 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:03.015 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:42:03.015 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.015 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:03.015 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:42:03.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.015 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:03.015 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:03.015 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:03.015 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.015 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:03.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.015 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:42:03.016 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:42:03.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.998+0000 7f44805de640 1 -- 172.21.15.98:0/1101987694 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f446406b2a0 msgr2=0x7f446406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:03.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.998+0000 7f44805de640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f446406b2a0 0x7f446406d760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f44780cb990 tx=0x7f446c008040 comp rx=0 tx=0).stop 2023-11-28T17:42:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.998+0000 7f44805de640 1 -- 172.21.15.98:0/1101987694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f447815a0f0 msgr2=0x7f447814cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.998+0000 7f44805de640 1 --2- 172.21.15.98:0/1101987694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f447815a0f0 0x7f447814cd10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f44700090b0 tx=0x7f4470004730 comp rx=0 tx=0).stop 2023-11-28T17:42:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.998+0000 7f447eb7b640 1 -- 172.21.15.98:0/1101987694 reap_dead start 2023-11-28T17:42:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.998+0000 7f447eb7b640 1 -- 172.21.15.98:0/1101987694 reap_dead start 2023-11-28T17:42:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.998+0000 7f44805de640 1 -- 172.21.15.98:0/1101987694 shutdown_connections 2023-11-28T17:42:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.998+0000 7f44805de640 1 -- 172.21.15.98:0/1101987694 >> 172.21.15.98:0/1101987694 conn(0x7f447806c0f0 msgr2=0x7f44780c5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.999+0000 7f44805de640 1 -- 172.21.15.98:0/1101987694 shutdown_connections 2023-11-28T17:42:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:02.999+0000 7f44805de640 1 -- 172.21.15.98:0/1101987694 wait complete. 2023-11-28T17:42:03.019 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:42:03.019 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:42:03.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:42:03.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.233+0000 7ff6708bb640 1 Processor -- start 2023-11-28T17:42:03.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.233+0000 7ff6708bb640 1 -- start start 2023-11-28T17:42:03.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.235+0000 7ff6708bb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff66c159db0 0x7ff66c0c7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:03.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.235+0000 7ff6708bb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66c159080 con 0x7ff66c14fce0 2023-11-28T17:42:03.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.235+0000 7ff6708bb640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66c1565b0 con 0x7ff66c067de0 2023-11-28T17:42:03.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.235+0000 7ff6708bb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff66c0c7cc0 con 0x7ff66c159db0 2023-11-28T17:42:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.236+0000 7ff66bfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff66c14fce0 0x7ff66c155a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48144/0 (socket says 172.21.15.98:48144) 2023-11-28T17:42:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.236+0000 7ff66bfff640 1 -- 172.21.15.98:0/2763814479 learned_addr learned my addr 172.21.15.98:0/2763814479 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:42:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.236+0000 7ff66b7fe640 1 --2- 172.21.15.98:0/2763814479 >> v2:172.21.15.98:3300/0 conn(0x7ff66c159db0 0x7ff66c0c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:03.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.236+0000 7ff66affd640 1 -- 172.21.15.98:0/2763814479 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251115185 0 0) 0x7ff66c159080 con 0x7ff66c14fce0 2023-11-28T17:42:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.236+0000 7ff66b7fe640 1 -- 172.21.15.98:0/2763814479 >> v1:172.21.15.113:6789/0 conn(0x7ff66c067de0 legacy=0x7ff66c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.237+0000 7ff66b7fe640 1 -- 172.21.15.98:0/2763814479 >> v1:172.21.15.98:6789/0 conn(0x7ff66c14fce0 legacy=0x7ff66c155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:03.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.237+0000 7ff66b7fe640 1 -- 172.21.15.98:0/2763814479 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66c156890 con 0x7ff66c159db0 2023-11-28T17:42:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.238+0000 7ff66b7fe640 1 --2- 172.21.15.98:0/2763814479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c159db0 0x7ff66c0c7780 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009720 tx=0x7ff65c02fbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54d94c97b92d57f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.238+0000 7ff66affd640 1 -- 172.21.15.98:0/2763814479 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c030980 con 0x7ff66c159db0 2023-11-28T17:42:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.238+0000 7ff66affd640 1 -- 172.21.15.98:0/2763814479 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c030ae0 con 0x7ff66c159db0 2023-11-28T17:42:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.238+0000 7ff66affd640 1 -- 172.21.15.98:0/2763814479 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c030e70 con 0x7ff66c159db0 2023-11-28T17:42:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.239+0000 7ff6708bb640 1 -- 172.21.15.98:0/2763814479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c159db0 msgr2=0x7ff66c0c7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.239+0000 7ff6708bb640 1 --2- 172.21.15.98:0/2763814479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c159db0 0x7ff66c0c7780 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009720 tx=0x7ff65c02fbd0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.239+0000 7ff6708bb640 1 -- 172.21.15.98:0/2763814479 shutdown_connections 2023-11-28T17:42:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.239+0000 7ff6708bb640 1 --2- 172.21.15.98:0/2763814479 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c159db0 0x7ff66c0c7780 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.239+0000 7ff6708bb640 1 -- 172.21.15.98:0/2763814479 >> 172.21.15.98:0/2763814479 conn(0x7ff66c06e130 msgr2=0x7ff66c06e510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.239+0000 7ff6708bb640 1 -- 172.21.15.98:0/2763814479 shutdown_connections 2023-11-28T17:42:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.239+0000 7ff6708bb640 1 -- 172.21.15.98:0/2763814479 wait complete. 2023-11-28T17:42:03.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.241+0000 7ff6708bb640 1 Processor -- start 2023-11-28T17:42:03.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.241+0000 7ff6708bb640 1 -- start start 2023-11-28T17:42:03.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.242+0000 7ff6708bb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c067de0 0x7ff66c157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:03.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.242+0000 7ff6708bb640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c14fce0 0x7ff66c0ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:03.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.242+0000 7ff6708bb640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff66c159db0 0x7ff66c1550d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:03.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.242+0000 7ff6708bb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c157b50 con 0x7ff66c067de0 2023-11-28T17:42:03.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.242+0000 7ff6708bb640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c1eb310 con 0x7ff66c159db0 2023-11-28T17:42:03.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.242+0000 7ff6708bb640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c1eb5f0 con 0x7ff66c14fce0 2023-11-28T17:42:03.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.242+0000 7ff66b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c067de0 0x7ff66c157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:03.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.242+0000 7ff66b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c067de0 0x7ff66c157610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50486/0 (socket says 172.21.15.98:50486) 2023-11-28T17:42:03.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.242+0000 7ff66b7fe640 1 -- 172.21.15.98:0/3661621162 learned_addr learned my addr 172.21.15.98:0/3661621162 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:03.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.243+0000 7ff66b7fe640 1 -- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c14fce0 msgr2=0x7ff66c0ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:03.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.243+0000 7ff66b7fe640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c14fce0 0x7ff66c0ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.243+0000 7ff66b7fe640 1 -- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff66c159db0 msgr2=0x7ff66c1550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:03.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.243+0000 7ff66b7fe640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff66c159db0 0x7ff66c1550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.243+0000 7ff66b7fe640 1 -- 172.21.15.98:0/3661621162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c009400 con 0x7ff66c067de0 2023-11-28T17:42:03.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.243+0000 7ff66b7fe640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c067de0 0x7ff66c157610 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009720 tx=0x7ff65c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:03.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.243+0000 7ff668ff9640 1 -- 172.21.15.98:0/3661621162 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c030920 con 0x7ff66c067de0 2023-11-28T17:42:03.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.243+0000 7ff668ff9640 1 -- 172.21.15.98:0/3661621162 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c038d00 con 0x7ff66c067de0 2023-11-28T17:42:03.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.243+0000 7ff6708bb640 1 -- 172.21.15.98:0/3661621162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff66c1eb8d0 con 0x7ff66c067de0 2023-11-28T17:42:03.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.243+0000 7ff668ff9640 1 -- 172.21.15.98:0/3661621162 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c043620 con 0x7ff66c067de0 2023-11-28T17:42:03.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.244+0000 7ff6708bb640 1 -- 172.21.15.98:0/3661621162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff66c1ebe90 con 0x7ff66c067de0 2023-11-28T17:42:03.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.244+0000 7ff6708bb640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c153ac0 0x7ff66c153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:03.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.244+0000 7ff6708bb640 1 -- 172.21.15.98:0/3661621162 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c066cc0 con 0x7ff66c153ac0 2023-11-28T17:42:03.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.245+0000 7ff663fff640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c153ac0 0x7ff66c153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:03.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.246+0000 7ff668ff9640 1 -- 172.21.15.98:0/3661621162 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff65c043780 con 0x7ff66c067de0 2023-11-28T17:42:03.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.246+0000 7ff663fff640 1 -- 172.21.15.98:0/3661621162 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff66c1595b0 con 0x7ff66c153ac0 2023-11-28T17:42:03.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.246+0000 7ff663fff640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c153ac0 0x7ff66c153ee0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff66c070a20 tx=0x7ff65400c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:03.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.247+0000 7ff668ff9640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff64c06b2a0 0x7ff64c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:03.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.247+0000 7ff668ff9640 1 -- 172.21.15.98:0/3661621162 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff65c0b02e0 con 0x7ff66c067de0 2023-11-28T17:42:03.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.247+0000 7ff66bfff640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff64c06b2a0 0x7ff64c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:03.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.247+0000 7ff668ff9640 1 -- 172.21.15.98:0/3661621162 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff654005480 con 0x7ff66c153ac0 2023-11-28T17:42:03.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.248+0000 7ff668ff9640 1 -- 172.21.15.98:0/3661621162 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7ff654003610 con 0x7ff66c153ac0 2023-11-28T17:42:03.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.248+0000 7ff668ff9640 1 -- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c153ac0 msgr2=0x7ff66c153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:03.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.248+0000 7ff668ff9640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c153ac0 0x7ff66c153ee0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff66c070a20 tx=0x7ff65400c900 comp rx=0 tx=0).stop 2023-11-28T17:42:03.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.248+0000 7ff668ff9640 1 -- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c153ac0 msgr2=0x7ff66c153ee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:03.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.248+0000 7ff668ff9640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c153ac0 0x7ff66c153ee0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff66c070a20 tx=0x7ff65400c900 comp rx=0 tx=0).stop 2023-11-28T17:42:03.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.248+0000 7ff66bfff640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff64c06b2a0 0x7ff64c06d760 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff664000f80 tx=0x7ff664007c80 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:03.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.255+0000 7ff6708bb640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c13ab40 0x7ff66c13af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:03.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.255+0000 7ff6708bb640 1 -- 172.21.15.98:0/3661621162 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c1561a0 con 0x7ff66c13ab40 2023-11-28T17:42:03.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.255+0000 7ff663fff640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c13ab40 0x7ff66c13af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:03.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.256+0000 7ff663fff640 1 -- 172.21.15.98:0/3661621162 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff66c159890 con 0x7ff66c13ab40 2023-11-28T17:42:03.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.256+0000 7ff663fff640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c13ab40 0x7ff66c13af20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff654012000 tx=0x7ff6540012b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:03.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.257+0000 7ff668ff9640 1 -- 172.21.15.98:0/3661621162 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65401add0 con 0x7ff66c13ab40 2023-11-28T17:42:03.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.258+0000 7ff668ff9640 1 -- 172.21.15.98:0/3661621162 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4805 (secure 0 0 0) 0x7ff6540223f0 con 0x7ff66c13ab40 2023-11-28T17:42:03.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.258+0000 7ff668ff9640 1 -- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c13ab40 msgr2=0x7ff66c13af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:03.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.258+0000 7ff668ff9640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c13ab40 0x7ff66c13af20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff654012000 tx=0x7ff6540012b0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.258+0000 7ff668ff9640 1 -- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c13ab40 msgr2=0x7ff66c13af20 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:03.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.258+0000 7ff668ff9640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff66c13ab40 0x7ff66c13af20 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff654012000 tx=0x7ff6540012b0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.259 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:42:03.259 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:42:03.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:42:03.259 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:42:03.259 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 8, 2023-11-28T17:42:03.260 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:42:03.260 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:42:03.260 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:42:03.260 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:42:03.260 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:03.260 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 65, 2023-11-28T17:42:03.260 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:03.260 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:42:03.260 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:42:03.260 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:03.260 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:03.260 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:03.260 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:03.261 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:03.261 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:03.261 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:03.261 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:03.261 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:03.261 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:03.261 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:03.261 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:42:03.261 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:42:03.261 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:03.261 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:03.261 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:03.262 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:03.262 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:03.262 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:03.262 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:03.262 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:03.262 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:03.262 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:03.262 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:03.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.262 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:42:03.262 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:42:03.262 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:42:03.263 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:42:03.263 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:42:03.263 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:42:03.263 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:03.263 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:03.263 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:42:03.263 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:42:03.263 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:42:03.263 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:42:03.263 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:42:03.263 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:42:03.263 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:42:03.263 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:03.264 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:42:03.264 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:03.264 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:03.264 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:03.264 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:03.264 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:03.264 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:03.264 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:03.264 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:03.264 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:03.264 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:03.264 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:03.265 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:42:03.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.265 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:42:03.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.265 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:42:03.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:42:03.265 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:03.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:03.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:03.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:42:03.265 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:03.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:03.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:42:03.266 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:03.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.267 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:03.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:42:03.267 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:42:03.267 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:03.267 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:03.267 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:03.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.268 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:42:03.268 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:42:03.268 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:03.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:03.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:03.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:42:03.268 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:03.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:03.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:42:03.269 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:03.269 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.269 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:03.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:42:03.269 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:42:03.269 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:03.269 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:03.269 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:03.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.269 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:42:03.270 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:42:03.270 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:03.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:03.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:03.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:42:03.270 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:03.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:03.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:42:03.271 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:03.271 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.271 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:03.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:42:03.271 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:42:03.271 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:03.271 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:03.272 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:03.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.272 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:03.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.272 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:42:03.272 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:42:03.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.272 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:03.272 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:03.272 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:03.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.272 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:03.272 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:42:03.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.273 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:03.273 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:03.273 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:42:03.273 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.273 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:03.273 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:42:03.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.273 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:03.273 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:03.273 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:03.273 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.273 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:03.274 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:42:03.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:03.274 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:03.274 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:03.274 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:03.274 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:03.274 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:03.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:03.274 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:42:03.274 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:42:03.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.258+0000 7ff6708bb640 1 -- 172.21.15.98:0/3661621162 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff64c06b2a0 msgr2=0x7ff64c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:03.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.258+0000 7ff6708bb640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff64c06b2a0 0x7ff64c06d760 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff664000f80 tx=0x7ff664007c80 comp rx=0 tx=0).stop 2023-11-28T17:42:03.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.258+0000 7ff6708bb640 1 -- 172.21.15.98:0/3661621162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c067de0 msgr2=0x7ff66c157610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:03.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.258+0000 7ff6708bb640 1 --2- 172.21.15.98:0/3661621162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff66c067de0 0x7ff66c157610 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009720 tx=0x7ff65c0046c0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.258+0000 7ff66bfff640 1 -- 172.21.15.98:0/3661621162 reap_dead start 2023-11-28T17:42:03.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.258+0000 7ff66bfff640 1 -- 172.21.15.98:0/3661621162 reap_dead start 2023-11-28T17:42:03.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.258+0000 7ff6708bb640 1 -- 172.21.15.98:0/3661621162 shutdown_connections 2023-11-28T17:42:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.258+0000 7ff6708bb640 1 -- 172.21.15.98:0/3661621162 >> 172.21.15.98:0/3661621162 conn(0x7ff66c06e130 msgr2=0x7ff66c0c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.258+0000 7ff6708bb640 1 -- 172.21.15.98:0/3661621162 shutdown_connections 2023-11-28T17:42:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.258+0000 7ff6708bb640 1 -- 172.21.15.98:0/3661621162 wait complete. 2023-11-28T17:42:03.277 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:42:03.278 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-11-28T17:42:03.278 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-11-28T17:42:03.278 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-11-28T17:42:03.278 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-11-28T17:42:03.279 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-28T17:42:03.279 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:42:03.306 INFO:tasks.ceph.mon.a:Stopped 2023-11-28T17:42:03.307 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-11-28T17:42:03.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:42:03.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.546+0000 7f55d8333640 1 Processor -- start 2023-11-28T17:42:03.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.546+0000 7f55d8333640 1 -- start start 2023-11-28T17:42:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.547+0000 7f55d8333640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d0157b60 0x7f55d015bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.547+0000 7f55d8333640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f55d00c4910 0x7f55d00c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:03.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.547+0000 7f55d8333640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d015b150 con 0x7f55d0151d80 2023-11-28T17:42:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.547+0000 7f55d8333640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d0157f90 con 0x7f55d00c4910 2023-11-28T17:42:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.547+0000 7f55d8333640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d015c150 con 0x7f55d0157b60 2023-11-28T17:42:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.547+0000 7f55d60cf640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f55d0151d80 legacy=0x7f55d00c5c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:42:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.547+0000 7f55d68d0640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f55d00c4910 msgr2=0x7f55d00c4da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.548+0000 7f55d68d0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f55d00c4910 0x7f55d00c4da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:03.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.548+0000 7f55d58ce640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d0157b60 0x7f55d015bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.548+0000 7f55d58ce640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d0157b60 0x7f55d015bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:49904/0 (socket says 172.21.15.98:49904) 2023-11-28T17:42:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.548+0000 7f55d58ce640 1 -- 172.21.15.98:0/2943873556 learned_addr learned my addr 172.21.15.98:0/2943873556 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.549+0000 7f55d58ce640 1 -- 172.21.15.98:0/2943873556 >> v2:172.21.15.98:3300/0 conn(0x7f55d00c4910 msgr2=0x7f55d00c4da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.549+0000 7f55d58ce640 1 --2- 172.21.15.98:0/2943873556 >> v2:172.21.15.98:3300/0 conn(0x7f55d00c4910 0x7f55d00c4da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.549+0000 7f55d58ce640 1 -- 172.21.15.98:0/2943873556 >> v1:172.21.15.98:6789/0 conn(0x7f55d0151d80 legacy=0x7f55d00c5c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:03.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.549+0000 7f55d58ce640 1 -- 172.21.15.98:0/2943873556 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d006e1f0 con 0x7f55d0157b60 2023-11-28T17:42:03.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.550+0000 7f55d58ce640 1 --2- 172.21.15.98:0/2943873556 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d0157b60 0x7f55d015bc10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009870 tx=0x7f55c402b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb06d0b60cc94a3c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:03.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.551+0000 7f55d50cd640 1 -- 172.21.15.98:0/2943873556 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c4030860 con 0x7f55d0157b60 2023-11-28T17:42:03.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.551+0000 7f55d50cd640 1 -- 172.21.15.98:0/2943873556 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c40309c0 con 0x7f55d0157b60 2023-11-28T17:42:03.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.551+0000 7f55d50cd640 1 -- 172.21.15.98:0/2943873556 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c4030d50 con 0x7f55d0157b60 2023-11-28T17:42:03.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.551+0000 7f55d8333640 1 -- 172.21.15.98:0/2943873556 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d0157b60 msgr2=0x7f55d015bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:03.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.551+0000 7f55d8333640 1 --2- 172.21.15.98:0/2943873556 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d0157b60 0x7f55d015bc10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009870 tx=0x7f55c402b9a0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.551+0000 7f55d8333640 1 -- 172.21.15.98:0/2943873556 shutdown_connections 2023-11-28T17:42:03.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.551+0000 7f55d8333640 1 --2- 172.21.15.98:0/2943873556 >> v2:172.21.15.98:3300/0 conn(0x7f55d00c4910 0x7f55d00c4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.551+0000 7f55d8333640 1 --2- 172.21.15.98:0/2943873556 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d0157b60 0x7f55d015bc10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.551+0000 7f55d8333640 1 -- 172.21.15.98:0/2943873556 >> 172.21.15.98:0/2943873556 conn(0x7f55d006a4d0 msgr2=0x7f55d006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:03.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.552+0000 7f55d8333640 1 -- 172.21.15.98:0/2943873556 shutdown_connections 2023-11-28T17:42:03.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.552+0000 7f55d8333640 1 -- 172.21.15.98:0/2943873556 wait complete. 2023-11-28T17:42:03.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.553+0000 7f55d8333640 1 Processor -- start 2023-11-28T17:42:03.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.553+0000 7f55d8333640 1 -- start start 2023-11-28T17:42:03.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.554+0000 7f55d8333640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55d00c4910 0x7f55d014f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:03.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.554+0000 7f55d8333640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d0151d80 0x7f55d01569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:03.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.554+0000 7f55d8333640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d0157b60 0x7f55d014e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:03.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.554+0000 7f55d8333640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d01f08f0 con 0x7f55d0151d80 2023-11-28T17:42:03.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.554+0000 7f55d8333640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d0146a20 con 0x7f55d00c4910 2023-11-28T17:42:03.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.554+0000 7f55d8333640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d01efc70 con 0x7f55d0157b60 2023-11-28T17:42:03.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.554+0000 7f55d58ce640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d0151d80 msgr2=0x7f55d01569e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:03.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.554+0000 7f55d58ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d0151d80 0x7f55d01569e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:03.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.555+0000 7f55d60cf640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55d00c4910 0x7f55d014f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:03.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.555+0000 7f55d68d0640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d0157b60 0x7f55d014e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:03.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.555+0000 7f55d60cf640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55d00c4910 0x7f55d014f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:52664/0 (socket says 172.21.15.98:52664) 2023-11-28T17:42:03.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.555+0000 7f55d60cf640 1 -- 172.21.15.98:0/1488491871 learned_addr learned my addr 172.21.15.98:0/1488491871 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:03.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.556+0000 7f55d60cf640 1 -- 172.21.15.98:0/1488491871 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d0157b60 msgr2=0x7f55d014e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:03.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.556+0000 7f55d60cf640 1 --2- 172.21.15.98:0/1488491871 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d0157b60 0x7f55d014e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.556+0000 7f55d60cf640 1 -- 172.21.15.98:0/1488491871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d0151d80 msgr2=0x7f55d01569e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:42:03.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.556+0000 7f55d60cf640 1 --2- 172.21.15.98:0/1488491871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d0151d80 0x7f55d01569e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.556+0000 7f55d60cf640 1 -- 172.21.15.98:0/1488491871 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c40094d0 con 0x7f55d00c4910 2023-11-28T17:42:03.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.556+0000 7f55d60cf640 1 --2- 172.21.15.98:0/1488491871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55d00c4910 0x7f55d014f800 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f55cc00b580 tx=0x7f55cc00ba40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:03.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.556+0000 7f55c2ffd640 1 -- 172.21.15.98:0/1488491871 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc003dc0 con 0x7f55d00c4910 2023-11-28T17:42:03.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.556+0000 7f55c2ffd640 1 -- 172.21.15.98:0/1488491871 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc005070 con 0x7f55d00c4910 2023-11-28T17:42:03.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.556+0000 7f55d8333640 1 -- 172.21.15.98:0/1488491871 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d01eff50 con 0x7f55d00c4910 2023-11-28T17:42:03.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.556+0000 7f55c2ffd640 1 -- 172.21.15.98:0/1488491871 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc005420 con 0x7f55d00c4910 2023-11-28T17:42:03.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.557+0000 7f55d8333640 1 -- 172.21.15.98:0/1488491871 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d01f0510 con 0x7f55d00c4910 2023-11-28T17:42:03.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.558+0000 7f55d8333640 1 -- 172.21.15.98:0/1488491871 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d0067860 con 0x7f55d00c4910 2023-11-28T17:42:03.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.559+0000 7f55c2ffd640 1 -- 172.21.15.98:0/1488491871 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f55cc005580 con 0x7f55d00c4910 2023-11-28T17:42:03.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.560+0000 7f55c2ffd640 1 --2- 172.21.15.98:0/1488491871 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f55b406b250 0x7f55b406d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:03.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.560+0000 7f55c2ffd640 1 -- 172.21.15.98:0/1488491871 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f55cc08a310 con 0x7f55d00c4910 2023-11-28T17:42:03.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.561+0000 7f55d58ce640 1 --2- 172.21.15.98:0/1488491871 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f55b406b250 0x7f55b406d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:03.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.561+0000 7f55d58ce640 1 --2- 172.21.15.98:0/1488491871 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f55b406b250 0x7f55b406d710 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f55c4036000 tx=0x7f55c403b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:03.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.575+0000 7f55c2ffd640 1 -- 172.21.15.98:0/1488491871 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55cc055060 con 0x7f55d00c4910 2023-11-28T17:42:03.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.839+0000 7f55d8333640 1 -- 172.21.15.98:0/1488491871 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f55d0067bf0 con 0x7f55d00c4910 2023-11-28T17:42:03.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.841+0000 7f55c2ffd640 1 -- 172.21.15.98:0/1488491871 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7f55cc058680 con 0x7f55d00c4910 2023-11-28T17:42:03.842 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":66,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:42:03.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.844+0000 7f55d8333640 1 -- 172.21.15.98:0/1488491871 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f55b406b250 msgr2=0x7f55b406d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:03.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.844+0000 7f55d8333640 1 --2- 172.21.15.98:0/1488491871 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f55b406b250 0x7f55b406d710 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f55c4036000 tx=0x7f55c403b040 comp rx=0 tx=0).stop 2023-11-28T17:42:03.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.844+0000 7f55d8333640 1 -- 172.21.15.98:0/1488491871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55d00c4910 msgr2=0x7f55d014f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:03.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.844+0000 7f55d8333640 1 --2- 172.21.15.98:0/1488491871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55d00c4910 0x7f55d014f800 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f55cc00b580 tx=0x7f55cc00ba40 comp rx=0 tx=0).stop 2023-11-28T17:42:03.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.844+0000 7f55d8333640 1 -- 172.21.15.98:0/1488491871 shutdown_connections 2023-11-28T17:42:03.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.844+0000 7f55d8333640 1 --2- 172.21.15.98:0/1488491871 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55d0157b60 0x7f55d014e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.844+0000 7f55d8333640 1 --2- 172.21.15.98:0/1488491871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d0151d80 0x7f55d01569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.844+0000 7f55d8333640 1 --2- 172.21.15.98:0/1488491871 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f55b406b250 0x7f55b406d710 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.844+0000 7f55d8333640 1 --2- 172.21.15.98:0/1488491871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55d00c4910 0x7f55d014f800 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:03.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.844+0000 7f55d8333640 1 -- 172.21.15.98:0/1488491871 >> 172.21.15.98:0/1488491871 conn(0x7f55d006a4d0 msgr2=0x7f55d0150620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:03.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.844+0000 7f55d8333640 1 -- 172.21.15.98:0/1488491871 shutdown_connections 2023-11-28T17:42:03.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:03.844+0000 7f55d8333640 1 -- 172.21.15.98:0/1488491871 wait complete. 2023-11-28T17:42:06.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:42:07.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.087+0000 7f1d35c83640 1 Processor -- start 2023-11-28T17:42:07.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.088+0000 7f1d35c83640 1 -- start start 2023-11-28T17:42:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.089+0000 7f1d35c83640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1d300c81e0 0x7f1d3015a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.089+0000 7f1d35c83640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3015e190 con 0x7f1d3015dd70 2023-11-28T17:42:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.089+0000 7f1d35c83640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d30159c50 con 0x7f1d30154040 2023-11-28T17:42:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.089+0000 7f1d35c83640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d300c2ad0 con 0x7f1d300c81e0 2023-11-28T17:42:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.089+0000 7f1d2ffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f1d300c81e0 msgr2=0x7f1d3015a150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.089+0000 7f1d2ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1d300c81e0 0x7f1d3015a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.089+0000 7f1d26ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f1d3015dd70 legacy=0x7f1d300c7ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:42:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.090+0000 7f1d2f7fe640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f1d30154040 0x7f1d300c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.98:34286/0 (socket says 172.21.15.98:34286) 2023-11-28T17:42:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.090+0000 7f1d2f7fe640 1 -- 172.21.15.98:0/1528089032 learned_addr learned my addr 172.21.15.98:0/1528089032 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:42:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.090+0000 7f1d2effd640 1 -- 172.21.15.98:0/1528089032 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445016875 0 0) 0x7f1d30159c50 con 0x7f1d30154040 2023-11-28T17:42:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.090+0000 7f1d2effd640 1 -- 172.21.15.98:0/1528089032 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d1c003520 con 0x7f1d30154040 2023-11-28T17:42:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.091+0000 7f1d2effd640 1 -- 172.21.15.98:0/1528089032 <== mon.1 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2833792183 0 0) 0x7f1d1c003520 con 0x7f1d30154040 2023-11-28T17:42:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.091+0000 7f1d2effd640 1 -- 172.21.15.98:0/1528089032 >> v2:172.21.15.98:3300/0 conn(0x7f1d300c81e0 msgr2=0x7f1d3015a150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.091+0000 7f1d2effd640 1 --2- 172.21.15.98:0/1528089032 >> v2:172.21.15.98:3300/0 conn(0x7f1d300c81e0 0x7f1d3015a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.091+0000 7f1d2effd640 1 -- 172.21.15.98:0/1528089032 >> v1:172.21.15.98:6789/0 conn(0x7f1d3015dd70 legacy=0x7f1d300c7ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.091+0000 7f1d2effd640 1 -- 172.21.15.98:0/1528089032 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d300cab90 con 0x7f1d30154040 2023-11-28T17:42:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.091+0000 7f1d2effd640 1 -- 172.21.15.98:0/1528089032 <== mon.1 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f1d20001670 con 0x7f1d30154040 2023-11-28T17:42:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.091+0000 7f1d2effd640 1 -- 172.21.15.98:0/1528089032 >> v1:172.21.15.113:6789/0 conn(0x7f1d30154040 legacy=0x7f1d300c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.091+0000 7f1d2effd640 1 --2- 172.21.15.98:0/1528089032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d1c003970 0x7f1d1c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.091+0000 7f1d2effd640 1 --2- 172.21.15.98:0/1528089032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d1c0047c0 0x7f1d1c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.091+0000 7f1d2effd640 1 --2- 172.21.15.98:0/1528089032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d1c0051f0 0x7f1d1c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.091+0000 7f1d2effd640 1 -- 172.21.15.98:0/1528089032 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d300c2ad0 con 0x7f1d1c0047c0 2023-11-28T17:42:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.091+0000 7f1d2effd640 1 -- 172.21.15.98:0/1528089032 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c00c140 con 0x7f1d1c0051f0 2023-11-28T17:42:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.091+0000 7f1d2effd640 1 -- 172.21.15.98:0/1528089032 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c00c300 con 0x7f1d1c003970 2023-11-28T17:42:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.091+0000 7f1d26ffd640 1 -- 172.21.15.98:0/1528089032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d1c0047c0 msgr2=0x7f1d1c004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.091+0000 7f1d26ffd640 1 --2- 172.21.15.98:0/1528089032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d1c0047c0 0x7f1d1c004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.092+0000 7f1d2f7fe640 1 --2- 172.21.15.98:0/1528089032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d1c003970 0x7f1d1c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.092+0000 7f1d2ffff640 1 --2- 172.21.15.98:0/1528089032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d1c0051f0 0x7f1d1c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.092+0000 7f1d2f7fe640 1 -- 172.21.15.98:0/1528089032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d1c0051f0 msgr2=0x7f1d1c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.092+0000 7f1d2f7fe640 1 --2- 172.21.15.98:0/1528089032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d1c0051f0 0x7f1d1c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.092+0000 7f1d2f7fe640 1 -- 172.21.15.98:0/1528089032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d1c0047c0 msgr2=0x7f1d1c004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.092+0000 7f1d2f7fe640 1 --2- 172.21.15.98:0/1528089032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d1c0047c0 0x7f1d1c004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.092+0000 7f1d2f7fe640 1 -- 172.21.15.98:0/1528089032 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1c003520 con 0x7f1d1c003970 2023-11-28T17:42:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.092+0000 7f1d2ffff640 1 -- 172.21.15.98:0/1528089032 reap_dead start 2023-11-28T17:42:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.094+0000 7f1d2f7fe640 1 --2- 172.21.15.98:0/1528089032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d1c003970 0x7f1d1c003d70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000c0e0 tx=0x7f1d1c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c824ec0e3f7ac3d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.094+0000 7f1d2effd640 1 -- 172.21.15.98:0/1528089032 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d20005ad0 con 0x7f1d1c003970 2023-11-28T17:42:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.094+0000 7f1d2effd640 1 -- 172.21.15.98:0/1528089032 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d20005c30 con 0x7f1d1c003970 2023-11-28T17:42:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.094+0000 7f1d2effd640 1 -- 172.21.15.98:0/1528089032 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2002c7c0 con 0x7f1d1c003970 2023-11-28T17:42:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.095+0000 7f1d35c83640 1 -- 172.21.15.98:0/1528089032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d1c003970 msgr2=0x7f1d1c003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.095+0000 7f1d35c83640 1 --2- 172.21.15.98:0/1528089032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d1c003970 0x7f1d1c003d70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000c0e0 tx=0x7f1d1c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:42:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.095+0000 7f1d35c83640 1 -- 172.21.15.98:0/1528089032 shutdown_connections 2023-11-28T17:42:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.095+0000 7f1d35c83640 1 --2- 172.21.15.98:0/1528089032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d1c003970 0x7f1d1c003d70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.095+0000 7f1d35c83640 1 -- 172.21.15.98:0/1528089032 >> 172.21.15.98:0/1528089032 conn(0x7f1d3006a4d0 msgr2=0x7f1d3006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.095+0000 7f1d35c83640 1 -- 172.21.15.98:0/1528089032 shutdown_connections 2023-11-28T17:42:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.095+0000 7f1d35c83640 1 -- 172.21.15.98:0/1528089032 wait complete. 2023-11-28T17:42:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.097+0000 7f1d35c83640 1 Processor -- start 2023-11-28T17:42:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.097+0000 7f1d35c83640 1 -- start start 2023-11-28T17:42:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.098+0000 7f1d35c83640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d1c003970 0x7f1d3015b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.098+0000 7f1d35c83640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d3014a900 0x7f1d30143a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.098+0000 7f1d35c83640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d3014ad80 0x7f1d30143f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.098+0000 7f1d35c83640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d30158fb0 con 0x7f1d3014ad80 2023-11-28T17:42:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.098+0000 7f1d35c83640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3015c280 con 0x7f1d1c003970 2023-11-28T17:42:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.098+0000 7f1d35c83640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3015b9e0 con 0x7f1d3014a900 2023-11-28T17:42:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.098+0000 7f1d2ffff640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d3014ad80 msgr2=0x7f1d30143f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.098+0000 7f1d2ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d3014ad80 0x7f1d30143f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.098+0000 7f1d26ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d3014a900 0x7f1d30143a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.098+0000 7f1d2f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d1c003970 0x7f1d3015b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.098+0000 7f1d2f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d1c003970 0x7f1d3015b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:52682/0 (socket says 172.21.15.98:52682) 2023-11-28T17:42:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.098+0000 7f1d2f7fe640 1 -- 172.21.15.98:0/2658662582 learned_addr learned my addr 172.21.15.98:0/2658662582 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:07.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.099+0000 7f1d26ffd640 1 -- 172.21.15.98:0/2658662582 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d1c003970 msgr2=0x7f1d3015b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:07.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.099+0000 7f1d26ffd640 1 --2- 172.21.15.98:0/2658662582 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d1c003970 0x7f1d3015b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:07.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.099+0000 7f1d26ffd640 1 -- 172.21.15.98:0/2658662582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d3014ad80 msgr2=0x7f1d30143f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:42:07.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.099+0000 7f1d26ffd640 1 --2- 172.21.15.98:0/2658662582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d3014ad80 0x7f1d30143f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:07.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.099+0000 7f1d26ffd640 1 -- 172.21.15.98:0/2658662582 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3015bcc0 con 0x7f1d3014a900 2023-11-28T17:42:07.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.099+0000 7f1d26ffd640 1 --2- 172.21.15.98:0/2658662582 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d3014a900 0x7f1d30143a50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800b4b0 tx=0x7f1d1800b970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:07.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.100+0000 7f1d2cff9640 1 -- 172.21.15.98:0/2658662582 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d18003c10 con 0x7f1d3014a900 2023-11-28T17:42:07.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.100+0000 7f1d2cff9640 1 -- 172.21.15.98:0/2658662582 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d18003d90 con 0x7f1d3014a900 2023-11-28T17:42:07.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.100+0000 7f1d2cff9640 1 -- 172.21.15.98:0/2658662582 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d180052b0 con 0x7f1d3014a900 2023-11-28T17:42:07.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.100+0000 7f1d35c83640 1 -- 172.21.15.98:0/2658662582 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d3015bfa0 con 0x7f1d3014a900 2023-11-28T17:42:07.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.100+0000 7f1d35c83640 1 -- 172.21.15.98:0/2658662582 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d30159440 con 0x7f1d3014a900 2023-11-28T17:42:07.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.101+0000 7f1d35c83640 1 -- 172.21.15.98:0/2658662582 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d30067bf0 con 0x7f1d3014a900 2023-11-28T17:42:07.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.103+0000 7f1d2cff9640 1 -- 172.21.15.98:0/2658662582 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f1d18005410 con 0x7f1d3014a900 2023-11-28T17:42:07.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.104+0000 7f1d2cff9640 1 --2- 172.21.15.98:0/2658662582 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1d1006b250 0x7f1d1006d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:07.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.104+0000 7f1d2cff9640 1 -- 172.21.15.98:0/2658662582 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1d1808a070 con 0x7f1d3014a900 2023-11-28T17:42:07.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.104+0000 7f1d2f7fe640 1 --2- 172.21.15.98:0/2658662582 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1d1006b250 0x7f1d1006d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:07.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.105+0000 7f1d2f7fe640 1 --2- 172.21.15.98:0/2658662582 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1d1006b250 0x7f1d1006d710 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1d20000fd0 tx=0x7f1d20039040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:07.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.119+0000 7f1d2cff9640 1 -- 172.21.15.98:0/2658662582 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d18054fd0 con 0x7f1d3014a900 2023-11-28T17:42:07.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:42:07.155 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:42:07.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.381+0000 7f1d35c83640 1 -- 172.21.15.98:0/2658662582 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1d30074f60 con 0x7f1d3014a900 2023-11-28T17:42:07.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.382+0000 7f1d2cff9640 1 -- 172.21.15.98:0/2658662582 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7f1d180585f0 con 0x7f1d3014a900 2023-11-28T17:42:07.383 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":69,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:42:07.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.384+0000 7f1d35c83640 1 -- 172.21.15.98:0/2658662582 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1d1006b250 msgr2=0x7f1d1006d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:07.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.384+0000 7f1d35c83640 1 --2- 172.21.15.98:0/2658662582 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1d1006b250 0x7f1d1006d710 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1d20000fd0 tx=0x7f1d20039040 comp rx=0 tx=0).stop 2023-11-28T17:42:07.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.385+0000 7f1d35c83640 1 -- 172.21.15.98:0/2658662582 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d3014a900 msgr2=0x7f1d30143a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:07.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.385+0000 7f1d35c83640 1 --2- 172.21.15.98:0/2658662582 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d3014a900 0x7f1d30143a50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800b4b0 tx=0x7f1d1800b970 comp rx=0 tx=0).stop 2023-11-28T17:42:07.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.385+0000 7f1d35c83640 1 -- 172.21.15.98:0/2658662582 shutdown_connections 2023-11-28T17:42:07.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.385+0000 7f1d35c83640 1 --2- 172.21.15.98:0/2658662582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d3014ad80 0x7f1d30143f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:07.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.385+0000 7f1d35c83640 1 --2- 172.21.15.98:0/2658662582 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1d3014a900 0x7f1d30143a50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:07.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.385+0000 7f1d35c83640 1 --2- 172.21.15.98:0/2658662582 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1d1006b250 0x7f1d1006d710 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:07.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.385+0000 7f1d35c83640 1 --2- 172.21.15.98:0/2658662582 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d1c003970 0x7f1d3015b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:07.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.385+0000 7f1d35c83640 1 -- 172.21.15.98:0/2658662582 >> 172.21.15.98:0/2658662582 conn(0x7f1d3006a4d0 msgr2=0x7f1d301524c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:07.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.385+0000 7f1d35c83640 1 -- 172.21.15.98:0/2658662582 shutdown_connections 2023-11-28T17:42:07.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:07.385+0000 7f1d35c83640 1 -- 172.21.15.98:0/2658662582 wait complete. 2023-11-28T17:42:10.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:42:10.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:10.625+0000 7f2cda4c9640 1 Processor -- start 2023-11-28T17:42:10.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:10.625+0000 7f2cda4c9640 1 -- start start 2023-11-28T17:42:10.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:10.626+0000 7f2cda4c9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 0x7f2cd4157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:10.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:10.626+0000 7f2cda4c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd415b150 con 0x7f2cd40c4910 2023-11-28T17:42:10.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:10.626+0000 7f2cda4c9640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd406e1f0 con 0x7f2cd415bc10 2023-11-28T17:42:10.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:10.626+0000 7f2cda4c9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd406e4d0 con 0x7f2cd4151d80 2023-11-28T17:42:10.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:10.626+0000 7f2cd8a66640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2cd40c4910 legacy=0x7f2cd41624e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:42:10.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:10.626+0000 7f2cd3fff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 msgr2=0x7f2cd4157b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:10.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:10.627+0000 7f2cd3fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 0x7f2cd4157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:10.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:10.627+0000 7f2ccbfff640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f2cd415bc10 0x7f2cd40c5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.98:56010/0 (socket says 172.21.15.98:56010) 2023-11-28T17:42:10.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:10.627+0000 7f2ccbfff640 1 -- 172.21.15.98:0/1385939643 learned_addr learned my addr 172.21.15.98:0/1385939643 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:42:10.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:10.828+0000 7f2cd8a66640 1 -- 172.21.15.98:0/1385939643 >> v1:172.21.15.98:6789/0 conn(0x7f2cd40c4910 legacy=0x7f2cd41624e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:42:10.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:10.828+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 msgr2=0x7f2cd4157b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:10.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:10.828+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 0x7f2cd4157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:42:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:11.229+0000 7f2cd8a66640 1 -- 172.21.15.98:0/1385939643 >> v1:172.21.15.98:6789/0 conn(0x7f2cd40c4910 legacy=0x7f2cd41624e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:42:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:11.229+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 msgr2=0x7f2cd4157b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:11.229+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 0x7f2cd4157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:42:12.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:12.030+0000 7f2cd8a66640 1 -- 172.21.15.98:0/1385939643 >> v1:172.21.15.98:6789/0 conn(0x7f2cd40c4910 legacy=0x7f2cd41624e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:42:12.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:12.030+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 msgr2=0x7f2cd4157b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:12.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:12.030+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 0x7f2cd4157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:42:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.627+0000 7f2cd27fc640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 msgr2=0x7f2cd4157b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.627+0000 7f2cd27fc640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 0x7f2cd4157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.627+0000 7f2cd27fc640 1 -- 172.21.15.98:0/1385939643 >> v1:172.21.15.113:6789/0 conn(0x7f2cd415bc10 legacy=0x7f2cd40c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.627+0000 7f2cd27fc640 1 -- 172.21.15.98:0/1385939643 >> v1:172.21.15.98:6789/0 conn(0x7f2cd40c4910 legacy=0x7f2cd41624e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.627+0000 7f2cd27fc640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 0x7f2cb4005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.627+0000 7f2cd27fc640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.627+0000 7f2cd27fc640 1 -- 172.21.15.98:0/1385939643 --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd406e4d0 con 0x7f2cb4006230 2023-11-28T17:42:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.627+0000 7f2cd27fc640 1 -- 172.21.15.98:0/1385939643 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb4008d80 con 0x7f2cb40036c0 2023-11-28T17:42:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.628+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 0x7f2cb4005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.628+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.628+0000 7f2ccbfff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:42:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.628+0000 7f2ccbfff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:42:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.628+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:42:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.628+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 msgr2=0x7f2cb4005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:42:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.628+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 msgr2=0x7f2cb4005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:42:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.628+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 0x7f2cb4005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:42:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.628+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.629+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 0x7f2cb4005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.830+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.830+0000 7f2ccbfff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:42:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.830+0000 7f2ccbfff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:42:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.830+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:42:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.830+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:42:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.830+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 0x7f2cb4005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.831+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 msgr2=0x7f2cb4005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:42:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.831+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 msgr2=0x7f2cb4005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:42:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.831+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 0x7f2cb4005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:42:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:13.831+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 0x7f2cb4005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:42:14.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:14.231+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:14.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:14.232+0000 7f2ccbfff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:42:14.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:14.232+0000 7f2ccbfff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:42:14.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:14.232+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:42:14.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:14.232+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:42:14.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:14.232+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 0x7f2cb4005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:14.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:14.233+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 msgr2=0x7f2cb4005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:42:14.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:14.233+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 msgr2=0x7f2cb4005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:42:14.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:14.233+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 0x7f2cb4005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:42:14.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:14.233+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 0x7f2cb4005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:42:15.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:15.034+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:15.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:15.034+0000 7f2ccbfff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:42:15.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:15.034+0000 7f2ccbfff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:42:15.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:15.034+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:42:15.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:15.034+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:42:15.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:15.034+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 0x7f2cb4005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:15.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:15.035+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 msgr2=0x7f2cb4005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:42:15.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:15.035+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 msgr2=0x7f2cb4005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:42:15.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:15.035+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 0x7f2cb4005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:42:15.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:15.035+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 0x7f2cb4005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:42:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.627+0000 7f2cd27fc640 1 -- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 msgr2=0x7f2cb4005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.627+0000 7f2cd27fc640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb40036c0 0x7f2cb4005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.627+0000 7f2cd27fc640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.627+0000 7f2cd27fc640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.627+0000 7f2cd8a66640 1 -- 172.21.15.98:0/1385939643 reap_dead start 2023-11-28T17:42:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.627+0000 7f2cd27fc640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.627+0000 7f2cd27fc640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 0x7f2cb400b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.627+0000 7f2cd27fc640 1 -- 172.21.15.98:0/1385939643 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd406e7b0 con 0x7f2cd40c4910 2023-11-28T17:42:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.628+0000 7f2cd27fc640 1 -- 172.21.15.98:0/1385939643 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd406e4d0 con 0x7f2cd4151d80 2023-11-28T17:42:16.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.628+0000 7f2cd27fc640 1 -- 172.21.15.98:0/1385939643 --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb4008d80 con 0x7f2cb4006230 2023-11-28T17:42:16.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.628+0000 7f2ccbfff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 msgr2=0x7f2cb400b150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:16.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.628+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 0x7f2cb400b150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:16.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.628+0000 7f2cd8a66640 1 -- 172.21.15.98:0/1385939643 >> v1:172.21.15.98:6789/0 conn(0x7f2cd40c4910 legacy=0x7f2cb400d6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:42:16.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.628+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:16.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.629+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:42:16.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.629+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:42:16.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.629+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:42:16.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.629+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:16.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.829+0000 7f2ccbfff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 msgr2=0x7f2cb400b150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.829+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 0x7f2cb400b150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:42:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.829+0000 7f2cd8a66640 1 -- 172.21.15.98:0/1385939643 >> v1:172.21.15.98:6789/0 conn(0x7f2cd40c4910 legacy=0x7f2cb400d6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:42:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.830+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.831+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:42:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.831+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:42:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.831+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:42:16.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:16.831+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:42:17.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:17.230+0000 7f2ccbfff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 msgr2=0x7f2cb400b150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:17.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:17.230+0000 7f2cd8a66640 1 -- 172.21.15.98:0/1385939643 >> v1:172.21.15.98:6789/0 conn(0x7f2cd40c4910 legacy=0x7f2cb400d6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:42:17.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:17.230+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 0x7f2cb400b150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:42:17.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:17.232+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:17.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:17.233+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:42:17.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:17.233+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:42:17.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:17.233+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:42:17.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:17.233+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:42:18.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:18.031+0000 7f2ccbfff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 msgr2=0x7f2cb400b150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:18.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:18.031+0000 7f2cd8a66640 1 -- 172.21.15.98:0/1385939643 >> v1:172.21.15.98:6789/0 conn(0x7f2cd40c4910 legacy=0x7f2cb400d6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:42:18.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:18.031+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 0x7f2cb400b150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:42:18.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:18.035+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:18.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:18.035+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:42:18.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:18.035+0000 7f2cd3fff640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:42:18.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:18.035+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:42:18.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:18.035+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:42:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.628+0000 7f2cd27fc640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 msgr2=0x7f2cb40086d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.628+0000 7f2cd27fc640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.113:3300/0 conn(0x7f2cb4006230 0x7f2cb40086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.628+0000 7f2cd27fc640 1 -- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 msgr2=0x7f2cb400b150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.628+0000 7f2cd27fc640 1 --2- 172.21.15.98:0/1385939643 >> v2:172.21.15.98:3300/0 conn(0x7f2cd4151d80 0x7f2cb400b150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.628+0000 7f2cd27fc640 1 -- 172.21.15.98:0/1385939643 >> v1:172.21.15.98:6789/0 conn(0x7f2cd40c4910 legacy=0x7f2cb400d6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:19.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.628+0000 7f2cd27fc640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb400f9b0 0x7f2cb4012ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:19.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.628+0000 7f2cd27fc640 1 -- 172.21.15.98:0/1385939643 --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd415b150 con 0x7f2cb400ee90 2023-11-28T17:42:19.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.628+0000 7f2cd27fc640 1 -- 172.21.15.98:0/1385939643 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd406e4d0 con 0x7f2cb400f9b0 2023-11-28T17:42:19.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.629+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb400f9b0 0x7f2cb4012ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.630+0000 7f2ccbfff640 1 -- 172.21.15.98:0/1385939643 >> v1:172.21.15.113:6789/0 conn(0x7f2cb400ee90 legacy=0x7f2cb400f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.630+0000 7f2ccbfff640 1 -- 172.21.15.98:0/1385939643 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd406e7b0 con 0x7f2cb400f9b0 2023-11-28T17:42:19.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.633+0000 7f2ccbfff640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb400f9b0 0x7f2cb4012ee0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009010 tx=0x7f2cbc02c5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88bee9feb8ef36e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:19.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.634+0000 7f2cd37fe640 1 -- 172.21.15.98:0/1385939643 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc035410 con 0x7f2cb400f9b0 2023-11-28T17:42:19.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.634+0000 7f2cd37fe640 1 -- 172.21.15.98:0/1385939643 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cbc02f030 con 0x7f2cb400f9b0 2023-11-28T17:42:19.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.634+0000 7f2cd37fe640 1 -- 172.21.15.98:0/1385939643 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc0355d0 con 0x7f2cb400f9b0 2023-11-28T17:42:19.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.635+0000 7f2cda4c9640 1 -- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb400f9b0 msgr2=0x7f2cb4012ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:19.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.635+0000 7f2cda4c9640 1 --2- 172.21.15.98:0/1385939643 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cb400f9b0 0x7f2cb4012ee0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009010 tx=0x7f2cbc02c5a0 comp rx=0 tx=0).stop 2023-11-28T17:42:19.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.635+0000 7f2cd8a66640 1 -- 172.21.15.98:0/1385939643 reap_dead start 2023-11-28T17:42:19.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.635+0000 7f2cda4c9640 1 -- 172.21.15.98:0/1385939643 shutdown_connections 2023-11-28T17:42:19.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.635+0000 7f2cda4c9640 1 -- 172.21.15.98:0/1385939643 >> 172.21.15.98:0/1385939643 conn(0x7f2cd406a4d0 msgr2=0x7f2cd406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:19.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.636+0000 7f2cda4c9640 1 -- 172.21.15.98:0/1385939643 shutdown_connections 2023-11-28T17:42:19.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.636+0000 7f2cda4c9640 1 -- 172.21.15.98:0/1385939643 wait complete. 2023-11-28T17:42:19.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.641+0000 7f2cda4c9640 1 Processor -- start 2023-11-28T17:42:19.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.641+0000 7f2cda4c9640 1 -- start start 2023-11-28T17:42:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.643+0000 7f2cda4c9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cb400f9b0 0x7f2cd4156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.644+0000 7f2cda4c9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cd40c2070 0x7f2cd4156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.644+0000 7f2cda4c9640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cd414a910 0x7f2cd414ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.644+0000 7f2cda4c9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd40c3fe0 con 0x7f2cb400f9b0 2023-11-28T17:42:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.644+0000 7f2cda4c9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd41eb540 con 0x7f2cd40c2070 2023-11-28T17:42:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.644+0000 7f2cda4c9640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd4155980 con 0x7f2cd414a910 2023-11-28T17:42:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.644+0000 7f2cd3fff640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cb400f9b0 msgr2=0x7f2cd4156950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:19.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.644+0000 7f2cd3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cb400f9b0 0x7f2cd4156950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:19.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.644+0000 7f2cd8a66640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cd414a910 0x7f2cd414ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:19.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.644+0000 7f2ccbfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cd40c2070 0x7f2cd4156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:19.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.645+0000 7f2cd8a66640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cd414a910 0x7f2cd414ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:57652/0 (socket says 172.21.15.98:57652) 2023-11-28T17:42:19.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.645+0000 7f2cd8a66640 1 -- 172.21.15.98:0/1961532659 learned_addr learned my addr 172.21.15.98:0/1961532659 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:19.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.645+0000 7f2cd8a66640 1 -- 172.21.15.98:0/1961532659 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cd40c2070 msgr2=0x7f2cd4156e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:19.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.645+0000 7f2cd8a66640 1 --2- 172.21.15.98:0/1961532659 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cd40c2070 0x7f2cd4156e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:19.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.645+0000 7f2cd8a66640 1 -- 172.21.15.98:0/1961532659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cb400f9b0 msgr2=0x7f2cd4156950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:42:19.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.645+0000 7f2cd8a66640 1 --2- 172.21.15.98:0/1961532659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cb400f9b0 0x7f2cd4156950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:19.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.645+0000 7f2cd8a66640 1 -- 172.21.15.98:0/1961532659 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc00f040 con 0x7f2cd414a910 2023-11-28T17:42:19.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.646+0000 7f2cd8a66640 1 --2- 172.21.15.98:0/1961532659 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cd414a910 0x7f2cd414ad90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2cd40c7c60 tx=0x7f2ccc00bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:19.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.646+0000 7f2cd17fa640 1 -- 172.21.15.98:0/1961532659 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc00cdb0 con 0x7f2cd414a910 2023-11-28T17:42:19.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.646+0000 7f2cd17fa640 1 -- 172.21.15.98:0/1961532659 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ccc010070 con 0x7f2cd414a910 2023-11-28T17:42:19.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.646+0000 7f2cd17fa640 1 -- 172.21.15.98:0/1961532659 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc0155d0 con 0x7f2cd414a910 2023-11-28T17:42:19.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.646+0000 7f2cda4c9640 1 -- 172.21.15.98:0/1961532659 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd40c7d00 con 0x7f2cd414a910 2023-11-28T17:42:19.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.647+0000 7f2cda4c9640 1 -- 172.21.15.98:0/1961532659 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd4155b90 con 0x7f2cd414a910 2023-11-28T17:42:19.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.648+0000 7f2cda4c9640 1 -- 172.21.15.98:0/1961532659 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd4067bf0 con 0x7f2cd414a910 2023-11-28T17:42:19.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.650+0000 7f2cd17fa640 1 -- 172.21.15.98:0/1961532659 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2ccc015730 con 0x7f2cd414a910 2023-11-28T17:42:19.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.651+0000 7f2cd17fa640 1 --2- 172.21.15.98:0/1961532659 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2cb406c980 0x7f2cb406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:19.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.651+0000 7f2cd17fa640 1 -- 172.21.15.98:0/1961532659 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2ccc089890 con 0x7f2cd414a910 2023-11-28T17:42:19.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.651+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1961532659 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2cb406c980 0x7f2cb406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:19.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.652+0000 7f2cd3fff640 1 --2- 172.21.15.98:0/1961532659 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2cb406c980 0x7f2cb406ee40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4000c00 tx=0x7f2cc4003e30 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:19.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.665+0000 7f2cd17fa640 1 -- 172.21.15.98:0/1961532659 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ccc054610 con 0x7f2cd414a910 2023-11-28T17:42:19.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.934+0000 7f2cda4c9640 1 -- 172.21.15.98:0/1961532659 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2cd4067ff0 con 0x7f2cd414a910 2023-11-28T17:42:19.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.936+0000 7f2cd17fa640 1 -- 172.21.15.98:0/1961532659 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f2ccc01b020 con 0x7f2cd414a910 2023-11-28T17:42:19.937 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:42:19.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.938+0000 7f2cda4c9640 1 -- 172.21.15.98:0/1961532659 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2cb406c980 msgr2=0x7f2cb406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.938+0000 7f2cda4c9640 1 --2- 172.21.15.98:0/1961532659 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2cb406c980 0x7f2cb406ee40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4000c00 tx=0x7f2cc4003e30 comp rx=0 tx=0).stop 2023-11-28T17:42:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.939+0000 7f2cda4c9640 1 -- 172.21.15.98:0/1961532659 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cd414a910 msgr2=0x7f2cd414ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.939+0000 7f2cda4c9640 1 --2- 172.21.15.98:0/1961532659 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cd414a910 0x7f2cd414ad90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2cd40c7c60 tx=0x7f2ccc00bee0 comp rx=0 tx=0).stop 2023-11-28T17:42:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.939+0000 7f2cda4c9640 1 -- 172.21.15.98:0/1961532659 shutdown_connections 2023-11-28T17:42:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.939+0000 7f2cda4c9640 1 --2- 172.21.15.98:0/1961532659 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2cd414a910 0x7f2cd414ad90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.939+0000 7f2cda4c9640 1 --2- 172.21.15.98:0/1961532659 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2cb406c980 0x7f2cb406ee40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.939+0000 7f2cda4c9640 1 --2- 172.21.15.98:0/1961532659 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2cd40c2070 0x7f2cd4156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.939+0000 7f2cda4c9640 1 --2- 172.21.15.98:0/1961532659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cb400f9b0 0x7f2cd4156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.939+0000 7f2cda4c9640 1 -- 172.21.15.98:0/1961532659 >> 172.21.15.98:0/1961532659 conn(0x7f2cd406a4d0 msgr2=0x7f2cd414f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.939+0000 7f2cda4c9640 1 -- 172.21.15.98:0/1961532659 shutdown_connections 2023-11-28T17:42:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:19.939+0000 7f2cda4c9640 1 -- 172.21.15.98:0/1961532659 wait complete. 2023-11-28T17:42:19.959 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-11-28T17:42:19.959 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-11-28T17:42:19.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:42:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.172+0000 7fc9bdf89640 1 Processor -- start 2023-11-28T17:42:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.172+0000 7fc9bdf89640 1 -- start start 2023-11-28T17:42:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.174+0000 7fc9bdf89640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc9b8151be0 0x7fc9b80c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.174+0000 7fc9bdf89640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc9b81584d0 0x7fc9b806dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.174+0000 7fc9bdf89640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b80c8150 con 0x7fc9b80c5a60 2023-11-28T17:42:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.174+0000 7fc9bdf89640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b80c8850 con 0x7fc9b8151be0 2023-11-28T17:42:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.174+0000 7fc9bdf89640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b80c5f50 con 0x7fc9b81584d0 2023-11-28T17:42:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.174+0000 7fc9b77fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7fc9b8151be0 msgr2=0x7fc9b80c75a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.174+0000 7fc9b77fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc9b8151be0 0x7fc9b80c75a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.175+0000 7fc9b7fff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc9b81584d0 0x7fc9b806dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.175+0000 7fc9b7fff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc9b81584d0 0x7fc9b806dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:39296/0 (socket says 172.21.15.98:39296) 2023-11-28T17:42:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.175+0000 7fc9b7fff640 1 -- 172.21.15.98:0/3555312719 learned_addr learned my addr 172.21.15.98:0/3555312719 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.175+0000 7fc9b7fff640 1 -- 172.21.15.98:0/3555312719 >> v2:172.21.15.98:3300/0 conn(0x7fc9b8151be0 msgr2=0x7fc9b80c75a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.175+0000 7fc9b7fff640 1 --2- 172.21.15.98:0/3555312719 >> v2:172.21.15.98:3300/0 conn(0x7fc9b8151be0 0x7fc9b80c75a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.175+0000 7fc9b7fff640 1 -- 172.21.15.98:0/3555312719 >> v1:172.21.15.113:6789/0 conn(0x7fc9b80c5a60 legacy=0x7fc9b80c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.175+0000 7fc9b7fff640 1 -- 172.21.15.98:0/3555312719 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b80c7b90 con 0x7fc9b81584d0 2023-11-28T17:42:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.177+0000 7fc9b7fff640 1 --2- 172.21.15.98:0/3555312719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b81584d0 0x7fc9b806dd80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc9b80c1950 tx=0x7fc9b002bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cad79fb8f93a31a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.177+0000 7fc9b6ffd640 1 -- 172.21.15.98:0/3555312719 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b002cb30 con 0x7fc9b81584d0 2023-11-28T17:42:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.177+0000 7fc9b6ffd640 1 -- 172.21.15.98:0/3555312719 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b002cc90 con 0x7fc9b81584d0 2023-11-28T17:42:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.177+0000 7fc9b6ffd640 1 -- 172.21.15.98:0/3555312719 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b00343f0 con 0x7fc9b81584d0 2023-11-28T17:42:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.178+0000 7fc9bdf89640 1 -- 172.21.15.98:0/3555312719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b81584d0 msgr2=0x7fc9b806dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.178+0000 7fc9bdf89640 1 --2- 172.21.15.98:0/3555312719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b81584d0 0x7fc9b806dd80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc9b80c1950 tx=0x7fc9b002bf90 comp rx=0 tx=0).stop 2023-11-28T17:42:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.178+0000 7fc9bdf89640 1 -- 172.21.15.98:0/3555312719 shutdown_connections 2023-11-28T17:42:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.178+0000 7fc9bdf89640 1 --2- 172.21.15.98:0/3555312719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b81584d0 0x7fc9b806dd80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.178+0000 7fc9bdf89640 1 --2- 172.21.15.98:0/3555312719 >> v2:172.21.15.98:3300/0 conn(0x7fc9b8151be0 0x7fc9b80c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.178+0000 7fc9bdf89640 1 -- 172.21.15.98:0/3555312719 >> 172.21.15.98:0/3555312719 conn(0x7fc9b806c0f0 msgr2=0x7fc9b806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.178+0000 7fc9bdf89640 1 -- 172.21.15.98:0/3555312719 shutdown_connections 2023-11-28T17:42:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.178+0000 7fc9bdf89640 1 -- 172.21.15.98:0/3555312719 wait complete. 2023-11-28T17:42:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.180+0000 7fc9bdf89640 1 Processor -- start 2023-11-28T17:42:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.180+0000 7fc9bdf89640 1 -- start start 2023-11-28T17:42:20.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.181+0000 7fc9bdf89640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b80c5a60 0x7fc9b8155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.181+0000 7fc9bdf89640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b8151be0 0x7fc9b8154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.181+0000 7fc9bdf89640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b81584d0 0x7fc9b8155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.181+0000 7fc9bdf89640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b81ef260 con 0x7fc9b81584d0 2023-11-28T17:42:20.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.181+0000 7fc9bdf89640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b8154690 con 0x7fc9b80c5a60 2023-11-28T17:42:20.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.181+0000 7fc9bdf89640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b8152430 con 0x7fc9b8151be0 2023-11-28T17:42:20.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.181+0000 7fc9b7fff640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b81584d0 msgr2=0x7fc9b8155080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:20.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.181+0000 7fc9b7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b81584d0 0x7fc9b8155080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:20.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.181+0000 7fc9b77fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b80c5a60 0x7fc9b8155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:20.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.181+0000 7fc9affff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b8151be0 0x7fc9b8154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:20.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.181+0000 7fc9affff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b8151be0 0x7fc9b8154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:40506/0 (socket says 172.21.15.98:40506) 2023-11-28T17:42:20.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.181+0000 7fc9affff640 1 -- 172.21.15.98:0/4221137076 learned_addr learned my addr 172.21.15.98:0/4221137076 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:20.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.182+0000 7fc9affff640 1 -- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b80c5a60 msgr2=0x7fc9b8155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:20.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.182+0000 7fc9affff640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b80c5a60 0x7fc9b8155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.182+0000 7fc9affff640 1 -- 172.21.15.98:0/4221137076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b81584d0 msgr2=0x7fc9b8155080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:42:20.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.182+0000 7fc9affff640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b81584d0 0x7fc9b8155080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.182+0000 7fc9affff640 1 -- 172.21.15.98:0/4221137076 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b0009b30 con 0x7fc9b8151be0 2023-11-28T17:42:20.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.183+0000 7fc9affff640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b8151be0 0x7fc9b8154b40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc9a000c3c0 tx=0x7fc9a000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:20.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.183+0000 7fc9b4ff9640 1 -- 172.21.15.98:0/4221137076 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a00052d0 con 0x7fc9b8151be0 2023-11-28T17:42:20.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.183+0000 7fc9bdf89640 1 -- 172.21.15.98:0/4221137076 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b8152710 con 0x7fc9b8151be0 2023-11-28T17:42:20.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.183+0000 7fc9b4ff9640 1 -- 172.21.15.98:0/4221137076 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a0005450 con 0x7fc9b8151be0 2023-11-28T17:42:20.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.183+0000 7fc9b4ff9640 1 -- 172.21.15.98:0/4221137076 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0007220 con 0x7fc9b8151be0 2023-11-28T17:42:20.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.183+0000 7fc9bdf89640 1 -- 172.21.15.98:0/4221137076 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b8152cd0 con 0x7fc9b8151be0 2023-11-28T17:42:20.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.184+0000 7fc9bdf89640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b8152e60 0x7fc9b8149cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.184+0000 7fc9bdf89640 1 -- 172.21.15.98:0/4221137076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b80744d0 con 0x7fc9b8152e60 2023-11-28T17:42:20.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.185+0000 7fc9b77fe640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b8152e60 0x7fc9b8149cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:20.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.185+0000 7fc9b77fe640 1 -- 172.21.15.98:0/4221137076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9b8158920 con 0x7fc9b8152e60 2023-11-28T17:42:20.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.186+0000 7fc9b77fe640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b8152e60 0x7fc9b8149cb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8003980 tx=0x7fc9a8009b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:20.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.186+0000 7fc9b4ff9640 1 -- 172.21.15.98:0/4221137076 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fc9a0007380 con 0x7fc9b8151be0 2023-11-28T17:42:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.187+0000 7fc9b4ff9640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc99806b280 0x7fc99806d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.187+0000 7fc9b4ff9640 1 -- 172.21.15.98:0/4221137076 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc9a0088f60 con 0x7fc9b8151be0 2023-11-28T17:42:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.187+0000 7fc9b7fff640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc99806b280 0x7fc99806d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.187+0000 7fc9b4ff9640 1 -- 172.21.15.98:0/4221137076 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8012600 con 0x7fc9b8152e60 2023-11-28T17:42:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.188+0000 7fc9b4ff9640 1 -- 172.21.15.98:0/4221137076 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fc9a8012760 con 0x7fc9b8152e60 2023-11-28T17:42:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.188+0000 7fc9b4ff9640 1 -- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b8152e60 msgr2=0x7fc9b8149cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.188+0000 7fc9b4ff9640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b8152e60 0x7fc9b8149cb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8003980 tx=0x7fc9a8009b00 comp rx=0 tx=0).stop 2023-11-28T17:42:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.188+0000 7fc9b4ff9640 1 -- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b8152e60 msgr2=0x7fc9b8149cb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.188+0000 7fc9b4ff9640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b8152e60 0x7fc9b8149cb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.188+0000 7fc9b7fff640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc99806b280 0x7fc99806d740 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0035000 tx=0x7fc9b0030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:20.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.195+0000 7fc9bdf89640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b814a1f0 0x7fc9b80c0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.195+0000 7fc9bdf89640 1 -- 172.21.15.98:0/4221137076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b80c84e0 con 0x7fc9b814a1f0 2023-11-28T17:42:20.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.195+0000 7fc9b77fe640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b814a1f0 0x7fc9b80c0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:20.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.196+0000 7fc9b77fe640 1 -- 172.21.15.98:0/4221137076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc9b80c4e40 con 0x7fc9b814a1f0 2023-11-28T17:42:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.196+0000 7fc9b77fe640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b814a1f0 0x7fc9b80c0580 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8003980 tx=0x7fc9a80059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:20.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.197+0000 7fc9b4ff9640 1 -- 172.21.15.98:0/4221137076 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8005c40 con 0x7fc9b814a1f0 2023-11-28T17:42:20.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.198+0000 7fc9b4ff9640 1 -- 172.21.15.98:0/4221137076 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4796 (secure 0 0 0) 0x7fc9a8005da0 con 0x7fc9b814a1f0 2023-11-28T17:42:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.198+0000 7fc9b4ff9640 1 -- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b814a1f0 msgr2=0x7fc9b80c0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.198+0000 7fc9b4ff9640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b814a1f0 0x7fc9b80c0580 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8003980 tx=0x7fc9a80059d0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.198+0000 7fc9b4ff9640 1 -- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b814a1f0 msgr2=0x7fc9b80c0580 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:20.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.198+0000 7fc9b4ff9640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9b814a1f0 0x7fc9b80c0580 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8003980 tx=0x7fc9a80059d0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.201 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:42:20.201 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:42:20.201 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:42:20.201 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:42:20.201 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 10, 2023-11-28T17:42:20.202 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:42:20.202 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:42:20.202 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:42:20.202 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:20.202 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2023-11-28T17:42:20.202 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:20.203 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:42:20.203 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:42:20.203 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:20.203 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:20.203 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:20.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:20.203 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:20.204 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:20.204 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:20.205 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:20.205 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:20.205 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:20.205 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:20.205 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:42:20.205 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:42:20.206 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:20.206 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:20.206 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:20.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:20.206 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:20.206 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:20.207 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:20.207 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:20.207 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:20.207 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:20.207 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:20.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.207 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:42:20.208 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:42:20.208 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:42:20.208 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:42:20.208 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:42:20.208 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:42:20.208 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:20.208 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:20.208 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:42:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:42:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:42:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:42:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:42:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:42:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:42:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:42:20.210 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:20.210 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:20.210 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:20.210 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:20.210 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:20.210 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:20.210 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:20.210 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:42:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:42:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:42:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:42:20.212 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:20.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:20.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:20.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:42:20.212 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:20.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:20.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:42:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:42:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:42:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.214 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:42:20.214 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:42:20.214 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:20.214 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:20.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:20.215 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:42:20.215 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:20.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.216 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:20.216 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:42:20.216 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:20.216 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.216 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:20.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.216 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:42:20.216 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:42:20.217 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:20.217 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:20.217 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:20.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.217 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:42:20.217 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:42:20.217 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:20.218 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:20.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.218 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:20.218 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:42:20.218 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:42:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:42:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:42:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:42:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:42:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:42:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:42:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:42:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:42:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:42:20.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.223 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:20.223 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:20.223 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:42:20.223 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.223 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:20.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.223 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:42:20.223 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:42:20.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.198+0000 7fc9bdf89640 1 -- 172.21.15.98:0/4221137076 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc99806b280 msgr2=0x7fc99806d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:20.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.198+0000 7fc9bdf89640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc99806b280 0x7fc99806d740 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0035000 tx=0x7fc9b0030000 comp rx=0 tx=0).stop 2023-11-28T17:42:20.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.198+0000 7fc9bdf89640 1 -- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b8151be0 msgr2=0x7fc9b8154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:20.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.198+0000 7fc9bdf89640 1 --2- 172.21.15.98:0/4221137076 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9b8151be0 0x7fc9b8154b40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc9a000c3c0 tx=0x7fc9a000c880 comp rx=0 tx=0).stop 2023-11-28T17:42:20.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.198+0000 7fc9b7fff640 1 -- 172.21.15.98:0/4221137076 reap_dead start 2023-11-28T17:42:20.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.198+0000 7fc9b7fff640 1 -- 172.21.15.98:0/4221137076 reap_dead start 2023-11-28T17:42:20.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.198+0000 7fc9bdf89640 1 -- 172.21.15.98:0/4221137076 shutdown_connections 2023-11-28T17:42:20.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.198+0000 7fc9bdf89640 1 -- 172.21.15.98:0/4221137076 >> 172.21.15.98:0/4221137076 conn(0x7fc9b806c0f0 msgr2=0x7fc9b81568a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:20.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.199+0000 7fc9bdf89640 1 -- 172.21.15.98:0/4221137076 shutdown_connections 2023-11-28T17:42:20.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.199+0000 7fc9bdf89640 1 -- 172.21.15.98:0/4221137076 wait complete. 2023-11-28T17:42:20.227 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:42:20.227 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-11-28T17:42:20.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:42:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.428+0000 7f39135f1640 1 Processor -- start 2023-11-28T17:42:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.428+0000 7f39135f1640 1 -- start start 2023-11-28T17:42:20.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.429+0000 7f39135f1640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c151be0 0x7f390c0c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.429+0000 7f39135f1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f390c0c5a60 0x7f390c1584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.429+0000 7f39135f1640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f390c0c4730 0x7f390c0c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.430+0000 7f39135f1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f390c158a10 con 0x7f390c0c5a60 2023-11-28T17:42:20.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.430+0000 7f39135f1640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f390c0c8150 con 0x7f390c0c4730 2023-11-28T17:42:20.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.430+0000 7f39135f1640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390c0c8850 con 0x7f390c151be0 2023-11-28T17:42:20.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.430+0000 7f3910b8c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f390c0c5a60 msgr2=0x7f390c1584d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:20.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.430+0000 7f3910b8c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f390c0c5a60 0x7f390c1584d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:20.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.430+0000 7f391138d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c151be0 0x7f390c0c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:20.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.430+0000 7f3911b8e640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f390c0c4730 0x7f390c0c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:20.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.430+0000 7f391138d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c151be0 0x7f390c0c75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:40518/0 (socket says 172.21.15.98:40518) 2023-11-28T17:42:20.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.430+0000 7f391138d640 1 -- 172.21.15.98:0/4265489954 learned_addr learned my addr 172.21.15.98:0/4265489954 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:20.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.431+0000 7f391138d640 1 -- 172.21.15.98:0/4265489954 >> v2:172.21.15.113:3300/0 conn(0x7f390c0c4730 msgr2=0x7f390c0c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:20.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.431+0000 7f391138d640 1 --2- 172.21.15.98:0/4265489954 >> v2:172.21.15.113:3300/0 conn(0x7f390c0c4730 0x7f390c0c4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.431+0000 7f391138d640 1 -- 172.21.15.98:0/4265489954 >> v2:172.21.15.98:3300/0 conn(0x7f390c0c5a60 msgr2=0x7f390c1584d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:20.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.431+0000 7f391138d640 1 --2- 172.21.15.98:0/4265489954 >> v2:172.21.15.98:3300/0 conn(0x7f390c0c5a60 0x7f390c1584d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.431+0000 7f391138d640 1 -- 172.21.15.98:0/4265489954 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390c0c5f50 con 0x7f390c151be0 2023-11-28T17:42:20.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.431+0000 7f3911b8e640 1 --2- 172.21.15.98:0/4265489954 >> v2:172.21.15.113:3300/0 conn(0x7f390c0c4730 0x7f390c0c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:42:20.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.432+0000 7f391138d640 1 --2- 172.21.15.98:0/4265489954 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c151be0 0x7f390c0c75a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f390c070240 tx=0x7f390802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20ea5c34f0ee29bc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:20.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.433+0000 7f3903fff640 1 -- 172.21.15.98:0/4265489954 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3908030800 con 0x7f390c151be0 2023-11-28T17:42:20.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.433+0000 7f3903fff640 1 -- 172.21.15.98:0/4265489954 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3908030960 con 0x7f390c151be0 2023-11-28T17:42:20.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.433+0000 7f3903fff640 1 -- 172.21.15.98:0/4265489954 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3908030cf0 con 0x7f390c151be0 2023-11-28T17:42:20.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.433+0000 7f39135f1640 1 -- 172.21.15.98:0/4265489954 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c151be0 msgr2=0x7f390c0c75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:20.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.433+0000 7f39135f1640 1 --2- 172.21.15.98:0/4265489954 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c151be0 0x7f390c0c75a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f390c070240 tx=0x7f390802b940 comp rx=0 tx=0).stop 2023-11-28T17:42:20.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.433+0000 7f39135f1640 1 -- 172.21.15.98:0/4265489954 shutdown_connections 2023-11-28T17:42:20.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.433+0000 7f39135f1640 1 --2- 172.21.15.98:0/4265489954 >> v2:172.21.15.113:3300/0 conn(0x7f390c0c4730 0x7f390c0c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.433+0000 7f39135f1640 1 --2- 172.21.15.98:0/4265489954 >> v2:172.21.15.98:3300/0 conn(0x7f390c0c5a60 0x7f390c1584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.433+0000 7f39135f1640 1 --2- 172.21.15.98:0/4265489954 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c151be0 0x7f390c0c75a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.433+0000 7f39135f1640 1 -- 172.21.15.98:0/4265489954 >> 172.21.15.98:0/4265489954 conn(0x7f390c06c0f0 msgr2=0x7f390c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:20.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.433+0000 7f39135f1640 1 -- 172.21.15.98:0/4265489954 shutdown_connections 2023-11-28T17:42:20.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.433+0000 7f39135f1640 1 -- 172.21.15.98:0/4265489954 wait complete. 2023-11-28T17:42:20.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.435+0000 7f39135f1640 1 Processor -- start 2023-11-28T17:42:20.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.435+0000 7f39135f1640 1 -- start start 2023-11-28T17:42:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.436+0000 7f39135f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f390c0c4730 0x7f390c154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.436+0000 7f39135f1640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c0c5a60 0x7f390c152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.436+0000 7f39135f1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f390c151be0 0x7f390c14c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.436+0000 7f39135f1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c154280 con 0x7f390c0c4730 2023-11-28T17:42:20.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.436+0000 7f39135f1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c0cae40 con 0x7f390c151be0 2023-11-28T17:42:20.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.436+0000 7f39135f1640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390c1533d0 con 0x7f390c0c5a60 2023-11-28T17:42:20.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.436+0000 7f391138d640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f390c0c4730 msgr2=0x7f390c154fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:42:20.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.436+0000 7f391138d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f390c0c4730 0x7f390c154fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:20.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.437+0000 7f3910b8c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c0c5a60 0x7f390c152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:20.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.437+0000 7f3911b8e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f390c151be0 0x7f390c14c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:20.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.437+0000 7f3910b8c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c0c5a60 0x7f390c152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:40520/0 (socket says 172.21.15.98:40520) 2023-11-28T17:42:20.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.437+0000 7f3910b8c640 1 -- 172.21.15.98:0/1455494905 learned_addr learned my addr 172.21.15.98:0/1455494905 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:20.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.437+0000 7f3910b8c640 1 -- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f390c151be0 msgr2=0x7f390c14c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:20.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.437+0000 7f3910b8c640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f390c151be0 0x7f390c14c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.437+0000 7f3910b8c640 1 -- 172.21.15.98:0/1455494905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f390c0c4730 msgr2=0x7f390c154fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:42:20.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.437+0000 7f3910b8c640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f390c0c4730 0x7f390c154fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.438+0000 7f3910b8c640 1 -- 172.21.15.98:0/1455494905 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904009a70 con 0x7f390c0c5a60 2023-11-28T17:42:20.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.438+0000 7f3910b8c640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c0c5a60 0x7f390c152e90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00b500 tx=0x7f38fc00b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:20.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.438+0000 7f3901ffb640 1 -- 172.21.15.98:0/1455494905 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc003dc0 con 0x7f390c0c5a60 2023-11-28T17:42:20.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.438+0000 7f3901ffb640 1 -- 172.21.15.98:0/1455494905 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38fc005070 con 0x7f390c0c5a60 2023-11-28T17:42:20.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.438+0000 7f3901ffb640 1 -- 172.21.15.98:0/1455494905 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc005400 con 0x7f390c0c5a60 2023-11-28T17:42:20.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.438+0000 7f39135f1640 1 -- 172.21.15.98:0/1455494905 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39080094d0 con 0x7f390c0c5a60 2023-11-28T17:42:20.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.439+0000 7f39135f1640 1 -- 172.21.15.98:0/1455494905 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390c1f3340 con 0x7f390c0c5a60 2023-11-28T17:42:20.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.439+0000 7f39135f1640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c152390 0x7f390c14ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.439+0000 7f39135f1640 1 -- 172.21.15.98:0/1455494905 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390c0744d0 con 0x7f390c152390 2023-11-28T17:42:20.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.440+0000 7f391138d640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c152390 0x7f390c14ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:20.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.441+0000 7f391138d640 1 -- 172.21.15.98:0/1455494905 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f390c0c84e0 con 0x7f390c152390 2023-11-28T17:42:20.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.441+0000 7f391138d640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c152390 0x7f390c14ce60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f390c0c5240 tx=0x7f39080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:20.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.441+0000 7f3901ffb640 1 -- 172.21.15.98:0/1455494905 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f38fc005560 con 0x7f390c0c5a60 2023-11-28T17:42:20.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.442+0000 7f3901ffb640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f38f006b250 0x7f38f006d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.442+0000 7f3901ffb640 1 -- 172.21.15.98:0/1455494905 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f38fc089890 con 0x7f390c0c5a60 2023-11-28T17:42:20.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.442+0000 7f3901ffb640 1 -- 172.21.15.98:0/1455494905 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390803e070 con 0x7f390c152390 2023-11-28T17:42:20.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.442+0000 7f3911b8e640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f38f006b250 0x7f38f006d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:20.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.442+0000 7f3901ffb640 1 -- 172.21.15.98:0/1455494905 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f3908030ad0 con 0x7f390c152390 2023-11-28T17:42:20.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.443+0000 7f3901ffb640 1 -- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c152390 msgr2=0x7f390c14ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:20.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.443+0000 7f3901ffb640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c152390 0x7f390c14ce60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f390c0c5240 tx=0x7f39080046c0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.443+0000 7f3901ffb640 1 -- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c152390 msgr2=0x7f390c14ce60 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:20.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.443+0000 7f3901ffb640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c152390 0x7f390c14ce60 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f390c0c5240 tx=0x7f39080046c0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.443+0000 7f3911b8e640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f38f006b250 0x7f38f006d710 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3904011580 tx=0x7f390400b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:20.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.450+0000 7f39135f1640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c1543c0 0x7f390c1547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:20.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.450+0000 7f39135f1640 1 -- 172.21.15.98:0/1455494905 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390c0c7e70 con 0x7f390c1543c0 2023-11-28T17:42:20.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.451+0000 7f391138d640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c1543c0 0x7f390c1547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:20.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.451+0000 7f391138d640 1 -- 172.21.15.98:0/1455494905 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f390c151fc0 con 0x7f390c1543c0 2023-11-28T17:42:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.452+0000 7f391138d640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c1543c0 0x7f390c1547d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3908000c00 tx=0x7f3908009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.452+0000 7f3901ffb640 1 -- 172.21.15.98:0/1455494905 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3908006e10 con 0x7f390c1543c0 2023-11-28T17:42:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.453+0000 7f3901ffb640 1 -- 172.21.15.98:0/1455494905 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4964 (secure 0 0 0) 0x7f390804d070 con 0x7f390c1543c0 2023-11-28T17:42:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.453+0000 7f3901ffb640 1 -- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c1543c0 msgr2=0x7f390c1547d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.453+0000 7f3901ffb640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c1543c0 0x7f390c1547d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3908000c00 tx=0x7f3908009240 comp rx=0 tx=0).stop 2023-11-28T17:42:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.453+0000 7f3901ffb640 1 -- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c1543c0 msgr2=0x7f390c1547d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.453+0000 7f3901ffb640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c1543c0 0x7f390c1547d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:20.454 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:42:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:42:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:42:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:42:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 10, 2023-11-28T17:42:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:42:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:42:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:42:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2023-11-28T17:42:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:42:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:42:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:42:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:42:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:42:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:42:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:42:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:42:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:42:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:42:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:42:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:42:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:42:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:42:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:42:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:42:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:42:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:42:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:42:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:42:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:42:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:42:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:42:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:42:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:42:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:42:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:42:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:42:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:42:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:42:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:42:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:42:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:42:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:42:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:42:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:42:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:42:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:42:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:42:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:42:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:42:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:42:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:42:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:42:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 6 2023-11-28T17:42:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:42:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:42:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:42:20.471 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:42:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.453+0000 7f39135f1640 1 -- 172.21.15.98:0/1455494905 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f38f006b250 msgr2=0x7f38f006d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.453+0000 7f39135f1640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f38f006b250 0x7f38f006d710 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3904011580 tx=0x7f390400b040 comp rx=0 tx=0).stop 2023-11-28T17:42:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.453+0000 7f39135f1640 1 -- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c0c5a60 msgr2=0x7f390c152e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.453+0000 7f39135f1640 1 --2- 172.21.15.98:0/1455494905 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f390c0c5a60 0x7f390c152e90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00b500 tx=0x7f38fc00b900 comp rx=0 tx=0).stop 2023-11-28T17:42:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.453+0000 7f3911b8e640 1 -- 172.21.15.98:0/1455494905 reap_dead start 2023-11-28T17:42:20.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.453+0000 7f3911b8e640 1 -- 172.21.15.98:0/1455494905 reap_dead start 2023-11-28T17:42:20.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.453+0000 7f39135f1640 1 -- 172.21.15.98:0/1455494905 shutdown_connections 2023-11-28T17:42:20.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.453+0000 7f39135f1640 1 -- 172.21.15.98:0/1455494905 >> 172.21.15.98:0/1455494905 conn(0x7f390c06c0f0 msgr2=0x7f390c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:20.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.453+0000 7f39135f1640 1 -- 172.21.15.98:0/1455494905 shutdown_connections 2023-11-28T17:42:20.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:20.453+0000 7f39135f1640 1 -- 172.21.15.98:0/1455494905 wait complete. 2023-11-28T17:42:20.474 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:42:20.474 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-11-28T17:42:37.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:42:37.578 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:42:40.475 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-11-28T17:42:40.476 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-11-28T17:42:40.476 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-28T17:42:40.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-28T17:42:40.482 INFO:tasks.ceph.mon.a:Started 2023-11-28T17:42:40.482 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:42:40.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:42:40.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.726+0000 7f0d217e6640 1 Processor -- start 2023-11-28T17:42:40.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.726+0000 7f0d217e6640 1 -- start start 2023-11-28T17:42:40.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.728+0000 7f0d217e6640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d1c15a6d0 0x7f0d1c0cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:40.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.728+0000 7f0d217e6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1c15ab10 con 0x7f0d1c0cf7c0 2023-11-28T17:42:40.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.728+0000 7f0d217e6640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1c15d330 con 0x7f0d1c153bb0 2023-11-28T17:42:40.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.728+0000 7f0d217e6640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c15d120 con 0x7f0d1c15a6d0 2023-11-28T17:42:40.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.728+0000 7f0d20fe5640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0d1c0cf7c0 0x7f0d1c0b3760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57686/0 (socket says 172.21.15.98:57686) 2023-11-28T17:42:40.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.728+0000 7f0d20fe5640 1 -- 172.21.15.98:0/2098720967 learned_addr learned my addr 172.21.15.98:0/2098720967 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:42:40.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.728+0000 7f0d1b7fe640 1 --2- 172.21.15.98:0/2098720967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d1c15a6d0 0x7f0d1c0cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:40.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.729+0000 7f0d1affd640 1 -- 172.21.15.98:0/2098720967 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473929575 0 0) 0x7f0d1c15d330 con 0x7f0d1c153bb0 2023-11-28T17:42:40.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.729+0000 7f0d1affd640 1 -- 172.21.15.98:0/2098720967 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d04003520 con 0x7f0d1c153bb0 2023-11-28T17:42:40.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.729+0000 7f0d1b7fe640 1 -- 172.21.15.98:0/2098720967 >> v1:172.21.15.113:6789/0 conn(0x7f0d1c153bb0 legacy=0x7f0d1c0ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:40.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.729+0000 7f0d1b7fe640 1 -- 172.21.15.98:0/2098720967 >> v1:172.21.15.98:6789/0 conn(0x7f0d1c0cf7c0 legacy=0x7f0d1c0b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:40.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.729+0000 7f0d1b7fe640 1 -- 172.21.15.98:0/2098720967 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c15d610 con 0x7f0d1c15a6d0 2023-11-28T17:42:40.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.731+0000 7f0d1b7fe640 1 --2- 172.21.15.98:0/2098720967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d1c15a6d0 0x7f0d1c0cf280 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0d100097d0 tx=0x7f0d1002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c946e0bb73a5fa0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:40.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.731+0000 7f0d1affd640 1 -- 172.21.15.98:0/2098720967 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10030880 con 0x7f0d1c15a6d0 2023-11-28T17:42:40.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.731+0000 7f0d1affd640 1 -- 172.21.15.98:0/2098720967 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d100309e0 con 0x7f0d1c15a6d0 2023-11-28T17:42:40.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.731+0000 7f0d1affd640 1 -- 172.21.15.98:0/2098720967 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10030d70 con 0x7f0d1c15a6d0 2023-11-28T17:42:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.732+0000 7f0d217e6640 1 -- 172.21.15.98:0/2098720967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d1c15a6d0 msgr2=0x7f0d1c0cf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.732+0000 7f0d217e6640 1 --2- 172.21.15.98:0/2098720967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d1c15a6d0 0x7f0d1c0cf280 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0d100097d0 tx=0x7f0d1002b9c0 comp rx=0 tx=0).stop 2023-11-28T17:42:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.732+0000 7f0d217e6640 1 -- 172.21.15.98:0/2098720967 shutdown_connections 2023-11-28T17:42:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.732+0000 7f0d217e6640 1 --2- 172.21.15.98:0/2098720967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d1c15a6d0 0x7f0d1c0cf280 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.732+0000 7f0d217e6640 1 -- 172.21.15.98:0/2098720967 >> 172.21.15.98:0/2098720967 conn(0x7f0d1c071c00 msgr2=0x7f0d1c072010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.732+0000 7f0d217e6640 1 -- 172.21.15.98:0/2098720967 shutdown_connections 2023-11-28T17:42:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.732+0000 7f0d217e6640 1 -- 172.21.15.98:0/2098720967 wait complete. 2023-11-28T17:42:40.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.734+0000 7f0d217e6640 1 Processor -- start 2023-11-28T17:42:40.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.735+0000 7f0d217e6640 1 -- start start 2023-11-28T17:42:40.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.736+0000 7f0d217e6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d1c0cf7c0 0x7f0d1c14a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:40.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.736+0000 7f0d217e6640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d1c153bb0 0x7f0d1c0c5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:40.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.736+0000 7f0d217e6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d1c15a6d0 0x7f0d1c0c5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:40.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.736+0000 7f0d217e6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c159ad0 con 0x7f0d1c0cf7c0 2023-11-28T17:42:40.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.736+0000 7f0d217e6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c14a5e0 con 0x7f0d1c15a6d0 2023-11-28T17:42:40.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.736+0000 7f0d217e6640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c158f00 con 0x7f0d1c153bb0 2023-11-28T17:42:40.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.736+0000 7f0d1bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d1c0cf7c0 0x7f0d1c14a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:40.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.737+0000 7f0d1bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d1c0cf7c0 0x7f0d1c14a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60564/0 (socket says 172.21.15.98:60564) 2023-11-28T17:42:40.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.737+0000 7f0d1bfff640 1 -- 172.21.15.98:0/2434015671 learned_addr learned my addr 172.21.15.98:0/2434015671 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:40.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.737+0000 7f0d1b7fe640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d1c153bb0 0x7f0d1c0c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:40.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.737+0000 7f0d1bfff640 1 -- 172.21.15.98:0/2434015671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d1c0cf7c0 msgr2=0x7f0d1c14a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-11-28T17:42:40.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.737+0000 7f0d1bfff640 1 -- 172.21.15.98:0/2434015671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d1c0cf7c0 msgr2=0x7f0d1c14a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:42:40.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.737+0000 7f0d1bfff640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d1c0cf7c0 0x7f0d1c14a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:42:40.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.737+0000 7f0d1bfff640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d1c0cf7c0 0x7f0d1c14a0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:40.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.737+0000 7f0d20fe5640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d1c15a6d0 0x7f0d1c0c5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:40.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.738+0000 7f0d1b7fe640 1 -- 172.21.15.98:0/2434015671 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d1c15a6d0 msgr2=0x7f0d1c0c5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:40.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.738+0000 7f0d1b7fe640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d1c15a6d0 0x7f0d1c0c5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:40.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.738+0000 7f0d1b7fe640 1 -- 172.21.15.98:0/2434015671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d1c0cf7c0 msgr2=0x7f0d1c14a0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:42:40.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.738+0000 7f0d1b7fe640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d1c0cf7c0 0x7f0d1c14a0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:40.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.738+0000 7f0d1b7fe640 1 -- 172.21.15.98:0/2434015671 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d10009400 con 0x7f0d1c153bb0 2023-11-28T17:42:40.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.739+0000 7f0d1b7fe640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d1c153bb0 0x7f0d1c0c5840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0d10033000 tx=0x7f0d100046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:40.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.739+0000 7f0d18ff9640 1 -- 172.21.15.98:0/2434015671 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1003e070 con 0x7f0d1c153bb0 2023-11-28T17:42:40.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.739+0000 7f0d18ff9640 1 -- 172.21.15.98:0/2434015671 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d10039ad0 con 0x7f0d1c153bb0 2023-11-28T17:42:40.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.739+0000 7f0d18ff9640 1 -- 172.21.15.98:0/2434015671 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10039e80 con 0x7f0d1c153bb0 2023-11-28T17:42:40.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.740+0000 7f0d217e6640 1 -- 172.21.15.98:0/2434015671 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1c1531e0 con 0x7f0d1c153bb0 2023-11-28T17:42:40.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.740+0000 7f0d217e6640 1 -- 172.21.15.98:0/2434015671 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c159480 con 0x7f0d1c153bb0 2023-11-28T17:42:40.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.741+0000 7f0d217e6640 1 -- 172.21.15.98:0/2434015671 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1c074db0 con 0x7f0d1c153bb0 2023-11-28T17:42:40.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.743+0000 7f0d18ff9640 1 -- 172.21.15.98:0/2434015671 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0d1002f3d0 con 0x7f0d1c153bb0 2023-11-28T17:42:40.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.744+0000 7f0d18ff9640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d0806b2a0 0x7f0d0806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:40.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.744+0000 7f0d18ff9640 1 -- 172.21.15.98:0/2434015671 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0d100abd30 con 0x7f0d1c153bb0 2023-11-28T17:42:40.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.745+0000 7f0d1bfff640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d0806b2a0 0x7f0d0806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:40.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.745+0000 7f0d1bfff640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d0806b2a0 0x7f0d0806d760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0011e0 tx=0x7f0d0c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:40.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:40.757+0000 7f0d18ff9640 1 -- 172.21.15.98:0/2434015671 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d10076b00 con 0x7f0d1c153bb0 2023-11-28T17:42:41.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.127+0000 7f0d217e6640 1 -- 172.21.15.98:0/2434015671 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0d1c157eb0 con 0x7f0d1c153bb0 2023-11-28T17:42:41.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.128+0000 7f0d18ff9640 1 -- 172.21.15.98:0/2434015671 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7f0d10034020 con 0x7f0d1c153bb0 2023-11-28T17:42:41.129 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:42:41.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.131+0000 7f0d217e6640 1 -- 172.21.15.98:0/2434015671 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d0806b2a0 msgr2=0x7f0d0806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.131+0000 7f0d217e6640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d0806b2a0 0x7f0d0806d760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0011e0 tx=0x7f0d0c009040 comp rx=0 tx=0).stop 2023-11-28T17:42:41.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.131+0000 7f0d217e6640 1 -- 172.21.15.98:0/2434015671 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d1c153bb0 msgr2=0x7f0d1c0c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.131+0000 7f0d217e6640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d1c153bb0 0x7f0d1c0c5840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0d10033000 tx=0x7f0d100046c0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.131+0000 7f0d217e6640 1 -- 172.21.15.98:0/2434015671 shutdown_connections 2023-11-28T17:42:41.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.131+0000 7f0d217e6640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d0806b2a0 0x7f0d0806d760 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.131+0000 7f0d217e6640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d1c15a6d0 0x7f0d1c0c5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.131+0000 7f0d217e6640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d1c153bb0 0x7f0d1c0c5840 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.131+0000 7f0d217e6640 1 --2- 172.21.15.98:0/2434015671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d1c0cf7c0 0x7f0d1c14a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.131+0000 7f0d217e6640 1 -- 172.21.15.98:0/2434015671 >> 172.21.15.98:0/2434015671 conn(0x7f0d1c071c00 msgr2=0x7f0d1c0ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.131+0000 7f0d217e6640 1 -- 172.21.15.98:0/2434015671 shutdown_connections 2023-11-28T17:42:41.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.132+0000 7f0d217e6640 1 -- 172.21.15.98:0/2434015671 wait complete. 2023-11-28T17:42:41.152 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:42:41.152 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:42:41.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:42:41.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.342+0000 7f9d01467640 1 Processor -- start 2023-11-28T17:42:41.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.342+0000 7f9d01467640 1 -- start start 2023-11-28T17:42:41.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.343+0000 7f9d01467640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9cfc0c69b0 0x7f9cfc0c7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.343+0000 7f9d01467640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cfc0c7c60 0x7f9cfc164440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.343+0000 7f9d01467640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cfc0c80f0 con 0x7f9cfc153c80 2023-11-28T17:42:41.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.343+0000 7f9d01467640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cfc15e070 con 0x7f9cfc0c69b0 2023-11-28T17:42:41.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.343+0000 7f9d01467640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc15d230 con 0x7f9cfc0c7c60 2023-11-28T17:42:41.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.344+0000 7f9cfb7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9cfc0c69b0 0x7f9cfc0c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.344+0000 7f9cfbfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9cfc153c80 0x7f9cfc06b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57700/0 (socket says 172.21.15.98:57700) 2023-11-28T17:42:41.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.344+0000 7f9cfbfff640 1 -- 172.21.15.98:0/2838604386 learned_addr learned my addr 172.21.15.98:0/2838604386 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:42:41.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.344+0000 7f9cfb7fe640 1 -- 172.21.15.98:0/2838604386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cfc0c7c60 msgr2=0x7f9cfc164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:41.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.344+0000 7f9d00c66640 1 --2- 172.21.15.98:0/2838604386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cfc0c7c60 0x7f9cfc164440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.344+0000 7f9cfb7fe640 1 --2- 172.21.15.98:0/2838604386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cfc0c7c60 0x7f9cfc164440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.344+0000 7f9cfaffd640 1 -- 172.21.15.98:0/2838604386 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795206861 0 0) 0x7f9cfc0c80f0 con 0x7f9cfc153c80 2023-11-28T17:42:41.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.344+0000 7f9cfb7fe640 1 -- 172.21.15.98:0/2838604386 >> v1:172.21.15.98:6789/0 conn(0x7f9cfc153c80 legacy=0x7f9cfc06b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.344+0000 7f9cfb7fe640 1 -- 172.21.15.98:0/2838604386 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc15d4f0 con 0x7f9cfc0c69b0 2023-11-28T17:42:41.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.344+0000 7f9d00c66640 1 --2- 172.21.15.98:0/2838604386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cfc0c7c60 0x7f9cfc164440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:42:41.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.346+0000 7f9cfb7fe640 1 --2- 172.21.15.98:0/2838604386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc0c69b0 0x7f9cfc0c7720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9cf00097a0 tx=0x7f9cf002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d539c13a30b74a5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:41.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.346+0000 7f9cfaffd640 1 -- 172.21.15.98:0/2838604386 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0030610 con 0x7f9cfc0c69b0 2023-11-28T17:42:41.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.346+0000 7f9cfaffd640 1 -- 172.21.15.98:0/2838604386 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf0030770 con 0x7f9cfc0c69b0 2023-11-28T17:42:41.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.346+0000 7f9cfaffd640 1 -- 172.21.15.98:0/2838604386 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0030b00 con 0x7f9cfc0c69b0 2023-11-28T17:42:41.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.346+0000 7f9d01467640 1 -- 172.21.15.98:0/2838604386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc0c69b0 msgr2=0x7f9cfc0c7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.346+0000 7f9d01467640 1 --2- 172.21.15.98:0/2838604386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc0c69b0 0x7f9cfc0c7720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9cf00097a0 tx=0x7f9cf002b960 comp rx=0 tx=0).stop 2023-11-28T17:42:41.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.347+0000 7f9d01467640 1 -- 172.21.15.98:0/2838604386 shutdown_connections 2023-11-28T17:42:41.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.347+0000 7f9d01467640 1 --2- 172.21.15.98:0/2838604386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cfc0c7c60 0x7f9cfc164440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.347+0000 7f9d01467640 1 --2- 172.21.15.98:0/2838604386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc0c69b0 0x7f9cfc0c7720 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.347+0000 7f9d01467640 1 -- 172.21.15.98:0/2838604386 >> 172.21.15.98:0/2838604386 conn(0x7f9cfc0cf300 msgr2=0x7f9cfc0cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:41.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.347+0000 7f9d01467640 1 -- 172.21.15.98:0/2838604386 shutdown_connections 2023-11-28T17:42:41.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.347+0000 7f9d01467640 1 -- 172.21.15.98:0/2838604386 wait complete. 2023-11-28T17:42:41.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.349+0000 7f9d01467640 1 Processor -- start 2023-11-28T17:42:41.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.349+0000 7f9d01467640 1 -- start start 2023-11-28T17:42:41.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.351+0000 7f9d01467640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cfc0c69b0 0x7f9cfc1f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.351+0000 7f9d01467640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cfc0c7c60 0x7f9cfc152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.351+0000 7f9d01467640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc153c80 0x7f9cfc1ef650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.351+0000 7f9d01467640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc1ef1c0 con 0x7f9cfc153c80 2023-11-28T17:42:41.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.351+0000 7f9d01467640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc1515e0 con 0x7f9cfc0c7c60 2023-11-28T17:42:41.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.351+0000 7f9d01467640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc159f90 con 0x7f9cfc0c69b0 2023-11-28T17:42:41.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.351+0000 7f9d00c66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc153c80 0x7f9cfc1ef650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.351+0000 7f9d00c66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc153c80 0x7f9cfc1ef650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60580/0 (socket says 172.21.15.98:60580) 2023-11-28T17:42:41.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.351+0000 7f9d00c66640 1 -- 172.21.15.98:0/288047434 learned_addr learned my addr 172.21.15.98:0/288047434 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:41.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.351+0000 7f9cfbfff640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cfc0c69b0 0x7f9cfc1f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.352+0000 7f9cfb7fe640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cfc0c7c60 0x7f9cfc152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.352+0000 7f9d00c66640 1 -- 172.21.15.98:0/288047434 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cfc0c69b0 msgr2=0x7f9cfc1f2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.352+0000 7f9d00c66640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9cfc0c69b0 0x7f9cfc1f2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.352+0000 7f9d00c66640 1 -- 172.21.15.98:0/288047434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cfc0c7c60 msgr2=0x7f9cfc152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.352+0000 7f9d00c66640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cfc0c7c60 0x7f9cfc152de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.352+0000 7f9d00c66640 1 -- 172.21.15.98:0/288047434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf0009400 con 0x7f9cfc153c80 2023-11-28T17:42:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.352+0000 7f9d00c66640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc153c80 0x7f9cfc1ef650 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400cef0 tx=0x7f9cf400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.352+0000 7f9cf8ff9640 1 -- 172.21.15.98:0/288047434 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf400d840 con 0x7f9cfc153c80 2023-11-28T17:42:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.353+0000 7f9cf8ff9640 1 -- 172.21.15.98:0/288047434 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf400d9c0 con 0x7f9cfc153c80 2023-11-28T17:42:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.353+0000 7f9cf8ff9640 1 -- 172.21.15.98:0/288047434 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf400dd70 con 0x7f9cfc153c80 2023-11-28T17:42:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.353+0000 7f9d01467640 1 -- 172.21.15.98:0/288047434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cfc15a250 con 0x7f9cfc153c80 2023-11-28T17:42:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.353+0000 7f9d01467640 1 -- 172.21.15.98:0/288047434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cfc15a810 con 0x7f9cfc153c80 2023-11-28T17:42:41.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.354+0000 7f9d01467640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc156ed0 0x7f9cfc14b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.354+0000 7f9d01467640 1 -- 172.21.15.98:0/288047434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc068170 con 0x7f9cfc156ed0 2023-11-28T17:42:41.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.354+0000 7f9cfbfff640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc156ed0 0x7f9cfc14b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.355+0000 7f9cfbfff640 1 -- 172.21.15.98:0/288047434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9cfc15dd90 con 0x7f9cfc156ed0 2023-11-28T17:42:41.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.355+0000 7f9cfbfff640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc156ed0 0x7f9cfc14b700 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9cec001380 tx=0x7f9cec001840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:41.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.356+0000 7f9cf8ff9640 1 -- 172.21.15.98:0/288047434 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec010510 con 0x7f9cfc156ed0 2023-11-28T17:42:41.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.356+0000 7f9cf8ff9640 1 -- 172.21.15.98:0/288047434 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f9cf401e3b0 con 0x7f9cfc153c80 2023-11-28T17:42:41.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.357+0000 7f9cf8ff9640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9ce806b390 0x7f9ce806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.357+0000 7f9cf8ff9640 1 -- 172.21.15.98:0/288047434 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9cf408a670 con 0x7f9cfc153c80 2023-11-28T17:42:41.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.358+0000 7f9cf8ff9640 1 -- 172.21.15.98:0/288047434 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f9cec010670 con 0x7f9cfc156ed0 2023-11-28T17:42:41.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.358+0000 7f9cf8ff9640 1 -- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc156ed0 msgr2=0x7f9cfc14b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.358+0000 7f9cf8ff9640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc156ed0 0x7f9cfc14b700 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9cec001380 tx=0x7f9cec001840 comp rx=0 tx=0).stop 2023-11-28T17:42:41.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.358+0000 7f9cf8ff9640 1 -- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc156ed0 msgr2=0x7f9cfc14b700 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:41.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.358+0000 7f9cf8ff9640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc156ed0 0x7f9cfc14b700 secure :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9cec001380 tx=0x7f9cec001840 comp rx=0 tx=0).stop 2023-11-28T17:42:41.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.358+0000 7f9cfb7fe640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9ce806b390 0x7f9ce806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.359+0000 7f9cfb7fe640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9ce806b390 0x7f9ce806d850 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0036000 tx=0x7f9cf0008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.365+0000 7f9d01467640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc14bc40 0x7f9cfc1589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.365+0000 7f9d01467640 1 -- 172.21.15.98:0/288047434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc15dab0 con 0x7f9cfc14bc40 2023-11-28T17:42:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.365+0000 7f9cfbfff640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc14bc40 0x7f9cfc1589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.366+0000 7f9cfbfff640 1 -- 172.21.15.98:0/288047434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9cfc0cbb10 con 0x7f9cfc14bc40 2023-11-28T17:42:41.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.366+0000 7f9cfbfff640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc14bc40 0x7f9cfc1589c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9cec001380 tx=0x7f9cec001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.366+0000 7f9cf8ff9640 1 -- 172.21.15.98:0/288047434 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec021070 con 0x7f9cfc14bc40 2023-11-28T17:42:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.367+0000 7f9cf8ff9640 1 -- 172.21.15.98:0/288047434 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4467 (secure 0 0 0) 0x7f9cec014480 con 0x7f9cfc14bc40 2023-11-28T17:42:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.367+0000 7f9cf8ff9640 1 -- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc14bc40 msgr2=0x7f9cfc1589c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.367+0000 7f9cf8ff9640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc14bc40 0x7f9cfc1589c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9cec001380 tx=0x7f9cec001080 comp rx=0 tx=0).stop 2023-11-28T17:42:41.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.367+0000 7f9cf8ff9640 1 -- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc14bc40 msgr2=0x7f9cfc1589c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:41.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.367+0000 7f9cf8ff9640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc14bc40 0x7f9cfc1589c0 secure :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9cec001380 tx=0x7f9cec001080 comp rx=0 tx=0).stop 2023-11-28T17:42:41.369 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:42:41.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:42:41.369 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:42:41.369 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:42:41.369 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 14, 2023-11-28T17:42:41.369 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:42:41.370 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:42:41.370 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:42:41.370 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:42:41.370 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.370 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:42:41.370 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:41.370 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:42:41.370 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:42:41.370 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:41.370 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:41.370 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:41.370 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:41.371 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:41.371 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:41.371 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:41.371 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:41.371 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:41.371 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:41.371 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.371 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:42:41.371 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:42:41.371 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:41.371 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:41.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:41.371 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:41.371 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:41.372 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:41.372 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:41.372 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:41.372 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:41.372 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:41.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.372 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:42:41.372 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:42:41.372 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:42:41.372 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:42:41.372 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:42:41.373 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:42:41.373 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:41.373 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:41.373 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:42:41.373 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:42:41.373 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:42:41.373 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:42:41.373 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:42:41.373 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:42:41.373 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:42:41.373 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:41.373 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:42:41.373 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:41.373 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:41.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:41.374 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:41.374 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:41.374 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:41.374 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:41.374 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:41.374 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:41.374 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:41.374 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.374 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:42:41.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:42:41.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.375 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:42:41.375 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:42:41.375 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:41.375 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:41.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:41.375 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:42:41.375 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:41.376 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:42:41.376 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.376 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.376 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.376 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:42:41.376 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:42:41.376 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:41.376 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:41.376 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:41.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.376 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:42:41.376 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:42:41.377 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:41.377 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:41.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:41.377 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:42:41.377 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:41.377 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:42:41.377 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.377 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.377 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.378 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:42:41.378 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:42:41.378 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:41.378 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:41.378 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:41.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.378 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:42:41.378 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:42:41.378 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:41.378 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:41.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:41.379 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:42:41.379 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:41.379 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:42:41.379 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.379 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.379 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.379 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:42:41.379 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:42:41.379 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:41.380 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:41.380 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:41.380 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.380 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.380 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:42:41.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:42:41.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.380 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:41.380 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:41.380 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:41.380 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.380 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.381 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:42:41.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.381 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:41.381 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:41.381 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:42:41.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.381 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.381 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:42:41.381 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:42:41.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.367+0000 7f9d01467640 1 -- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9ce806b390 msgr2=0x7f9ce806d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.367+0000 7f9d01467640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9ce806b390 0x7f9ce806d850 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0036000 tx=0x7f9cf0008030 comp rx=0 tx=0).stop 2023-11-28T17:42:41.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.367+0000 7f9d01467640 1 -- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc153c80 msgr2=0x7f9cfc1ef650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.367+0000 7f9d01467640 1 --2- 172.21.15.98:0/288047434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc153c80 0x7f9cfc1ef650 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400cef0 tx=0x7f9cf400a710 comp rx=0 tx=0).stop 2023-11-28T17:42:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.367+0000 7f9d00c66640 1 -- 172.21.15.98:0/288047434 reap_dead start 2023-11-28T17:42:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.367+0000 7f9d00c66640 1 -- 172.21.15.98:0/288047434 reap_dead start 2023-11-28T17:42:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.367+0000 7f9d01467640 1 -- 172.21.15.98:0/288047434 shutdown_connections 2023-11-28T17:42:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.367+0000 7f9d01467640 1 -- 172.21.15.98:0/288047434 >> 172.21.15.98:0/288047434 conn(0x7f9cfc0cf300 msgr2=0x7f9cfc151f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.368+0000 7f9d01467640 1 -- 172.21.15.98:0/288047434 shutdown_connections 2023-11-28T17:42:41.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.368+0000 7f9d01467640 1 -- 172.21.15.98:0/288047434 wait complete. 2023-11-28T17:42:41.390 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:42:41.390 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:42:41.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:42:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.589+0000 7f3ab759e640 1 Processor -- start 2023-11-28T17:42:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.589+0000 7f3ab759e640 1 -- start start 2023-11-28T17:42:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.590+0000 7f3ab759e640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3ab8153c80 0x7f3ab806b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.591+0000 7f3ab759e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3ab80c69b0 0x7f3ab806be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.591+0000 7f3ab759e640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab815d230 con 0x7f3ab80c7720 2023-11-28T17:42:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.591+0000 7f3ab759e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab815e070 con 0x7f3ab80c69b0 2023-11-28T17:42:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.591+0000 7f3ab759e640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab815d4f0 con 0x7f3ab8153c80 2023-11-28T17:42:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.591+0000 7f3ab5d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3ab80c69b0 0x7f3ab806be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.591+0000 7f3ab5d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3ab80c69b0 0x7f3ab806be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60616/0 (socket says 172.21.15.98:60616) 2023-11-28T17:42:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.591+0000 7f3ab5d9b640 1 -- 172.21.15.98:0/1065629216 learned_addr learned my addr 172.21.15.98:0/1065629216 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.591+0000 7f3ab659c640 1 --2- 172.21.15.98:0/1065629216 >> v2:172.21.15.113:3300/0 conn(0x7f3ab8153c80 0x7f3ab806b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.592+0000 7f3ab5d9b640 1 -- 172.21.15.98:0/1065629216 >> v2:172.21.15.113:3300/0 conn(0x7f3ab8153c80 msgr2=0x7f3ab806b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.592+0000 7f3ab5d9b640 1 --2- 172.21.15.98:0/1065629216 >> v2:172.21.15.113:3300/0 conn(0x7f3ab8153c80 0x7f3ab806b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.592+0000 7f3ab5d9b640 1 -- 172.21.15.98:0/1065629216 >> v1:172.21.15.113:6789/0 conn(0x7f3ab80c7720 legacy=0x7f3ab80c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.592+0000 7f3ab5d9b640 1 -- 172.21.15.98:0/1065629216 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab815d7d0 con 0x7f3ab80c69b0 2023-11-28T17:42:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.593+0000 7f3ab5d9b640 1 --2- 172.21.15.98:0/1065629216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab80c69b0 0x7f3ab806be80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3ab806c630 tx=0x7f3aac02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6605ad6ae563512d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.593+0000 7f3ab559a640 1 -- 172.21.15.98:0/1065629216 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac02f850 con 0x7f3ab80c69b0 2023-11-28T17:42:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.593+0000 7f3ab559a640 1 -- 172.21.15.98:0/1065629216 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aac02f9b0 con 0x7f3ab80c69b0 2023-11-28T17:42:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.593+0000 7f3ab559a640 1 -- 172.21.15.98:0/1065629216 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac02fd40 con 0x7f3ab80c69b0 2023-11-28T17:42:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.594+0000 7f3ab759e640 1 -- 172.21.15.98:0/1065629216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab80c69b0 msgr2=0x7f3ab806be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.594+0000 7f3ab759e640 1 --2- 172.21.15.98:0/1065629216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab80c69b0 0x7f3ab806be80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3ab806c630 tx=0x7f3aac02ba30 comp rx=0 tx=0).stop 2023-11-28T17:42:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.594+0000 7f3ab759e640 1 -- 172.21.15.98:0/1065629216 shutdown_connections 2023-11-28T17:42:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.594+0000 7f3ab759e640 1 --2- 172.21.15.98:0/1065629216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab80c69b0 0x7f3ab806be80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.594+0000 7f3ab759e640 1 --2- 172.21.15.98:0/1065629216 >> v2:172.21.15.113:3300/0 conn(0x7f3ab8153c80 0x7f3ab806b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.594+0000 7f3ab759e640 1 -- 172.21.15.98:0/1065629216 >> 172.21.15.98:0/1065629216 conn(0x7f3ab80cf300 msgr2=0x7f3ab80cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.595+0000 7f3ab759e640 1 -- 172.21.15.98:0/1065629216 shutdown_connections 2023-11-28T17:42:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.595+0000 7f3ab759e640 1 -- 172.21.15.98:0/1065629216 wait complete. 2023-11-28T17:42:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.597+0000 7f3ab759e640 1 Processor -- start 2023-11-28T17:42:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.598+0000 7f3ab759e640 1 -- start start 2023-11-28T17:42:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.599+0000 7f3ab759e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ab80c69b0 0x7f3ab81e4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.599+0000 7f3ab759e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ab80c7720 0x7f3ab8154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.599+0000 7f3ab759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8153c80 0x7f3ab8155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.599+0000 7f3ab759e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab81e3b70 con 0x7f3ab8153c80 2023-11-28T17:42:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.599+0000 7f3ab759e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab81e3890 con 0x7f3ab80c7720 2023-11-28T17:42:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.599+0000 7f3ab759e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab8152320 con 0x7f3ab80c69b0 2023-11-28T17:42:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.599+0000 7f3ab6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8153c80 0x7f3ab8155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.599+0000 7f3ab6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8153c80 0x7f3ab8155450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60620/0 (socket says 172.21.15.98:60620) 2023-11-28T17:42:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.599+0000 7f3ab6d9d640 1 -- 172.21.15.98:0/3259120296 learned_addr learned my addr 172.21.15.98:0/3259120296 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.600+0000 7f3ab659c640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ab80c69b0 0x7f3ab81e4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.600+0000 7f3ab6d9d640 1 -- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ab80c69b0 msgr2=0x7f3ab81e4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.600+0000 7f3ab6d9d640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3ab80c69b0 0x7f3ab81e4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.600+0000 7f3ab6d9d640 1 -- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ab80c7720 msgr2=0x7f3ab8154f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.600+0000 7f3ab5d9b640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ab80c7720 0x7f3ab8154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.600+0000 7f3ab6d9d640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ab80c7720 0x7f3ab8154f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.600+0000 7f3ab6d9d640 1 -- 172.21.15.98:0/3259120296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac0094d0 con 0x7f3ab8153c80 2023-11-28T17:42:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.600+0000 7f3ab5d9b640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ab80c7720 0x7f3ab8154f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:42:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.600+0000 7f3ab6d9d640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8153c80 0x7f3ab8155450 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000c7a0 tx=0x7f3ab000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.601+0000 7f3aa6ffd640 1 -- 172.21.15.98:0/3259120296 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0003b70 con 0x7f3ab8153c80 2023-11-28T17:42:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.601+0000 7f3aa6ffd640 1 -- 172.21.15.98:0/3259120296 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab0003cf0 con 0x7f3ab8153c80 2023-11-28T17:42:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.601+0000 7f3ab759e640 1 -- 172.21.15.98:0/3259120296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab81e5000 con 0x7f3ab8153c80 2023-11-28T17:42:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.601+0000 7f3aa6ffd640 1 -- 172.21.15.98:0/3259120296 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0003ea0 con 0x7f3ab8153c80 2023-11-28T17:42:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.601+0000 7f3ab759e640 1 -- 172.21.15.98:0/3259120296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab81e3240 con 0x7f3ab8153c80 2023-11-28T17:42:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.603+0000 7f3ab759e640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a9c002a30 0x7f3a9c004ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.603+0000 7f3ab759e640 1 -- 172.21.15.98:0/3259120296 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c007e80 con 0x7f3a9c002a30 2023-11-28T17:42:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.603+0000 7f3aa6ffd640 1 -- 172.21.15.98:0/3259120296 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3ab000aa90 con 0x7f3ab8153c80 2023-11-28T17:42:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.603+0000 7f3ab659c640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a9c002a30 0x7f3a9c004ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.604+0000 7f3ab659c640 1 -- 172.21.15.98:0/3259120296 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a9c008040 con 0x7f3a9c002a30 2023-11-28T17:42:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.604+0000 7f3ab659c640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a9c002a30 0x7f3a9c004ef0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800f440 tx=0x7f3aa800f900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.605+0000 7f3aa6ffd640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3a9806b2a0 0x7f3a9806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.605+0000 7f3aa6ffd640 1 -- 172.21.15.98:0/3259120296 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3ab0089a90 con 0x7f3ab8153c80 2023-11-28T17:42:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.605+0000 7f3ab5d9b640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3a9806b2a0 0x7f3a9806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.605+0000 7f3aa6ffd640 1 -- 172.21.15.98:0/3259120296 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8004020 con 0x7f3a9c002a30 2023-11-28T17:42:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.606+0000 7f3aa6ffd640 1 -- 172.21.15.98:0/3259120296 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f3aa8004180 con 0x7f3a9c002a30 2023-11-28T17:42:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.606+0000 7f3aa6ffd640 1 -- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a9c002a30 msgr2=0x7f3a9c004ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.606+0000 7f3aa6ffd640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a9c002a30 0x7f3a9c004ef0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800f440 tx=0x7f3aa800f900 comp rx=0 tx=0).stop 2023-11-28T17:42:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.606+0000 7f3aa6ffd640 1 -- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a9c002a30 msgr2=0x7f3a9c004ef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.606+0000 7f3aa6ffd640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a9c002a30 0x7f3a9c004ef0 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800f440 tx=0x7f3aa800f900 comp rx=0 tx=0).stop 2023-11-28T17:42:41.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.606+0000 7f3ab5d9b640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3a9806b2a0 0x7f3a9806d760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3aac036000 tx=0x7f3aac008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:41.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.618+0000 7f3ab759e640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a9c006de0 0x7f3a9c007250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.618+0000 7f3ab759e640 1 -- 172.21.15.98:0/3259120296 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c00a3e0 con 0x7f3a9c006de0 2023-11-28T17:42:41.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.619+0000 7f3ab659c640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a9c006de0 0x7f3a9c007250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.620+0000 7f3ab659c640 1 -- 172.21.15.98:0/3259120296 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3a9c007790 con 0x7f3a9c006de0 2023-11-28T17:42:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.620+0000 7f3ab659c640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a9c006de0 0x7f3a9c007250 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800f440 tx=0x7f3aa80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:41.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.621+0000 7f3aa6ffd640 1 -- 172.21.15.98:0/3259120296 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8014d80 con 0x7f3a9c006de0 2023-11-28T17:42:41.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.621+0000 7f3aa6ffd640 1 -- 172.21.15.98:0/3259120296 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4805 (secure 0 0 0) 0x7f3aa8003610 con 0x7f3a9c006de0 2023-11-28T17:42:41.623 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:42:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:42:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:42:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:42:41.623 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 14, 2023-11-28T17:42:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:42:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:42:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:42:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:42:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:42:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:42:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:42:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:41.624 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:42:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:42:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:41.625 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:42:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:42:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:42:41.626 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:42:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:42:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:42:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:42:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:42:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:42:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:42:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:42:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:42:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:42:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:42:41.627 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:42:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:42:41.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:42:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:42:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:42:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:42:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:42:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:42:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:42:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:42:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:41.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:42:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:42:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:42:41.631 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:42:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:42:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:42:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:42:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:41.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:42:41.633 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.633 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:42:41.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:42:41.633 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:41.633 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:41.633 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:41.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.633 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.634 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:42:41.634 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:42:41.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.634 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:41.634 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:41.634 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:41.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.634 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.634 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:42:41.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.634 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:41.634 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:41.634 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:42:41.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.635 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.635 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:42:41.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.635 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:41.635 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:41.635 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:42:41.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.635 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.635 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:42:41.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.635 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:41.635 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:41.635 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:42:41.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.636 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.636 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:42:41.636 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:42:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.621+0000 7f3aa6ffd640 1 -- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a9c006de0 msgr2=0x7f3a9c007250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.621+0000 7f3aa6ffd640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a9c006de0 0x7f3a9c007250 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800f440 tx=0x7f3aa80046c0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.621+0000 7f3aa6ffd640 1 -- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a9c006de0 msgr2=0x7f3a9c007250 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.621+0000 7f3aa6ffd640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a9c006de0 0x7f3a9c007250 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800f440 tx=0x7f3aa80046c0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.622+0000 7f3ab759e640 1 -- 172.21.15.98:0/3259120296 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3a9806b2a0 msgr2=0x7f3a9806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.622+0000 7f3ab759e640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3a9806b2a0 0x7f3a9806d760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3aac036000 tx=0x7f3aac008030 comp rx=0 tx=0).stop 2023-11-28T17:42:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.622+0000 7f3ab759e640 1 -- 172.21.15.98:0/3259120296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8153c80 msgr2=0x7f3ab8155450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.622+0000 7f3ab759e640 1 --2- 172.21.15.98:0/3259120296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8153c80 0x7f3ab8155450 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000c7a0 tx=0x7f3ab000cc60 comp rx=0 tx=0).stop 2023-11-28T17:42:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.622+0000 7f3ab6d9d640 1 -- 172.21.15.98:0/3259120296 reap_dead start 2023-11-28T17:42:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.622+0000 7f3ab6d9d640 1 -- 172.21.15.98:0/3259120296 reap_dead start 2023-11-28T17:42:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.622+0000 7f3ab759e640 1 -- 172.21.15.98:0/3259120296 shutdown_connections 2023-11-28T17:42:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.622+0000 7f3ab759e640 1 -- 172.21.15.98:0/3259120296 >> 172.21.15.98:0/3259120296 conn(0x7f3ab80cf300 msgr2=0x7f3ab8151a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:41.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.622+0000 7f3ab759e640 1 -- 172.21.15.98:0/3259120296 shutdown_connections 2023-11-28T17:42:41.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.622+0000 7f3ab759e640 1 -- 172.21.15.98:0/3259120296 wait complete. 2023-11-28T17:42:41.639 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:42:41.639 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:42:41.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:42:41.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.817+0000 7fddebfff640 1 Processor -- start 2023-11-28T17:42:41.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.818+0000 7fddebfff640 1 -- start start 2023-11-28T17:42:41.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.819+0000 7fddebfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fddec153c80 0x7fddec06b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.819+0000 7fddebfff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fddec0c69b0 0x7fddec06be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.819+0000 7fddebfff640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddec15d230 con 0x7fddec0c7720 2023-11-28T17:42:41.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.819+0000 7fddebfff640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddec15e070 con 0x7fddec153c80 2023-11-28T17:42:41.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.819+0000 7fddebfff640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddec15d4f0 con 0x7fddec0c69b0 2023-11-28T17:42:41.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.819+0000 7fddeaffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fddec153c80 0x7fddec06b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.820+0000 7fddeaffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fddec153c80 0x7fddec06b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60630/0 (socket says 172.21.15.98:60630) 2023-11-28T17:42:41.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.820+0000 7fddeaffd640 1 -- 172.21.15.98:0/3683942720 learned_addr learned my addr 172.21.15.98:0/3683942720 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:41.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.820+0000 7fddea7fc640 1 --2- 172.21.15.98:0/3683942720 >> v2:172.21.15.113:3300/0 conn(0x7fddec0c69b0 0x7fddec06be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.820+0000 7fddeaffd640 1 -- 172.21.15.98:0/3683942720 >> v2:172.21.15.113:3300/0 conn(0x7fddec0c69b0 msgr2=0x7fddec06be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:41.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.820+0000 7fddeaffd640 1 --2- 172.21.15.98:0/3683942720 >> v2:172.21.15.113:3300/0 conn(0x7fddec0c69b0 0x7fddec06be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.820+0000 7fddeaffd640 1 -- 172.21.15.98:0/3683942720 >> v1:172.21.15.113:6789/0 conn(0x7fddec0c7720 legacy=0x7fddec0c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:41.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.820+0000 7fddeaffd640 1 -- 172.21.15.98:0/3683942720 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddec15d7d0 con 0x7fddec153c80 2023-11-28T17:42:41.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.822+0000 7fddeaffd640 1 --2- 172.21.15.98:0/3683942720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddec153c80 0x7fddec06b940 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fddec06c580 tx=0x7fdddc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9dd9ad56069ea server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:41.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.822+0000 7fdde9ffb640 1 -- 172.21.15.98:0/3683942720 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc030860 con 0x7fddec153c80 2023-11-28T17:42:41.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.822+0000 7fdde9ffb640 1 -- 172.21.15.98:0/3683942720 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdddc0309c0 con 0x7fddec153c80 2023-11-28T17:42:41.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.822+0000 7fdde9ffb640 1 -- 172.21.15.98:0/3683942720 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc030d50 con 0x7fddec153c80 2023-11-28T17:42:41.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.823+0000 7fddebfff640 1 -- 172.21.15.98:0/3683942720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddec153c80 msgr2=0x7fddec06b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.823+0000 7fddebfff640 1 --2- 172.21.15.98:0/3683942720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddec153c80 0x7fddec06b940 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fddec06c580 tx=0x7fdddc02b9a0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.823+0000 7fddebfff640 1 -- 172.21.15.98:0/3683942720 shutdown_connections 2023-11-28T17:42:41.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.823+0000 7fddebfff640 1 --2- 172.21.15.98:0/3683942720 >> v2:172.21.15.113:3300/0 conn(0x7fddec0c69b0 0x7fddec06be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.823+0000 7fddebfff640 1 --2- 172.21.15.98:0/3683942720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddec153c80 0x7fddec06b940 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.823+0000 7fddebfff640 1 -- 172.21.15.98:0/3683942720 >> 172.21.15.98:0/3683942720 conn(0x7fddec0cf300 msgr2=0x7fddec0cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:41.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.823+0000 7fddebfff640 1 -- 172.21.15.98:0/3683942720 shutdown_connections 2023-11-28T17:42:41.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.823+0000 7fddebfff640 1 -- 172.21.15.98:0/3683942720 wait complete. 2023-11-28T17:42:41.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.825+0000 7fddebfff640 1 Processor -- start 2023-11-28T17:42:41.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.826+0000 7fddebfff640 1 -- start start 2023-11-28T17:42:41.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.827+0000 7fddebfff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec0c69b0 0x7fddec1e53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.827+0000 7fddebfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddec0c7720 0x7fddec14c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.827+0000 7fddebfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddec153c80 0x7fddec14ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.827+0000 7fddebfff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddec15cde0 con 0x7fddec153c80 2023-11-28T17:42:41.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.827+0000 7fddebfff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddec1508e0 con 0x7fddec0c7720 2023-11-28T17:42:41.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.827+0000 7fddebfff640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddec1ef6d0 con 0x7fddec0c69b0 2023-11-28T17:42:41.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.827+0000 7fddeb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddec153c80 0x7fddec14ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.828+0000 7fddeb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddec153c80 0x7fddec14ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60640/0 (socket says 172.21.15.98:60640) 2023-11-28T17:42:41.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.828+0000 7fddeb7fe640 1 -- 172.21.15.98:0/1249810860 learned_addr learned my addr 172.21.15.98:0/1249810860 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:41.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.828+0000 7fddeaffd640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec0c69b0 0x7fddec1e53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.828+0000 7fddea7fc640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddec0c7720 0x7fddec14c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.828+0000 7fddeb7fe640 1 -- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec0c69b0 msgr2=0x7fddec1e53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.828+0000 7fddeb7fe640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec0c69b0 0x7fddec1e53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.828+0000 7fddeb7fe640 1 -- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddec0c7720 msgr2=0x7fddec14c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.828+0000 7fddeb7fe640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddec0c7720 0x7fddec14c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.828+0000 7fddeb7fe640 1 -- 172.21.15.98:0/1249810860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc0094d0 con 0x7fddec153c80 2023-11-28T17:42:41.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.828+0000 7fddea7fc640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddec0c7720 0x7fddec14c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:42:41.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.828+0000 7fddeb7fe640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddec153c80 0x7fddec14ca80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdde400c780 tx=0x7fdde400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:41.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.829+0000 7fddd3fff640 1 -- 172.21.15.98:0/1249810860 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4003d80 con 0x7fddec153c80 2023-11-28T17:42:41.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.829+0000 7fddd3fff640 1 -- 172.21.15.98:0/1249810860 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde400ce90 con 0x7fddec153c80 2023-11-28T17:42:41.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.829+0000 7fddebfff640 1 -- 172.21.15.98:0/1249810860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddec1f0550 con 0x7fddec153c80 2023-11-28T17:42:41.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.829+0000 7fddd3fff640 1 -- 172.21.15.98:0/1249810860 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde40052b0 con 0x7fddec153c80 2023-11-28T17:42:41.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.829+0000 7fddebfff640 1 -- 172.21.15.98:0/1249810860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddec1efc20 con 0x7fddec153c80 2023-11-28T17:42:41.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.830+0000 7fddebfff640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec1f2b30 0x7fddec1efe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.830+0000 7fddebfff640 1 -- 172.21.15.98:0/1249810860 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddec068170 con 0x7fddec1f2b30 2023-11-28T17:42:41.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.831+0000 7fddeaffd640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec1f2b30 0x7fddec1efe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.831+0000 7fddeaffd640 1 -- 172.21.15.98:0/1249810860 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fddec0cbb10 con 0x7fddec1f2b30 2023-11-28T17:42:41.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.832+0000 7fddeaffd640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec1f2b30 0x7fddec1efe40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fddec067c00 tx=0x7fdddc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:41.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.832+0000 7fddd3fff640 1 -- 172.21.15.98:0/1249810860 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fdde4006b10 con 0x7fddec153c80 2023-11-28T17:42:41.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.833+0000 7fddd3fff640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fddd806b2a0 0x7fddd806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.833+0000 7fddd3fff640 1 -- 172.21.15.98:0/1249810860 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdde40891f0 con 0x7fddec153c80 2023-11-28T17:42:41.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.833+0000 7fddea7fc640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fddd806b2a0 0x7fddd806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.833+0000 7fddd3fff640 1 -- 172.21.15.98:0/1249810860 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc039d00 con 0x7fddec1f2b30 2023-11-28T17:42:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.833+0000 7fddd3fff640 1 -- 172.21.15.98:0/1249810860 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fdddc039e60 con 0x7fddec1f2b30 2023-11-28T17:42:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.833+0000 7fddd3fff640 1 -- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec1f2b30 msgr2=0x7fddec1efe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.833+0000 7fddd3fff640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec1f2b30 0x7fddec1efe40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fddec067c00 tx=0x7fdddc004730 comp rx=0 tx=0).stop 2023-11-28T17:42:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.833+0000 7fddd3fff640 1 -- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec1f2b30 msgr2=0x7fddec1efe40 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.833+0000 7fddd3fff640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec1f2b30 0x7fddec1efe40 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fddec067c00 tx=0x7fdddc004730 comp rx=0 tx=0).stop 2023-11-28T17:42:41.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.834+0000 7fddea7fc640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fddd806b2a0 0x7fddd806d760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fddec14ded0 tx=0x7fdde0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:41.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.845+0000 7fddebfff640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec14b440 0x7fddec14b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:41.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.845+0000 7fddebfff640 1 -- 172.21.15.98:0/1249810860 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddec15dd90 con 0x7fddec14b440 2023-11-28T17:42:41.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.846+0000 7fddeaffd640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec14b440 0x7fddec14b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:41.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.847+0000 7fddeaffd640 1 -- 172.21.15.98:0/1249810860 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fddec0b50d0 con 0x7fddec14b440 2023-11-28T17:42:41.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.847+0000 7fddeaffd640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec14b440 0x7fddec14b850 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fddec072a70 tx=0x7fdddc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:41.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.848+0000 7fddd3fff640 1 -- 172.21.15.98:0/1249810860 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc003ea0 con 0x7fddec14b440 2023-11-28T17:42:41.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.848+0000 7fddd3fff640 1 -- 172.21.15.98:0/1249810860 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4975 (secure 0 0 0) 0x7fdddc04c6c0 con 0x7fddec14b440 2023-11-28T17:42:41.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.848+0000 7fddd3fff640 1 -- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec14b440 msgr2=0x7fddec14b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.848+0000 7fddd3fff640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec14b440 0x7fddec14b850 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fddec072a70 tx=0x7fdddc0046c0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.848+0000 7fddd3fff640 1 -- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec14b440 msgr2=0x7fddec14b850 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:41.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.848+0000 7fddd3fff640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddec14b440 0x7fddec14b850 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fddec072a70 tx=0x7fdddc0046c0 comp rx=0 tx=0).stop 2023-11-28T17:42:41.851 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:42:41.852 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:42:41.852 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:42:41.852 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:42:41.852 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 14, 2023-11-28T17:42:41.852 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:42:41.852 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:42:41.852 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:42:41.853 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:42:41.853 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.853 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:42:41.853 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:41.853 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:42:41.853 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:42:41.853 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:41.853 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:41.854 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:41.854 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:41.854 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:41.854 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:41.854 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:41.854 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:41.854 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:41.854 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:41.855 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.855 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:42:41.855 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:42:41.855 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:41.855 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:41.855 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:41.855 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:41.856 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:41.856 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:41.856 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:41.856 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:41.856 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:41.856 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:41.856 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.857 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:42:41.857 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:42:41.857 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:42:41.857 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:42:41.857 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:42:41.857 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:42:41.857 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:41.857 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:41.857 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:42:41.858 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:42:41.858 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:42:41.858 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:42:41.858 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:42:41.858 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:42:41.859 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:42:41.859 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:41.859 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:42:41.859 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:41.859 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:41.859 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:41.859 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:41.859 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:41.860 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:41.860 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:41.860 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:41.860 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:41.860 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:41.860 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.860 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:42:41.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.860 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:42:41.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.861 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:42:41.861 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:42:41.861 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:41.861 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:41.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.861 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:41.862 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:42:41.862 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.862 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:41.862 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:42:41.862 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.862 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.862 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.862 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:42:41.863 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:42:41.863 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:41.863 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:41.863 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:41.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.863 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:42:41.863 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:42:41.863 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:41.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:41.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:41.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:42:41.863 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:41.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:42:41.864 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.864 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.864 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:42:41.864 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:42:41.864 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:41.864 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:41.865 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:41.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.865 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:42:41.865 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:42:41.865 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:41.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:41.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:41.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:42:41.866 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:41.867 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:42:41.867 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:41.867 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.868 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:42:41.868 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:42:41.868 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:41.869 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:41.869 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:41.869 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.869 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.869 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:42:41.870 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:42:41.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.870 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:41.870 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:41.870 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:41.870 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.870 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.870 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:42:41.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.870 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:41.871 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:41.871 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:41.871 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.871 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.871 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:42:41.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.871 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:41.871 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:41.872 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:42:41.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.872 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.872 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:42:41.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.873 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:41.873 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:41.873 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:42:41.873 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.873 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:41.873 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:42:41.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:41.873 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:41.873 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:41.873 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:42:41.873 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:41.873 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:41.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:41.874 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:42:41.874 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:42:41.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.849+0000 7fddebfff640 1 -- 172.21.15.98:0/1249810860 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fddd806b2a0 msgr2=0x7fddd806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.849+0000 7fddebfff640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fddd806b2a0 0x7fddd806d760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fddec14ded0 tx=0x7fdde0008040 comp rx=0 tx=0).stop 2023-11-28T17:42:41.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.849+0000 7fddebfff640 1 -- 172.21.15.98:0/1249810860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddec153c80 msgr2=0x7fddec14ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:41.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.849+0000 7fddebfff640 1 --2- 172.21.15.98:0/1249810860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddec153c80 0x7fddec14ca80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdde400c780 tx=0x7fdde400cc40 comp rx=0 tx=0).stop 2023-11-28T17:42:41.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.849+0000 7fddeb7fe640 1 -- 172.21.15.98:0/1249810860 reap_dead start 2023-11-28T17:42:41.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.849+0000 7fddeb7fe640 1 -- 172.21.15.98:0/1249810860 reap_dead start 2023-11-28T17:42:41.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.849+0000 7fddebfff640 1 -- 172.21.15.98:0/1249810860 shutdown_connections 2023-11-28T17:42:41.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.849+0000 7fddebfff640 1 -- 172.21.15.98:0/1249810860 >> 172.21.15.98:0/1249810860 conn(0x7fddec0cf300 msgr2=0x7fddec0ca7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:41.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.849+0000 7fddebfff640 1 -- 172.21.15.98:0/1249810860 shutdown_connections 2023-11-28T17:42:41.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:41.849+0000 7fddebfff640 1 -- 172.21.15.98:0/1249810860 wait complete. 2023-11-28T17:42:41.877 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:42:41.877 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-11-28T17:42:41.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-11-28T17:42:42.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.095+0000 7fc5bb59e640 1 Processor -- start 2023-11-28T17:42:42.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.095+0000 7fc5bb59e640 1 -- start start 2023-11-28T17:42:42.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.097+0000 7fc5bb59e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5bc0728a0 0x7fc5bc158240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:42.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.097+0000 7fc5bb59e640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc5bc15ac70 0x7fc5bc15b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:42.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.097+0000 7fc5bb59e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc06ade0 con 0x7fc5bc1518d0 2023-11-28T17:42:42.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.097+0000 7fc5bb59e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5bc151cb0 con 0x7fc5bc0728a0 2023-11-28T17:42:42.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.097+0000 7fc5bb59e640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5bc06b030 con 0x7fc5bc15ac70 2023-11-28T17:42:42.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.097+0000 7fc5b9d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5bc0728a0 0x7fc5bc158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:42.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.097+0000 7fc5b9d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5bc0728a0 0x7fc5bc158240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60648/0 (socket says 172.21.15.98:60648) 2023-11-28T17:42:42.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.097+0000 7fc5b9d9b640 1 -- 172.21.15.98:0/781435965 learned_addr learned my addr 172.21.15.98:0/781435965 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:42.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.098+0000 7fc5bad9d640 1 --2- 172.21.15.98:0/781435965 >> v2:172.21.15.113:3300/0 conn(0x7fc5bc15ac70 0x7fc5bc15b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:42.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.098+0000 7fc5b9d9b640 1 -- 172.21.15.98:0/781435965 >> v2:172.21.15.113:3300/0 conn(0x7fc5bc15ac70 msgr2=0x7fc5bc15b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:42.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.098+0000 7fc5b959a640 1 -- 172.21.15.98:0/781435965 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930132071 0 0) 0x7fc5bc06ade0 con 0x7fc5bc1518d0 2023-11-28T17:42:42.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.098+0000 7fc5b9d9b640 1 --2- 172.21.15.98:0/781435965 >> v2:172.21.15.113:3300/0 conn(0x7fc5bc15ac70 0x7fc5bc15b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:42.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.098+0000 7fc5b9d9b640 1 -- 172.21.15.98:0/781435965 >> v1:172.21.15.98:6789/0 conn(0x7fc5bc1518d0 legacy=0x7fc5bc070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:42.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.098+0000 7fc5b9d9b640 1 -- 172.21.15.98:0/781435965 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc0700e0 con 0x7fc5bc0728a0 2023-11-28T17:42:42.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.100+0000 7fc5b9d9b640 1 --2- 172.21.15.98:0/781435965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5bc0728a0 0x7fc5bc158240 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc072dd0 tx=0x7fc5b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b083a48be343f6b8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:42.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.100+0000 7fc5b959a640 1 -- 172.21.15.98:0/781435965 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0031870 con 0x7fc5bc0728a0 2023-11-28T17:42:42.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.100+0000 7fc5b959a640 1 -- 172.21.15.98:0/781435965 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b00319d0 con 0x7fc5bc0728a0 2023-11-28T17:42:42.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.100+0000 7fc5b959a640 1 -- 172.21.15.98:0/781435965 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0031d60 con 0x7fc5bc0728a0 2023-11-28T17:42:42.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.100+0000 7fc5bb59e640 1 -- 172.21.15.98:0/781435965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5bc0728a0 msgr2=0x7fc5bc158240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:42.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.100+0000 7fc5bb59e640 1 --2- 172.21.15.98:0/781435965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5bc0728a0 0x7fc5bc158240 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc072dd0 tx=0x7fc5b002b910 comp rx=0 tx=0).stop 2023-11-28T17:42:42.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.101+0000 7fc5bb59e640 1 -- 172.21.15.98:0/781435965 shutdown_connections 2023-11-28T17:42:42.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.101+0000 7fc5bb59e640 1 --2- 172.21.15.98:0/781435965 >> v2:172.21.15.113:3300/0 conn(0x7fc5bc15ac70 0x7fc5bc15b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:42.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.101+0000 7fc5bb59e640 1 --2- 172.21.15.98:0/781435965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5bc0728a0 0x7fc5bc158240 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:42.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.101+0000 7fc5bb59e640 1 -- 172.21.15.98:0/781435965 >> 172.21.15.98:0/781435965 conn(0x7fc5bc066230 msgr2=0x7fc5bc06a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:42.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.101+0000 7fc5bb59e640 1 -- 172.21.15.98:0/781435965 shutdown_connections 2023-11-28T17:42:42.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.101+0000 7fc5bb59e640 1 -- 172.21.15.98:0/781435965 wait complete. 2023-11-28T17:42:42.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.103+0000 7fc5bb59e640 1 Processor -- start 2023-11-28T17:42:42.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.103+0000 7fc5bb59e640 1 -- start start 2023-11-28T17:42:42.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.105+0000 7fc5bb59e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5bc0728a0 0x7fc5bc0c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:42.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.105+0000 7fc5bb59e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5bc1518d0 0x7fc5bc15b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:42.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.105+0000 7fc5bb59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5bc15ac70 0x7fc5bc1ed120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:42.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.105+0000 7fc5bb59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc1f10d0 con 0x7fc5bc15ac70 2023-11-28T17:42:42.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.105+0000 7fc5bb59e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc1f14f0 con 0x7fc5bc1518d0 2023-11-28T17:42:42.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.105+0000 7fc5bb59e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc0c9800 con 0x7fc5bc0728a0 2023-11-28T17:42:42.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.105+0000 7fc5bad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5bc15ac70 0x7fc5bc1ed120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:42.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.105+0000 7fc5bad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5bc15ac70 0x7fc5bc1ed120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60664/0 (socket says 172.21.15.98:60664) 2023-11-28T17:42:42.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.105+0000 7fc5bad9d640 1 -- 172.21.15.98:0/758075447 learned_addr learned my addr 172.21.15.98:0/758075447 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:42.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.106+0000 7fc5b9d9b640 1 --2- 172.21.15.98:0/758075447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5bc1518d0 0x7fc5bc15b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:42.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.106+0000 7fc5bad9d640 1 -- 172.21.15.98:0/758075447 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5bc0728a0 msgr2=0x7fc5bc0c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:42.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.106+0000 7fc5bad9d640 1 --2- 172.21.15.98:0/758075447 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5bc0728a0 0x7fc5bc0c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:42.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.106+0000 7fc5bad9d640 1 -- 172.21.15.98:0/758075447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5bc1518d0 msgr2=0x7fc5bc15b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:42.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.106+0000 7fc5bad9d640 1 --2- 172.21.15.98:0/758075447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5bc1518d0 0x7fc5bc15b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:42.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.106+0000 7fc5bad9d640 1 -- 172.21.15.98:0/758075447 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b00094d0 con 0x7fc5bc15ac70 2023-11-28T17:42:42.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.106+0000 7fc5bad9d640 1 --2- 172.21.15.98:0/758075447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5bc15ac70 0x7fc5bc1ed120 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400efc0 tx=0x7fc5b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:42.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.106+0000 7fc5aaffd640 1 -- 172.21.15.98:0/758075447 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4014480 con 0x7fc5bc15ac70 2023-11-28T17:42:42.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.106+0000 7fc5aaffd640 1 -- 172.21.15.98:0/758075447 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b400f040 con 0x7fc5bc15ac70 2023-11-28T17:42:42.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.106+0000 7fc5bb59e640 1 -- 172.21.15.98:0/758075447 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5bc1f0730 con 0x7fc5bc15ac70 2023-11-28T17:42:42.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.106+0000 7fc5aaffd640 1 -- 172.21.15.98:0/758075447 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4014820 con 0x7fc5bc15ac70 2023-11-28T17:42:42.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.107+0000 7fc5bb59e640 1 -- 172.21.15.98:0/758075447 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5bc1f0cf0 con 0x7fc5bc15ac70 2023-11-28T17:42:42.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.108+0000 7fc5bb59e640 1 -- 172.21.15.98:0/758075447 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5bc1f1210 con 0x7fc5bc15ac70 2023-11-28T17:42:42.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.109+0000 7fc5aaffd640 1 -- 172.21.15.98:0/758075447 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fc5b4014980 con 0x7fc5bc15ac70 2023-11-28T17:42:42.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.110+0000 7fc5aaffd640 1 --2- 172.21.15.98:0/758075447 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc59c06b2d0 0x7fc59c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:42.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.110+0000 7fc5aaffd640 1 -- 172.21.15.98:0/758075447 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc5b408ab20 con 0x7fc5bc15ac70 2023-11-28T17:42:42.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.112+0000 7fc5ba59c640 1 --2- 172.21.15.98:0/758075447 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc59c06b2d0 0x7fc59c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:42.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.113+0000 7fc5ba59c640 1 --2- 172.21.15.98:0/758075447 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc59c06b2d0 0x7fc59c06d790 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac001560 tx=0x7fc5ac007930 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:42.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.115+0000 7fc5aaffd640 1 -- 172.21.15.98:0/758075447 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5bc1f1210 con 0x7fc5bc15ac70 2023-11-28T17:42:42.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.435+0000 7fc5bb59e640 1 -- 172.21.15.98:0/758075447 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc5bc1f1210 con 0x7fc5bc15ac70 2023-11-28T17:42:42.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.437+0000 7fc5aaffd640 1 -- 172.21.15.98:0/758075447 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fc5bc1f1210 con 0x7fc5bc15ac70 2023-11-28T17:42:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.440+0000 7fc5bb59e640 1 -- 172.21.15.98:0/758075447 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc59c06b2d0 msgr2=0x7fc59c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.440+0000 7fc5bb59e640 1 --2- 172.21.15.98:0/758075447 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc59c06b2d0 0x7fc59c06d790 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac001560 tx=0x7fc5ac007930 comp rx=0 tx=0).stop 2023-11-28T17:42:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.440+0000 7fc5bb59e640 1 -- 172.21.15.98:0/758075447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5bc15ac70 msgr2=0x7fc5bc1ed120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.440+0000 7fc5bb59e640 1 --2- 172.21.15.98:0/758075447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5bc15ac70 0x7fc5bc1ed120 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400efc0 tx=0x7fc5b400c490 comp rx=0 tx=0).stop 2023-11-28T17:42:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.441+0000 7fc5bb59e640 1 -- 172.21.15.98:0/758075447 shutdown_connections 2023-11-28T17:42:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.441+0000 7fc5bb59e640 1 --2- 172.21.15.98:0/758075447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5bc15ac70 0x7fc5bc1ed120 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.441+0000 7fc5bb59e640 1 --2- 172.21.15.98:0/758075447 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc59c06b2d0 0x7fc59c06d790 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.441+0000 7fc5bb59e640 1 --2- 172.21.15.98:0/758075447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5bc1518d0 0x7fc5bc15b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.441+0000 7fc5bb59e640 1 --2- 172.21.15.98:0/758075447 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc5bc0728a0 0x7fc5bc0c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.441+0000 7fc5bb59e640 1 -- 172.21.15.98:0/758075447 >> 172.21.15.98:0/758075447 conn(0x7fc5bc066230 msgr2=0x7fc5bc0c36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.441+0000 7fc5bb59e640 1 -- 172.21.15.98:0/758075447 shutdown_connections 2023-11-28T17:42:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:42.441+0000 7fc5bb59e640 1 -- 172.21.15.98:0/758075447 wait complete. 2023-11-28T17:42:42.461 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2023-11-28T17:42:52.462 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:42:52.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:42:52.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.740+0000 7fcb9c9db640 1 Processor -- start 2023-11-28T17:42:52.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.740+0000 7fcb9c9db640 1 -- start start 2023-11-28T17:42:52.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.742+0000 7fcb9c9db640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcb9815a6d0 0x7fcb980cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:52.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.742+0000 7fcb9c9db640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9815ab10 con 0x7fcb98153bb0 2023-11-28T17:42:52.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.742+0000 7fcb9c9db640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9815d330 con 0x7fcb980cf7c0 2023-11-28T17:42:52.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.742+0000 7fcb9c9db640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb9815d120 con 0x7fcb9815a6d0 2023-11-28T17:42:52.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.742+0000 7fcb96d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcb98153bb0 0x7fcb980ca390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45090/0 (socket says 172.21.15.98:45090) 2023-11-28T17:42:52.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.742+0000 7fcb96d9d640 1 -- 172.21.15.98:0/3325182734 learned_addr learned my addr 172.21.15.98:0/3325182734 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:42:52.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.742+0000 7fcb9659c640 1 --2- 172.21.15.98:0/3325182734 >> v2:172.21.15.98:3300/0 conn(0x7fcb9815a6d0 0x7fcb980cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:52.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.743+0000 7fcb95d9b640 1 -- 172.21.15.98:0/3325182734 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026883561 0 0) 0x7fcb9815ab10 con 0x7fcb98153bb0 2023-11-28T17:42:52.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.743+0000 7fcb95d9b640 1 -- 172.21.15.98:0/3325182734 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb80003520 con 0x7fcb98153bb0 2023-11-28T17:42:52.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.743+0000 7fcb95d9b640 1 -- 172.21.15.98:0/3325182734 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 450959738 0 0) 0x7fcb80003520 con 0x7fcb98153bb0 2023-11-28T17:42:52.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.743+0000 7fcb95d9b640 1 -- 172.21.15.98:0/3325182734 >> v2:172.21.15.98:3300/0 conn(0x7fcb9815a6d0 msgr2=0x7fcb980cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.743+0000 7fcb95d9b640 1 --2- 172.21.15.98:0/3325182734 >> v2:172.21.15.98:3300/0 conn(0x7fcb9815a6d0 0x7fcb980cf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.743+0000 7fcb95d9b640 1 -- 172.21.15.98:0/3325182734 >> v1:172.21.15.113:6789/0 conn(0x7fcb980cf7c0 legacy=0x7fcb980b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.743+0000 7fcb95d9b640 1 -- 172.21.15.98:0/3325182734 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9815d610 con 0x7fcb98153bb0 2023-11-28T17:42:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.744+0000 7fcb95d9b640 1 -- 172.21.15.98:0/3325182734 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7fcb88001670 con 0x7fcb98153bb0 2023-11-28T17:42:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.744+0000 7fcb95d9b640 1 -- 172.21.15.98:0/3325182734 >> v1:172.21.15.98:6789/0 conn(0x7fcb98153bb0 legacy=0x7fcb980ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.744+0000 7fcb95d9b640 1 --2- 172.21.15.98:0/3325182734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb80003970 0x7fcb80003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.744+0000 7fcb95d9b640 1 --2- 172.21.15.98:0/3325182734 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb800047c0 0x7fcb80004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.744+0000 7fcb95d9b640 1 --2- 172.21.15.98:0/3325182734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb80005180 0x7fcb8000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.744+0000 7fcb95d9b640 1 -- 172.21.15.98:0/3325182734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9815d120 con 0x7fcb80005180 2023-11-28T17:42:52.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.744+0000 7fcb95d9b640 1 -- 172.21.15.98:0/3325182734 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8000c030 con 0x7fcb80003970 2023-11-28T17:42:52.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.744+0000 7fcb95d9b640 1 -- 172.21.15.98:0/3325182734 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8000c1f0 con 0x7fcb800047c0 2023-11-28T17:42:52.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.744+0000 7fcb9659c640 1 --2- 172.21.15.98:0/3325182734 >> v2:172.21.15.98:3300/0 conn(0x7fcb9815a6d0 0x7fcb980cf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:42:52.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.744+0000 7fcb9759e640 1 --2- 172.21.15.98:0/3325182734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb80005180 0x7fcb8000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:52.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.744+0000 7fcb96d9d640 1 --2- 172.21.15.98:0/3325182734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb80003970 0x7fcb80003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:52.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.745+0000 7fcb9659c640 1 --2- 172.21.15.98:0/3325182734 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb800047c0 0x7fcb80004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:52.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.745+0000 7fcb9759e640 1 -- 172.21.15.98:0/3325182734 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb800047c0 msgr2=0x7fcb80004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:52.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.745+0000 7fcb9759e640 1 --2- 172.21.15.98:0/3325182734 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb800047c0 0x7fcb80004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:52.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.745+0000 7fcb9759e640 1 -- 172.21.15.98:0/3325182734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb80003970 msgr2=0x7fcb80003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:52.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.745+0000 7fcb9759e640 1 --2- 172.21.15.98:0/3325182734 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb80003970 0x7fcb80003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:52.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.745+0000 7fcb9759e640 1 -- 172.21.15.98:0/3325182734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9815d610 con 0x7fcb80005180 2023-11-28T17:42:52.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.746+0000 7fcb9759e640 1 -- 172.21.15.98:0/3325182734 reap_dead start 2023-11-28T17:42:52.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.746+0000 7fcb9759e640 1 --2- 172.21.15.98:0/3325182734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb80005180 0x7fcb8000b980 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcb90001e50 tx=0x7fcb800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e841af1abfb69f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:52.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.747+0000 7fcb95d9b640 1 -- 172.21.15.98:0/3325182734 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90003b70 con 0x7fcb80005180 2023-11-28T17:42:52.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.747+0000 7fcb95d9b640 1 -- 172.21.15.98:0/3325182734 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb90003d10 con 0x7fcb80005180 2023-11-28T17:42:52.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.747+0000 7fcb95d9b640 1 -- 172.21.15.98:0/3325182734 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90005310 con 0x7fcb80005180 2023-11-28T17:42:52.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.748+0000 7fcb9c9db640 1 -- 172.21.15.98:0/3325182734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb80005180 msgr2=0x7fcb8000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:52.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.748+0000 7fcb9c9db640 1 --2- 172.21.15.98:0/3325182734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb80005180 0x7fcb8000b980 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcb90001e50 tx=0x7fcb800057f0 comp rx=0 tx=0).stop 2023-11-28T17:42:52.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.748+0000 7fcb9c9db640 1 -- 172.21.15.98:0/3325182734 shutdown_connections 2023-11-28T17:42:52.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.748+0000 7fcb9c9db640 1 --2- 172.21.15.98:0/3325182734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb80005180 0x7fcb8000b980 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:52.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.748+0000 7fcb9c9db640 1 -- 172.21.15.98:0/3325182734 >> 172.21.15.98:0/3325182734 conn(0x7fcb98071c00 msgr2=0x7fcb98072010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:52.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.748+0000 7fcb9c9db640 1 -- 172.21.15.98:0/3325182734 shutdown_connections 2023-11-28T17:42:52.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.748+0000 7fcb9c9db640 1 -- 172.21.15.98:0/3325182734 wait complete. 2023-11-28T17:42:52.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.751+0000 7fcb9c9db640 1 Processor -- start 2023-11-28T17:42:52.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.751+0000 7fcb9c9db640 1 -- start start 2023-11-28T17:42:52.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.752+0000 7fcb9c9db640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb80005180 0x7fcb9814d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:52.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.752+0000 7fcb9c9db640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb9814c310 0x7fcb98149720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:52.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.752+0000 7fcb9c9db640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb981f14e0 0x7fcb981f1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.752+0000 7fcb9c9db640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb980c6ac0 con 0x7fcb9814c310 2023-11-28T17:42:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.752+0000 7fcb9c9db640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb98150e90 con 0x7fcb981f14e0 2023-11-28T17:42:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.752+0000 7fcb9c9db640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb981f1ea0 con 0x7fcb80005180 2023-11-28T17:42:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.753+0000 7fcb9659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb9814c310 0x7fcb98149720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.753+0000 7fcb9659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb9814c310 0x7fcb98149720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44518/0 (socket says 172.21.15.98:44518) 2023-11-28T17:42:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.753+0000 7fcb9659c640 1 -- 172.21.15.98:0/3848707775 learned_addr learned my addr 172.21.15.98:0/3848707775 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.753+0000 7fcb96d9d640 1 --2- 172.21.15.98:0/3848707775 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb80005180 0x7fcb9814d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.753+0000 7fcb9759e640 1 --2- 172.21.15.98:0/3848707775 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb981f14e0 0x7fcb981f1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.753+0000 7fcb9659c640 1 -- 172.21.15.98:0/3848707775 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb80005180 msgr2=0x7fcb9814d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.753+0000 7fcb9659c640 1 --2- 172.21.15.98:0/3848707775 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb80005180 0x7fcb9814d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.753+0000 7fcb9659c640 1 -- 172.21.15.98:0/3848707775 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb981f14e0 msgr2=0x7fcb981f1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.753+0000 7fcb9659c640 1 --2- 172.21.15.98:0/3848707775 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb981f14e0 0x7fcb981f1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:52.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.753+0000 7fcb9659c640 1 -- 172.21.15.98:0/3848707775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9814c9f0 con 0x7fcb9814c310 2023-11-28T17:42:52.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.754+0000 7fcb9659c640 1 --2- 172.21.15.98:0/3848707775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb9814c310 0x7fcb98149720 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0116f0 tx=0x7fcb8c011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:52.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.754+0000 7fcb7f7fe640 1 -- 172.21.15.98:0/3848707775 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c00f9c0 con 0x7fcb9814c310 2023-11-28T17:42:52.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.754+0000 7fcb7f7fe640 1 -- 172.21.15.98:0/3848707775 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8c00fb20 con 0x7fcb9814c310 2023-11-28T17:42:52.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.754+0000 7fcb9c9db640 1 -- 172.21.15.98:0/3848707775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb981f3a10 con 0x7fcb9814c310 2023-11-28T17:42:52.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.754+0000 7fcb7f7fe640 1 -- 172.21.15.98:0/3848707775 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c00fe90 con 0x7fcb9814c310 2023-11-28T17:42:52.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.755+0000 7fcb9c9db640 1 -- 172.21.15.98:0/3848707775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb981f0ce0 con 0x7fcb9814c310 2023-11-28T17:42:52.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.756+0000 7fcb9c9db640 1 -- 172.21.15.98:0/3848707775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9814ced0 con 0x7fcb9814c310 2023-11-28T17:42:52.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.758+0000 7fcb7f7fe640 1 -- 172.21.15.98:0/3848707775 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fcb8c015460 con 0x7fcb9814c310 2023-11-28T17:42:52.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.760+0000 7fcb7f7fe640 1 --2- 172.21.15.98:0/3848707775 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcb8406b2f0 0x7fcb8406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:52.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.760+0000 7fcb7f7fe640 1 -- 172.21.15.98:0/3848707775 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcb8c08a620 con 0x7fcb9814c310 2023-11-28T17:42:52.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.760+0000 7fcb96d9d640 1 --2- 172.21.15.98:0/3848707775 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcb8406b2f0 0x7fcb8406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:52.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.761+0000 7fcb96d9d640 1 --2- 172.21.15.98:0/3848707775 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcb8406b2f0 0x7fcb8406d7b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcb88000fb0 tx=0x7fcb88009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:52.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:52.770+0000 7fcb7f7fe640 1 -- 172.21.15.98:0/3848707775 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb9814ced0 con 0x7fcb9814c310 2023-11-28T17:42:53.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.169+0000 7fcb9c9db640 1 -- 172.21.15.98:0/3848707775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcb981f0b40 con 0x7fcb9814c310 2023-11-28T17:42:53.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.170+0000 7fcb7f7fe640 1 -- 172.21.15.98:0/3848707775 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7fcb981f0b40 con 0x7fcb9814c310 2023-11-28T17:42:53.172 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:42:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.172+0000 7fcb9c9db640 1 -- 172.21.15.98:0/3848707775 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcb8406b2f0 msgr2=0x7fcb8406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.172+0000 7fcb9c9db640 1 --2- 172.21.15.98:0/3848707775 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcb8406b2f0 0x7fcb8406d7b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcb88000fb0 tx=0x7fcb88009040 comp rx=0 tx=0).stop 2023-11-28T17:42:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.172+0000 7fcb9c9db640 1 -- 172.21.15.98:0/3848707775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb9814c310 msgr2=0x7fcb98149720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.172+0000 7fcb9c9db640 1 --2- 172.21.15.98:0/3848707775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb9814c310 0x7fcb98149720 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0116f0 tx=0x7fcb8c011bb0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.172+0000 7fcb9c9db640 1 -- 172.21.15.98:0/3848707775 shutdown_connections 2023-11-28T17:42:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.172+0000 7fcb9c9db640 1 --2- 172.21.15.98:0/3848707775 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fcb8406b2f0 0x7fcb8406d7b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.172+0000 7fcb9c9db640 1 --2- 172.21.15.98:0/3848707775 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb981f14e0 0x7fcb981f1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.172+0000 7fcb9c9db640 1 --2- 172.21.15.98:0/3848707775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb9814c310 0x7fcb98149720 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.172+0000 7fcb9c9db640 1 --2- 172.21.15.98:0/3848707775 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fcb80005180 0x7fcb9814d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.172+0000 7fcb9c9db640 1 -- 172.21.15.98:0/3848707775 >> 172.21.15.98:0/3848707775 conn(0x7fcb98071c00 msgr2=0x7fcb98151440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:53.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.173+0000 7fcb9c9db640 1 -- 172.21.15.98:0/3848707775 shutdown_connections 2023-11-28T17:42:53.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.173+0000 7fcb9c9db640 1 -- 172.21.15.98:0/3848707775 wait complete. 2023-11-28T17:42:53.192 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:42:53.192 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-11-28T17:42:53.192 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:42:53.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:42:53.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.382+0000 7f971b59e640 1 Processor -- start 2023-11-28T17:42:53.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.382+0000 7f971b59e640 1 -- start start 2023-11-28T17:42:53.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.383+0000 7f971b59e640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f971c153c80 0x7f971c06b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.383+0000 7f971b59e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f971c0c7e30 0x7f971c06be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.383+0000 7f971b59e640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f971c15d230 con 0x7f971c0c69b0 2023-11-28T17:42:53.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.383+0000 7f971b59e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f971c15e070 con 0x7f971c0c7e30 2023-11-28T17:42:53.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.383+0000 7f971b59e640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f971c15d4f0 con 0x7f971c153c80 2023-11-28T17:42:53.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.384+0000 7f971ad9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f971c0c7e30 0x7f971c06be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.384+0000 7f971ad9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f971c0c7e30 0x7f971c06be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44522/0 (socket says 172.21.15.98:44522) 2023-11-28T17:42:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.384+0000 7f971ad9d640 1 -- 172.21.15.98:0/1265487100 learned_addr learned my addr 172.21.15.98:0/1265487100 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.384+0000 7f971a59c640 1 --2- 172.21.15.98:0/1265487100 >> v2:172.21.15.113:3300/0 conn(0x7f971c153c80 0x7f971c06b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.385+0000 7f971ad9d640 1 -- 172.21.15.98:0/1265487100 >> v2:172.21.15.113:3300/0 conn(0x7f971c153c80 msgr2=0x7f971c06b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.385+0000 7f971ad9d640 1 --2- 172.21.15.98:0/1265487100 >> v2:172.21.15.113:3300/0 conn(0x7f971c153c80 0x7f971c06b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.385+0000 7f971ad9d640 1 -- 172.21.15.98:0/1265487100 >> v1:172.21.15.113:6789/0 conn(0x7f971c0c69b0 legacy=0x7f971c0c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.385+0000 7f971ad9d640 1 -- 172.21.15.98:0/1265487100 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971c15d7d0 con 0x7f971c0c7e30 2023-11-28T17:42:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.386+0000 7f971ad9d640 1 --2- 172.21.15.98:0/1265487100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c7e30 0x7f971c06be80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f971c06c630 tx=0x7f971402c030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16d9024da72637a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.386+0000 7f971959a640 1 -- 172.21.15.98:0/1265487100 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971402cbd0 con 0x7f971c0c7e30 2023-11-28T17:42:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.387+0000 7f971959a640 1 -- 172.21.15.98:0/1265487100 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f971402cd30 con 0x7f971c0c7e30 2023-11-28T17:42:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.387+0000 7f971959a640 1 -- 172.21.15.98:0/1265487100 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9714034610 con 0x7f971c0c7e30 2023-11-28T17:42:53.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.387+0000 7f971b59e640 1 -- 172.21.15.98:0/1265487100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c7e30 msgr2=0x7f971c06be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.387+0000 7f971b59e640 1 --2- 172.21.15.98:0/1265487100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c7e30 0x7f971c06be80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f971c06c630 tx=0x7f971402c030 comp rx=0 tx=0).stop 2023-11-28T17:42:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.388+0000 7f971b59e640 1 -- 172.21.15.98:0/1265487100 shutdown_connections 2023-11-28T17:42:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.388+0000 7f971b59e640 1 --2- 172.21.15.98:0/1265487100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c7e30 0x7f971c06be80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.388+0000 7f971b59e640 1 --2- 172.21.15.98:0/1265487100 >> v2:172.21.15.113:3300/0 conn(0x7f971c153c80 0x7f971c06b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.388+0000 7f971b59e640 1 -- 172.21.15.98:0/1265487100 >> 172.21.15.98:0/1265487100 conn(0x7f971c0cf300 msgr2=0x7f971c0cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.388+0000 7f971b59e640 1 -- 172.21.15.98:0/1265487100 shutdown_connections 2023-11-28T17:42:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.388+0000 7f971b59e640 1 -- 172.21.15.98:0/1265487100 wait complete. 2023-11-28T17:42:53.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.390+0000 7f971b59e640 1 Processor -- start 2023-11-28T17:42:53.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.391+0000 7f971b59e640 1 -- start start 2023-11-28T17:42:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.392+0000 7f971b59e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f971c0c69b0 0x7f971c0ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.392+0000 7f971b59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c7e30 0x7f971c14dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.392+0000 7f971b59e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f971c153c80 0x7f971c15c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.392+0000 7f971b59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1ef4a0 con 0x7f971c0c7e30 2023-11-28T17:42:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.392+0000 7f971b59e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c0cec60 con 0x7f971c0c69b0 2023-11-28T17:42:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.392+0000 7f971b59e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971c0c3030 con 0x7f971c153c80 2023-11-28T17:42:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.392+0000 7f9719d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c7e30 0x7f971c14dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.393+0000 7f9719d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c7e30 0x7f971c14dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44536/0 (socket says 172.21.15.98:44536) 2023-11-28T17:42:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.393+0000 7f9719d9b640 1 -- 172.21.15.98:0/801236911 learned_addr learned my addr 172.21.15.98:0/801236911 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.393+0000 7f971a59c640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f971c0c69b0 0x7f971c0ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.393+0000 7f971ad9d640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f971c153c80 0x7f971c15c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.393+0000 7f9719d9b640 1 -- 172.21.15.98:0/801236911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f971c153c80 msgr2=0x7f971c15c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.393+0000 7f9719d9b640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f971c153c80 0x7f971c15c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.393+0000 7f9719d9b640 1 -- 172.21.15.98:0/801236911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f971c0c69b0 msgr2=0x7f971c0ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.393+0000 7f9719d9b640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f971c0c69b0 0x7f971c0ce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.393+0000 7f9719d9b640 1 -- 172.21.15.98:0/801236911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9714009b30 con 0x7f971c0c7e30 2023-11-28T17:42:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.393+0000 7f9719d9b640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c7e30 0x7f971c14dc20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f971000c450 tx=0x7f971000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.394+0000 7f970affd640 1 -- 172.21.15.98:0/801236911 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97100052d0 con 0x7f971c0c7e30 2023-11-28T17:42:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.394+0000 7f970affd640 1 -- 172.21.15.98:0/801236911 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9710005430 con 0x7f971c0c7e30 2023-11-28T17:42:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.394+0000 7f970affd640 1 -- 172.21.15.98:0/801236911 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9710012870 con 0x7f971c0c7e30 2023-11-28T17:42:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.394+0000 7f971b59e640 1 -- 172.21.15.98:0/801236911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971c0c3310 con 0x7f971c0c7e30 2023-11-28T17:42:53.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.394+0000 7f971b59e640 1 -- 172.21.15.98:0/801236911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971c0cef40 con 0x7f971c0c7e30 2023-11-28T17:42:53.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.395+0000 7f971b59e640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c3500 0x7f971c145140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.396+0000 7f971b59e640 1 -- 172.21.15.98:0/801236911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c068170 con 0x7f971c0c3500 2023-11-28T17:42:53.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.396+0000 7f971a59c640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c3500 0x7f971c145140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.396+0000 7f971a59c640 1 -- 172.21.15.98:0/801236911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f971c0cbb10 con 0x7f971c0c3500 2023-11-28T17:42:53.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.397+0000 7f971a59c640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c3500 0x7f971c145140 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f970c011890 tx=0x7f970c011d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:53.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.397+0000 7f970affd640 1 -- 172.21.15.98:0/801236911 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c004240 con 0x7f971c0c3500 2023-11-28T17:42:53.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.397+0000 7f970affd640 1 -- 172.21.15.98:0/801236911 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f97100129d0 con 0x7f971c0c7e30 2023-11-28T17:42:53.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.398+0000 7f970affd640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f96fc06b340 0x7f96fc06d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.398+0000 7f970affd640 1 -- 172.21.15.98:0/801236911 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9710089780 con 0x7f971c0c7e30 2023-11-28T17:42:53.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.399+0000 7f970affd640 1 -- 172.21.15.98:0/801236911 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f970c0043a0 con 0x7f971c0c3500 2023-11-28T17:42:53.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.399+0000 7f971ad9d640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f96fc06b340 0x7f96fc06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.399+0000 7f970affd640 1 -- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c3500 msgr2=0x7f971c145140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.399+0000 7f970affd640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c3500 0x7f971c145140 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f970c011890 tx=0x7f970c011d50 comp rx=0 tx=0).stop 2023-11-28T17:42:53.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.399+0000 7f970affd640 1 -- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c3500 msgr2=0x7f971c145140 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:53.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.399+0000 7f970affd640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c3500 0x7f971c145140 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f970c011890 tx=0x7f970c011d50 comp rx=0 tx=0).stop 2023-11-28T17:42:53.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.400+0000 7f971ad9d640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f96fc06b340 0x7f96fc06d800 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9714035000 tx=0x7f9714030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:53.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.406+0000 7f971b59e640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c15cc80 0x7f971c145680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.406+0000 7f971b59e640 1 -- 172.21.15.98:0/801236911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c0c31a0 con 0x7f971c15cc80 2023-11-28T17:42:53.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.406+0000 7f971a59c640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c15cc80 0x7f971c145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.407+0000 7f971a59c640 1 -- 172.21.15.98:0/801236911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f971c15dd90 con 0x7f971c15cc80 2023-11-28T17:42:53.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.407+0000 7f971a59c640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c15cc80 0x7f971c145680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f970c011890 tx=0x7f970c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:53.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.407+0000 7f970affd640 1 -- 172.21.15.98:0/801236911 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c01e450 con 0x7f971c15cc80 2023-11-28T17:42:53.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.407+0000 7f970affd640 1 -- 172.21.15.98:0/801236911 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4468 (secure 0 0 0) 0x7f970c024040 con 0x7f971c15cc80 2023-11-28T17:42:53.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.408+0000 7f970affd640 1 -- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c15cc80 msgr2=0x7f971c145680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.408+0000 7f970affd640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c15cc80 0x7f971c145680 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f970c011890 tx=0x7f970c0046c0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.408+0000 7f970affd640 1 -- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c15cc80 msgr2=0x7f971c145680 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:53.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.408+0000 7f970affd640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c15cc80 0x7f971c145680 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f970c011890 tx=0x7f970c0046c0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.410 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:42:53.410 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:42:53.410 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:42:53.410 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:42:53.411 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 14, 2023-11-28T17:42:53.411 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:42:53.411 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:42:53.411 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:42:53.411 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:42:53.411 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.411 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 12, 2023-11-28T17:42:53.411 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:53.411 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:42:53.411 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:42:53.411 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:53.411 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:53.412 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:53.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:53.412 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:53.412 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:53.412 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:53.412 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:53.412 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:53.412 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:53.412 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.412 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:42:53.412 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:42:53.413 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:53.413 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:53.413 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:53.413 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:53.413 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:53.413 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:53.413 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:53.413 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:53.413 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:53.413 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:53.413 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.414 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:42:53.414 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:42:53.414 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:42:53.414 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:42:53.414 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:42:53.414 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:42:53.414 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:53.414 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:53.415 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:42:53.415 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:42:53.415 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:42:53.415 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:42:53.415 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:42:53.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:42:53.416 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:42:53.416 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:53.416 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:42:53.416 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:53.416 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:53.417 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:53.417 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:53.417 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:53.417 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:53.417 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:53.418 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:53.418 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:53.418 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:53.418 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.418 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:42:53.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.419 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:42:53.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.419 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:42:53.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:42:53.420 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:53.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:53.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:53.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:42:53.421 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:53.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:42:53.421 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.421 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.421 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:42:53.421 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:42:53.422 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:53.422 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:53.422 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:53.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:42:53.422 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:42:53.422 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:53.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:53.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:53.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:42:53.423 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:53.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:42:53.423 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:42:53.423 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:42:53.424 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:53.424 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:53.424 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:53.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:42:53.424 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:42:53.424 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:53.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:53.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:53.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:42:53.425 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:53.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:42:53.425 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.425 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:42:53.426 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:42:53.426 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:53.426 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:53.426 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:53.426 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.426 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.426 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:42:53.426 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:42:53.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.427 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:53.427 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:53.427 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:53.427 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.427 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.427 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:42:53.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.427 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:53.427 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:53.427 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:42:53.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.428 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.428 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:42:53.428 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:42:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.408+0000 7f971b59e640 1 -- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f96fc06b340 msgr2=0x7f96fc06d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.408+0000 7f971b59e640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f96fc06b340 0x7f96fc06d800 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9714035000 tx=0x7f9714030000 comp rx=0 tx=0).stop 2023-11-28T17:42:53.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.408+0000 7f971b59e640 1 -- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c7e30 msgr2=0x7f971c14dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.408+0000 7f971b59e640 1 --2- 172.21.15.98:0/801236911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f971c0c7e30 0x7f971c14dc20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f971000c450 tx=0x7f971000c910 comp rx=0 tx=0).stop 2023-11-28T17:42:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.408+0000 7f971ad9d640 1 -- 172.21.15.98:0/801236911 reap_dead start 2023-11-28T17:42:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.408+0000 7f971ad9d640 1 -- 172.21.15.98:0/801236911 reap_dead start 2023-11-28T17:42:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.409+0000 7f971b59e640 1 -- 172.21.15.98:0/801236911 shutdown_connections 2023-11-28T17:42:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.409+0000 7f971b59e640 1 -- 172.21.15.98:0/801236911 >> 172.21.15.98:0/801236911 conn(0x7f971c0cf300 msgr2=0x7f971c0cf6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.409+0000 7f971b59e640 1 -- 172.21.15.98:0/801236911 shutdown_connections 2023-11-28T17:42:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.409+0000 7f971b59e640 1 -- 172.21.15.98:0/801236911 wait complete. 2023-11-28T17:42:53.430 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:42:53.431 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:42:53.431 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:42:53.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.606+0000 7f55be644640 1 Processor -- start 2023-11-28T17:42:53.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.607+0000 7f55be644640 1 -- start start 2023-11-28T17:42:53.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.608+0000 7f55be644640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55b8151be0 0x7f55b80c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.608+0000 7f55be644640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f55b80c5a60 0x7f55b81584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.608+0000 7f55be644640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b8158a10 con 0x7f55b80c4730 2023-11-28T17:42:53.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.608+0000 7f55be644640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b80c8150 con 0x7f55b80c5a60 2023-11-28T17:42:53.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.608+0000 7f55be644640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b80c8850 con 0x7f55b8151be0 2023-11-28T17:42:53.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.609+0000 7f55b77fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f55b80c5a60 0x7f55b81584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.609+0000 7f55b7fff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55b8151be0 0x7f55b80c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.609+0000 7f55b77fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f55b80c5a60 0x7f55b81584d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:40230/0 (socket says 172.21.15.98:40230) 2023-11-28T17:42:53.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.609+0000 7f55b77fe640 1 -- 172.21.15.98:0/1324209830 learned_addr learned my addr 172.21.15.98:0/1324209830 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:53.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.609+0000 7f55b7fff640 1 -- 172.21.15.98:0/1324209830 >> v2:172.21.15.113:3300/0 conn(0x7f55b80c5a60 msgr2=0x7f55b81584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:53.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.609+0000 7f55b7fff640 1 --2- 172.21.15.98:0/1324209830 >> v2:172.21.15.113:3300/0 conn(0x7f55b80c5a60 0x7f55b81584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.609+0000 7f55b7fff640 1 -- 172.21.15.98:0/1324209830 >> v1:172.21.15.113:6789/0 conn(0x7f55b80c4730 legacy=0x7f55b806dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.609+0000 7f55b7fff640 1 -- 172.21.15.98:0/1324209830 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b80c5f50 con 0x7f55b8151be0 2023-11-28T17:42:53.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.611+0000 7f55b7fff640 1 --2- 172.21.15.98:0/1324209830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55b8151be0 0x7f55b80c75a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f55a80097a0 tx=0x7f55a802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=829555bbaa0c0c30 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:53.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.611+0000 7f55b6ffd640 1 -- 172.21.15.98:0/1324209830 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a802d040 con 0x7f55b8151be0 2023-11-28T17:42:53.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.611+0000 7f55b6ffd640 1 -- 172.21.15.98:0/1324209830 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a8031810 con 0x7f55b8151be0 2023-11-28T17:42:53.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.611+0000 7f55b6ffd640 1 -- 172.21.15.98:0/1324209830 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a8031ba0 con 0x7f55b8151be0 2023-11-28T17:42:53.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.612+0000 7f55be644640 1 -- 172.21.15.98:0/1324209830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55b8151be0 msgr2=0x7f55b80c75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.612+0000 7f55be644640 1 --2- 172.21.15.98:0/1324209830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55b8151be0 0x7f55b80c75a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f55a80097a0 tx=0x7f55a802b8b0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.612+0000 7f55be644640 1 -- 172.21.15.98:0/1324209830 shutdown_connections 2023-11-28T17:42:53.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.612+0000 7f55be644640 1 --2- 172.21.15.98:0/1324209830 >> v2:172.21.15.113:3300/0 conn(0x7f55b80c5a60 0x7f55b81584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.612+0000 7f55be644640 1 --2- 172.21.15.98:0/1324209830 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55b8151be0 0x7f55b80c75a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.612+0000 7f55be644640 1 -- 172.21.15.98:0/1324209830 >> 172.21.15.98:0/1324209830 conn(0x7f55b806c0f0 msgr2=0x7f55b806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:53.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.613+0000 7f55be644640 1 -- 172.21.15.98:0/1324209830 shutdown_connections 2023-11-28T17:42:53.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.613+0000 7f55be644640 1 -- 172.21.15.98:0/1324209830 wait complete. 2023-11-28T17:42:53.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.615+0000 7f55be644640 1 Processor -- start 2023-11-28T17:42:53.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.615+0000 7f55be644640 1 -- start start 2023-11-28T17:42:53.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.617+0000 7f55be644640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b80c4730 0x7f55b8152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.617+0000 7f55be644640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55b80c5a60 0x7f55b81f0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.617+0000 7f55be644640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b8151be0 0x7f55b80c2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.617+0000 7f55be644640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b81f0140 con 0x7f55b80c4730 2023-11-28T17:42:53.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.617+0000 7f55be644640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b80715b0 con 0x7f55b8151be0 2023-11-28T17:42:53.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.617+0000 7f55be644640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b80c3df0 con 0x7f55b80c5a60 2023-11-28T17:42:53.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.617+0000 7f55b7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b80c4730 0x7f55b8152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.617+0000 7f55b7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b80c4730 0x7f55b8152370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44558/0 (socket says 172.21.15.98:44558) 2023-11-28T17:42:53.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.617+0000 7f55b7fff640 1 -- 172.21.15.98:0/4260960201 learned_addr learned my addr 172.21.15.98:0/4260960201 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:53.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.618+0000 7f55bcbe1640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b8151be0 0x7f55b80c2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.618+0000 7f55b77fe640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55b80c5a60 0x7f55b81f0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.618+0000 7f55b7fff640 1 -- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55b80c5a60 msgr2=0x7f55b81f0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.618+0000 7f55b7fff640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55b80c5a60 0x7f55b81f0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.618+0000 7f55b7fff640 1 -- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b8151be0 msgr2=0x7f55b80c2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.618+0000 7f55b7fff640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b8151be0 0x7f55b80c2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.618+0000 7f55b7fff640 1 -- 172.21.15.98:0/4260960201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a8009400 con 0x7f55b80c4730 2023-11-28T17:42:53.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.618+0000 7f55b77fe640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f55b80c5a60 0x7f55b81f0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:42:53.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.619+0000 7f55b7fff640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b80c4730 0x7f55b8152370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f55a8000c00 tx=0x7f55a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:53.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.619+0000 7f55b4ff9640 1 -- 172.21.15.98:0/4260960201 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a8006e10 con 0x7f55b80c4730 2023-11-28T17:42:53.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.619+0000 7f55b4ff9640 1 -- 172.21.15.98:0/4260960201 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a803ad60 con 0x7f55b80c4730 2023-11-28T17:42:53.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.619+0000 7f55b4ff9640 1 -- 172.21.15.98:0/4260960201 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a8044630 con 0x7f55b80c4730 2023-11-28T17:42:53.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.619+0000 7f55be644640 1 -- 172.21.15.98:0/4260960201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b80c40d0 con 0x7f55b80c4730 2023-11-28T17:42:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.619+0000 7f55be644640 1 -- 172.21.15.98:0/4260960201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b81efd60 con 0x7f55b80c4730 2023-11-28T17:42:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.620+0000 7f55be644640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b81554d0 0x7f55b80c2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.620+0000 7f55be644640 1 -- 172.21.15.98:0/4260960201 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b80744d0 con 0x7f55b81554d0 2023-11-28T17:42:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.621+0000 7f55b77fe640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b81554d0 0x7f55b80c2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.622+0000 7f55b77fe640 1 -- 172.21.15.98:0/4260960201 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f55b80c6240 con 0x7f55b81554d0 2023-11-28T17:42:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.622+0000 7f55b77fe640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b81554d0 0x7f55b80c2cd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f55b80717e0 tx=0x7f55ac011c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.622+0000 7f55b4ff9640 1 -- 172.21.15.98:0/4260960201 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac00fa00 con 0x7f55b81554d0 2023-11-28T17:42:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.622+0000 7f55b4ff9640 1 -- 172.21.15.98:0/4260960201 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f55a804d450 con 0x7f55b80c4730 2023-11-28T17:42:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.623+0000 7f55b4ff9640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f55a406b340 0x7f55a406d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.624+0000 7f55b4ff9640 1 -- 172.21.15.98:0/4260960201 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f55a80abde0 con 0x7f55b80c4730 2023-11-28T17:42:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.624+0000 7f55b4ff9640 1 -- 172.21.15.98:0/4260960201 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f55ac00fb60 con 0x7f55b81554d0 2023-11-28T17:42:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.624+0000 7f55b4ff9640 1 -- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b81554d0 msgr2=0x7f55b80c2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.624+0000 7f55b4ff9640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b81554d0 0x7f55b80c2cd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f55b80717e0 tx=0x7f55ac011c90 comp rx=0 tx=0).stop 2023-11-28T17:42:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.624+0000 7f55b4ff9640 1 -- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b81554d0 msgr2=0x7f55b80c2cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.624+0000 7f55b4ff9640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b81554d0 0x7f55b80c2cd0 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f55b80717e0 tx=0x7f55ac011c90 comp rx=0 tx=0).stop 2023-11-28T17:42:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.624+0000 7f55bcbe1640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f55a406b340 0x7f55a406d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.625+0000 7f55bcbe1640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f55a406b340 0x7f55a406d800 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f55b00012e0 tx=0x7f55b0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.631+0000 7f55be644640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b80c06c0 0x7f55b80c0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.631+0000 7f55be644640 1 -- 172.21.15.98:0/4260960201 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b81eff50 con 0x7f55b80c06c0 2023-11-28T17:42:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.632+0000 7f55b77fe640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b80c06c0 0x7f55b80c0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.632+0000 7f55b77fe640 1 -- 172.21.15.98:0/4260960201 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f55b80c84e0 con 0x7f55b80c06c0 2023-11-28T17:42:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.633+0000 7f55b77fe640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b80c06c0 0x7f55b80c0ad0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f55b8071260 tx=0x7f55ac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:53.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.633+0000 7f55b4ff9640 1 -- 172.21.15.98:0/4260960201 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac021040 con 0x7f55b80c06c0 2023-11-28T17:42:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.634+0000 7f55b4ff9640 1 -- 172.21.15.98:0/4260960201 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4806 (secure 0 0 0) 0x7f55ac026070 con 0x7f55b80c06c0 2023-11-28T17:42:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.634+0000 7f55b4ff9640 1 -- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b80c06c0 msgr2=0x7f55b80c0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.634+0000 7f55b4ff9640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b80c06c0 0x7f55b80c0ad0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f55b8071260 tx=0x7f55ac0046c0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.634+0000 7f55b4ff9640 1 -- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b80c06c0 msgr2=0x7f55b80c0ad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.634+0000 7f55b4ff9640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f55b80c06c0 0x7f55b80c0ad0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f55b8071260 tx=0x7f55ac0046c0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.636 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:42:53.637 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:42:53.637 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:42:53.637 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:42:53.637 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 14, 2023-11-28T17:42:53.637 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:42:53.637 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:42:53.637 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:42:53.638 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:42:53.638 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.638 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 12, 2023-11-28T17:42:53.638 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:53.638 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:42:53.638 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:42:53.638 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:53.639 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:53.639 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:53.639 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:53.639 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:53.639 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:53.639 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:53.639 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:53.639 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:53.639 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:53.639 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.639 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:42:53.640 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:42:53.640 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:53.640 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:53.640 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:53.640 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:53.640 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:53.640 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:53.640 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:53.640 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:53.640 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:53.640 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:53.640 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.641 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:42:53.641 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:42:53.641 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:42:53.641 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:42:53.641 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:42:53.641 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:42:53.641 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:53.641 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:53.641 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:42:53.641 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:42:53.641 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:42:53.641 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:42:53.641 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:42:53.642 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:42:53.642 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:42:53.642 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:53.642 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:42:53.642 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:53.642 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:53.642 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:53.642 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:53.642 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:53.642 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:53.642 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:53.642 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:53.643 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:53.643 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:53.643 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.643 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:42:53.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.643 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:42:53.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.643 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:42:53.643 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:42:53.643 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:53.643 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:53.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.644 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:53.644 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:42:53.644 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.644 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:53.644 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:42:53.644 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.644 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.644 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.644 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:42:53.645 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:42:53.645 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:53.645 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:53.645 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:53.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.645 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:42:53.645 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:42:53.645 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:53.645 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:53.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.645 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:53.645 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:42:53.646 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.646 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:53.646 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:42:53.646 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.646 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.646 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.646 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:42:53.646 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:42:53.646 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:53.648 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:42:53.648 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.648 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.648 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.648 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:42:53.648 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:42:53.648 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:53.648 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:53.648 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:53.648 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.649 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.649 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:42:53.649 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:42:53.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.649 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:53.649 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:53.649 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:53.649 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.649 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.649 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:42:53.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.650 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:53.650 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:53.650 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:42:53.650 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.650 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.650 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:42:53.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.650 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:53.650 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:53.650 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:42:53.650 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.650 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.650 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:42:53.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.651 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:53.651 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:53.651 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:42:53.651 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.651 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.651 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:42:53.651 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:42:53.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.635+0000 7f55be644640 1 -- 172.21.15.98:0/4260960201 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f55a406b340 msgr2=0x7f55a406d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.635+0000 7f55be644640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f55a406b340 0x7f55a406d800 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f55b00012e0 tx=0x7f55b0009040 comp rx=0 tx=0).stop 2023-11-28T17:42:53.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.635+0000 7f55be644640 1 -- 172.21.15.98:0/4260960201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b80c4730 msgr2=0x7f55b8152370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.635+0000 7f55be644640 1 --2- 172.21.15.98:0/4260960201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55b80c4730 0x7f55b8152370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f55a8000c00 tx=0x7f55a80046c0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.635+0000 7f55bcbe1640 1 -- 172.21.15.98:0/4260960201 reap_dead start 2023-11-28T17:42:53.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.635+0000 7f55bcbe1640 1 -- 172.21.15.98:0/4260960201 reap_dead start 2023-11-28T17:42:53.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.635+0000 7f55be644640 1 -- 172.21.15.98:0/4260960201 shutdown_connections 2023-11-28T17:42:53.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.635+0000 7f55be644640 1 -- 172.21.15.98:0/4260960201 >> 172.21.15.98:0/4260960201 conn(0x7f55b806c0f0 msgr2=0x7f55b806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:53.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.635+0000 7f55be644640 1 -- 172.21.15.98:0/4260960201 shutdown_connections 2023-11-28T17:42:53.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.635+0000 7f55be644640 1 -- 172.21.15.98:0/4260960201 wait complete. 2023-11-28T17:42:53.654 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:42:53.654 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:42:53.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:42:53.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.826+0000 7fca84b43640 1 Processor -- start 2023-11-28T17:42:53.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.826+0000 7fca84b43640 1 -- start start 2023-11-28T17:42:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.828+0000 7fca84b43640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fca800c5a60 0x7fca800c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.828+0000 7fca84b43640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca801584d0 0x7fca8006dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.828+0000 7fca84b43640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca800c8150 con 0x7fca80151be0 2023-11-28T17:42:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.828+0000 7fca84b43640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca800c8850 con 0x7fca800c5a60 2023-11-28T17:42:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.828+0000 7fca84b43640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca800c5f50 con 0x7fca801584d0 2023-11-28T17:42:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.828+0000 7fca7dd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fca800c5a60 0x7fca800c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.828+0000 7fca7dd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fca800c5a60 0x7fca800c75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44570/0 (socket says 172.21.15.98:44570) 2023-11-28T17:42:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.828+0000 7fca7dd9b640 1 -- 172.21.15.98:0/3971595051 learned_addr learned my addr 172.21.15.98:0/3971595051 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.828+0000 7fca7ed9d640 1 --2- 172.21.15.98:0/3971595051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca801584d0 0x7fca8006dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.829+0000 7fca7dd9b640 1 -- 172.21.15.98:0/3971595051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca801584d0 msgr2=0x7fca8006dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.829+0000 7fca7dd9b640 1 --2- 172.21.15.98:0/3971595051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca801584d0 0x7fca8006dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.829+0000 7fca7dd9b640 1 -- 172.21.15.98:0/3971595051 >> v1:172.21.15.113:6789/0 conn(0x7fca80151be0 legacy=0x7fca800c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:53.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.829+0000 7fca7dd9b640 1 -- 172.21.15.98:0/3971595051 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca800c7b90 con 0x7fca800c5a60 2023-11-28T17:42:53.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.830+0000 7fca7dd9b640 1 --2- 172.21.15.98:0/3971595051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca800c5a60 0x7fca800c75a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fca74009870 tx=0x7fca7402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632bf37fa8504f53 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:53.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.830+0000 7fca7d59a640 1 -- 172.21.15.98:0/3971595051 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca74031640 con 0x7fca800c5a60 2023-11-28T17:42:53.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.830+0000 7fca7d59a640 1 -- 172.21.15.98:0/3971595051 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca740317a0 con 0x7fca800c5a60 2023-11-28T17:42:53.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.830+0000 7fca7d59a640 1 -- 172.21.15.98:0/3971595051 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca74031b30 con 0x7fca800c5a60 2023-11-28T17:42:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.831+0000 7fca84b43640 1 -- 172.21.15.98:0/3971595051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca800c5a60 msgr2=0x7fca800c75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.831+0000 7fca84b43640 1 --2- 172.21.15.98:0/3971595051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca800c5a60 0x7fca800c75a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fca74009870 tx=0x7fca7402b8f0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.831+0000 7fca84b43640 1 -- 172.21.15.98:0/3971595051 shutdown_connections 2023-11-28T17:42:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.831+0000 7fca84b43640 1 --2- 172.21.15.98:0/3971595051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca801584d0 0x7fca8006dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.831+0000 7fca84b43640 1 --2- 172.21.15.98:0/3971595051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca800c5a60 0x7fca800c75a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.831+0000 7fca84b43640 1 -- 172.21.15.98:0/3971595051 >> 172.21.15.98:0/3971595051 conn(0x7fca8006c0f0 msgr2=0x7fca8006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.831+0000 7fca84b43640 1 -- 172.21.15.98:0/3971595051 shutdown_connections 2023-11-28T17:42:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.831+0000 7fca84b43640 1 -- 172.21.15.98:0/3971595051 wait complete. 2023-11-28T17:42:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.833+0000 7fca84b43640 1 Processor -- start 2023-11-28T17:42:53.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.833+0000 7fca84b43640 1 -- start start 2023-11-28T17:42:53.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.834+0000 7fca84b43640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800c5a60 0x7fca80154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.834+0000 7fca84b43640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca80151be0 0x7fca801f30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.834+0000 7fca84b43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca801584d0 0x7fca8014e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.834+0000 7fca84b43640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca801541c0 con 0x7fca801584d0 2023-11-28T17:42:53.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.834+0000 7fca84b43640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca80154a60 con 0x7fca80151be0 2023-11-28T17:42:53.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.834+0000 7fca84b43640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca801f3b90 con 0x7fca800c5a60 2023-11-28T17:42:53.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.834+0000 7fca7ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca801584d0 0x7fca8014e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.834+0000 7fca7ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca801584d0 0x7fca8014e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44578/0 (socket says 172.21.15.98:44578) 2023-11-28T17:42:53.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.834+0000 7fca7ed9d640 1 -- 172.21.15.98:0/154169057 learned_addr learned my addr 172.21.15.98:0/154169057 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:53.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.835+0000 7fca7dd9b640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca80151be0 0x7fca801f30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.835+0000 7fca7e59c640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800c5a60 0x7fca80154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.835+0000 7fca7ed9d640 1 -- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800c5a60 msgr2=0x7fca80154ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.835+0000 7fca7ed9d640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800c5a60 0x7fca80154ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.835+0000 7fca7ed9d640 1 -- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca80151be0 msgr2=0x7fca801f30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.835+0000 7fca7ed9d640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fca80151be0 0x7fca801f30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:53.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.835+0000 7fca7ed9d640 1 -- 172.21.15.98:0/154169057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca740094d0 con 0x7fca801584d0 2023-11-28T17:42:53.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.835+0000 7fca7ed9d640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca801584d0 0x7fca8014e8b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fca7800efc0 tx=0x7fca7800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:53.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.836+0000 7fca6effd640 1 -- 172.21.15.98:0/154169057 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca78014480 con 0x7fca801584d0 2023-11-28T17:42:53.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.836+0000 7fca6effd640 1 -- 172.21.15.98:0/154169057 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7800f040 con 0x7fca801584d0 2023-11-28T17:42:53.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.836+0000 7fca6effd640 1 -- 172.21.15.98:0/154169057 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca78014820 con 0x7fca801584d0 2023-11-28T17:42:53.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.836+0000 7fca84b43640 1 -- 172.21.15.98:0/154169057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca80154410 con 0x7fca801584d0 2023-11-28T17:42:53.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.836+0000 7fca84b43640 1 -- 172.21.15.98:0/154169057 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca80153220 con 0x7fca801584d0 2023-11-28T17:42:53.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.837+0000 7fca84b43640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800ca930 0x7fca8014edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.837+0000 7fca84b43640 1 -- 172.21.15.98:0/154169057 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca800744d0 con 0x7fca800ca930 2023-11-28T17:42:53.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.838+0000 7fca7e59c640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800ca930 0x7fca8014edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.838+0000 7fca7e59c640 1 -- 172.21.15.98:0/154169057 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fca800c6240 con 0x7fca800ca930 2023-11-28T17:42:53.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.839+0000 7fca7e59c640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800ca930 0x7fca8014edf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fca7000e4d0 tx=0x7fca7000e990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:53.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.839+0000 7fca6effd640 1 -- 172.21.15.98:0/154169057 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70007240 con 0x7fca800ca930 2023-11-28T17:42:53.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.839+0000 7fca6effd640 1 -- 172.21.15.98:0/154169057 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fca78014980 con 0x7fca801584d0 2023-11-28T17:42:53.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.840+0000 7fca6effd640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fca6006b340 0x7fca6006d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.841+0000 7fca6effd640 1 -- 172.21.15.98:0/154169057 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fca7808a880 con 0x7fca801584d0 2023-11-28T17:42:53.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.841+0000 7fca7dd9b640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fca6006b340 0x7fca6006d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.841+0000 7fca6effd640 1 -- 172.21.15.98:0/154169057 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fca700073a0 con 0x7fca800ca930 2023-11-28T17:42:53.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.841+0000 7fca6effd640 1 -- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800ca930 msgr2=0x7fca8014edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.841+0000 7fca6effd640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800ca930 0x7fca8014edf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fca7000e4d0 tx=0x7fca7000e990 comp rx=0 tx=0).stop 2023-11-28T17:42:53.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.841+0000 7fca6effd640 1 -- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800ca930 msgr2=0x7fca8014edf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:53.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.841+0000 7fca6effd640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800ca930 0x7fca8014edf0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fca7000e4d0 tx=0x7fca7000e990 comp rx=0 tx=0).stop 2023-11-28T17:42:53.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.842+0000 7fca7dd9b640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fca6006b340 0x7fca6006d800 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fca74036000 tx=0x7fca7403c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:53.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.848+0000 7fca84b43640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800c26c0 0x7fca801ef8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:53.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.848+0000 7fca84b43640 1 -- 172.21.15.98:0/154169057 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca80153410 con 0x7fca800c26c0 2023-11-28T17:42:53.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.849+0000 7fca7e59c640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800c26c0 0x7fca801ef8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:53.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.849+0000 7fca7e59c640 1 -- 172.21.15.98:0/154169057 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fca80158960 con 0x7fca800c26c0 2023-11-28T17:42:53.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.850+0000 7fca7e59c640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800c26c0 0x7fca801ef8d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fca7000e4d0 tx=0x7fca70003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:53.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.850+0000 7fca6effd640 1 -- 172.21.15.98:0/154169057 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7001e960 con 0x7fca800c26c0 2023-11-28T17:42:53.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.851+0000 7fca6effd640 1 -- 172.21.15.98:0/154169057 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4976 (secure 0 0 0) 0x7fca70006e20 con 0x7fca800c26c0 2023-11-28T17:42:53.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.851+0000 7fca6effd640 1 -- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800c26c0 msgr2=0x7fca801ef8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.851+0000 7fca6effd640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800c26c0 0x7fca801ef8d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fca7000e4d0 tx=0x7fca70003760 comp rx=0 tx=0).stop 2023-11-28T17:42:53.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.851+0000 7fca6effd640 1 -- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800c26c0 msgr2=0x7fca801ef8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:42:53.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.851+0000 7fca6effd640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fca800c26c0 0x7fca801ef8d0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fca7000e4d0 tx=0x7fca70003760 comp rx=0 tx=0).stop 2023-11-28T17:42:53.853 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:42:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:42:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:42:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:42:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 14, 2023-11-28T17:42:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:42:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:42:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:42:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:42:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 12, 2023-11-28T17:42:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:42:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:42:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:42:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:42:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:42:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:42:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:42:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:42:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:42:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:42:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:42:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:42:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:42:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:42:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:42:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:42:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:42:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:42:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:42:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:42:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:42:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:42:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:42:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:42:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:42:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:42:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:42:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:42:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:42:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:42:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:42:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:42:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:42:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:42:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:42:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:42:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:42:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:42:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:53.862 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:42:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:42:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:42:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:42:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:42:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:42:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:53.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:42:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:42:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:42:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:42:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:42:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:42:53.865 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:42:53.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:42:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:42:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.867 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:42:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:42:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:42:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:42:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:42:53.868 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:42:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:42:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:53.869 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:42:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:53.870 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:42:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:42:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:42:53.871 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:42:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:42:53.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:42:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:42:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:42:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:42:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:42:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:42:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:42:53.873 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:42:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:42:53.874 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:42:53.874 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:42:53.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.852+0000 7fca84b43640 1 -- 172.21.15.98:0/154169057 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fca6006b340 msgr2=0x7fca6006d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.852+0000 7fca84b43640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fca6006b340 0x7fca6006d800 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fca74036000 tx=0x7fca7403c040 comp rx=0 tx=0).stop 2023-11-28T17:42:53.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.852+0000 7fca84b43640 1 -- 172.21.15.98:0/154169057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca801584d0 msgr2=0x7fca8014e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:53.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.852+0000 7fca84b43640 1 --2- 172.21.15.98:0/154169057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca801584d0 0x7fca8014e8b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fca7800efc0 tx=0x7fca7800c490 comp rx=0 tx=0).stop 2023-11-28T17:42:53.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.852+0000 7fca7ed9d640 1 -- 172.21.15.98:0/154169057 reap_dead start 2023-11-28T17:42:53.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.852+0000 7fca7ed9d640 1 -- 172.21.15.98:0/154169057 reap_dead start 2023-11-28T17:42:53.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.852+0000 7fca84b43640 1 -- 172.21.15.98:0/154169057 shutdown_connections 2023-11-28T17:42:53.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.852+0000 7fca84b43640 1 -- 172.21.15.98:0/154169057 >> 172.21.15.98:0/154169057 conn(0x7fca8006c0f0 msgr2=0x7fca80070270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:53.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.852+0000 7fca84b43640 1 -- 172.21.15.98:0/154169057 shutdown_connections 2023-11-28T17:42:53.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:53.852+0000 7fca84b43640 1 -- 172.21.15.98:0/154169057 wait complete. 2023-11-28T17:42:53.876 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:42:53.877 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-11-28T17:42:53.877 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-11-28T17:42:53.877 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-11-28T17:42:53.877 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-11-28T17:42:53.877 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-28T17:42:53.877 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:42:53.917 INFO:tasks.ceph.mon.b:Stopped 2023-11-28T17:42:53.917 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-11-28T17:42:53.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:42:54.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.089+0000 7f41ac968640 1 Processor -- start 2023-11-28T17:42:54.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.089+0000 7f41ac968640 1 -- start start 2023-11-28T17:42:54.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.091+0000 7f41ac968640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41a8154040 0x7f41a815dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:54.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.091+0000 7f41ac968640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f41a80c9000 0x7f41a80c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:54.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.091+0000 7f41ac968640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a8159c50 con 0x7f41a80c9420 2023-11-28T17:42:54.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.091+0000 7f41ac968640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a80c2ad0 con 0x7f41a80c9000 2023-11-28T17:42:54.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.091+0000 7f41ac968640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a8154420 con 0x7f41a8154040 2023-11-28T17:42:54.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.091+0000 7f41a5d9b640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f41a80c9000 msgr2=0x7f41a80c7ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:42:54.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.092+0000 7f41a6d9d640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f41a80c9420 legacy=0x7f41a815a150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:42:54.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.092+0000 7f41a5d9b640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f41a80c9000 0x7f41a80c7ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:54.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.092+0000 7f41a659c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41a8154040 0x7f41a815dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:54.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.092+0000 7f41a659c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41a8154040 0x7f41a815dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:32864/0 (socket says 172.21.15.98:32864) 2023-11-28T17:42:54.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.092+0000 7f41a659c640 1 -- 172.21.15.98:0/555439042 learned_addr learned my addr 172.21.15.98:0/555439042 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:54.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.093+0000 7f41a659c640 1 -- 172.21.15.98:0/555439042 >> v2:172.21.15.113:3300/0 conn(0x7f41a80c9000 msgr2=0x7f41a80c7ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:54.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.093+0000 7f41a659c640 1 --2- 172.21.15.98:0/555439042 >> v2:172.21.15.113:3300/0 conn(0x7f41a80c9000 0x7f41a80c7ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:54.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.093+0000 7f41a659c640 1 -- 172.21.15.98:0/555439042 >> v1:172.21.15.113:6789/0 conn(0x7f41a80c9420 legacy=0x7f41a815a150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:54.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.093+0000 7f41a659c640 1 -- 172.21.15.98:0/555439042 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80cab90 con 0x7f41a8154040 2023-11-28T17:42:54.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.094+0000 7f41a659c640 1 --2- 172.21.15.98:0/555439042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41a8154040 0x7f41a815dd70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4198009870 tx=0x7f419802b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89779ed8f2df4e47 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:54.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.095+0000 7f41a559a640 1 -- 172.21.15.98:0/555439042 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4198030860 con 0x7f41a8154040 2023-11-28T17:42:54.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.095+0000 7f41a559a640 1 -- 172.21.15.98:0/555439042 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41980309c0 con 0x7f41a8154040 2023-11-28T17:42:54.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.095+0000 7f41a559a640 1 -- 172.21.15.98:0/555439042 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4198030d50 con 0x7f41a8154040 2023-11-28T17:42:54.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.096+0000 7f41ac968640 1 -- 172.21.15.98:0/555439042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41a8154040 msgr2=0x7f41a815dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:54.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.096+0000 7f41ac968640 1 --2- 172.21.15.98:0/555439042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41a8154040 0x7f41a815dd70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4198009870 tx=0x7f419802b9a0 comp rx=0 tx=0).stop 2023-11-28T17:42:54.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.096+0000 7f41ac968640 1 -- 172.21.15.98:0/555439042 shutdown_connections 2023-11-28T17:42:54.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.096+0000 7f41ac968640 1 --2- 172.21.15.98:0/555439042 >> v2:172.21.15.113:3300/0 conn(0x7f41a80c9000 0x7f41a80c7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:54.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.096+0000 7f41ac968640 1 --2- 172.21.15.98:0/555439042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41a8154040 0x7f41a815dd70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:54.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.096+0000 7f41ac968640 1 -- 172.21.15.98:0/555439042 >> 172.21.15.98:0/555439042 conn(0x7f41a806a4d0 msgr2=0x7f41a806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:54.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.096+0000 7f41ac968640 1 -- 172.21.15.98:0/555439042 shutdown_connections 2023-11-28T17:42:54.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.096+0000 7f41ac968640 1 -- 172.21.15.98:0/555439042 wait complete. 2023-11-28T17:42:54.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.098+0000 7f41ac968640 1 Processor -- start 2023-11-28T17:42:54.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.098+0000 7f41ac968640 1 -- start start 2023-11-28T17:42:54.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.099+0000 7f41ac968640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41a80c9000 0x7f41a80c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:54.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.099+0000 7f41ac968640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41a80c9420 0x7f41a8152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:54.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.099+0000 7f41ac968640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41a8154040 0x7f41a81f1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:54.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.099+0000 7f41ac968640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a80c4e10 con 0x7f41a80c9420 2023-11-28T17:42:54.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.099+0000 7f41ac968640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a81e0740 con 0x7f41a8154040 2023-11-28T17:42:54.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.099+0000 7f41ac968640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a81f3b70 con 0x7f41a80c9000 2023-11-28T17:42:54.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.099+0000 7f41a6d9d640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41a8154040 msgr2=0x7f41a81f1e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:42:54.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.099+0000 7f41a6d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41a8154040 0x7f41a81f1e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:54.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.099+0000 7f41a5d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41a80c9420 0x7f41a8152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:54.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.100+0000 7f41a5d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41a80c9420 0x7f41a8152c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44642/0 (socket says 172.21.15.98:44642) 2023-11-28T17:42:54.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.100+0000 7f41a5d9b640 1 -- 172.21.15.98:0/3723859859 learned_addr learned my addr 172.21.15.98:0/3723859859 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:54.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.100+0000 7f41a659c640 1 --2- 172.21.15.98:0/3723859859 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41a80c9000 0x7f41a80c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:54.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.100+0000 7f41a5d9b640 1 -- 172.21.15.98:0/3723859859 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41a80c9000 msgr2=0x7f41a80c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:54.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.100+0000 7f41a5d9b640 1 --2- 172.21.15.98:0/3723859859 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41a80c9000 0x7f41a80c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:54.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.100+0000 7f41a5d9b640 1 -- 172.21.15.98:0/3723859859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41a8154040 msgr2=0x7f41a81f1e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:42:54.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.100+0000 7f41a5d9b640 1 --2- 172.21.15.98:0/3723859859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41a8154040 0x7f41a81f1e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:54.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.100+0000 7f41a5d9b640 1 -- 172.21.15.98:0/3723859859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41980094d0 con 0x7f41a80c9420 2023-11-28T17:42:54.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.101+0000 7f41a5d9b640 1 --2- 172.21.15.98:0/3723859859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41a80c9420 0x7f41a8152c30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f419c00b550 tx=0x7f419c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:54.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.101+0000 7f4196ffd640 1 -- 172.21.15.98:0/3723859859 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c003dc0 con 0x7f41a80c9420 2023-11-28T17:42:54.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.101+0000 7f4196ffd640 1 -- 172.21.15.98:0/3723859859 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419c005070 con 0x7f41a80c9420 2023-11-28T17:42:54.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.101+0000 7f41ac968640 1 -- 172.21.15.98:0/3723859859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a81f3dd0 con 0x7f41a80c9420 2023-11-28T17:42:54.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.101+0000 7f4196ffd640 1 -- 172.21.15.98:0/3723859859 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c005420 con 0x7f41a80c9420 2023-11-28T17:42:54.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.101+0000 7f41ac968640 1 -- 172.21.15.98:0/3723859859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a81f4070 con 0x7f41a80c9420 2023-11-28T17:42:54.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.103+0000 7f41ac968640 1 -- 172.21.15.98:0/3723859859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a8067bf0 con 0x7f41a80c9420 2023-11-28T17:42:54.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.103+0000 7f4196ffd640 1 -- 172.21.15.98:0/3723859859 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f419c005580 con 0x7f41a80c9420 2023-11-28T17:42:54.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.105+0000 7f4196ffd640 1 --2- 172.21.15.98:0/3723859859 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f418806b250 0x7f418806d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:54.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.105+0000 7f4196ffd640 1 -- 172.21.15.98:0/3723859859 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f419c089c40 con 0x7f41a80c9420 2023-11-28T17:42:54.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.105+0000 7f41a659c640 1 --2- 172.21.15.98:0/3723859859 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f418806b250 0x7f418806d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:54.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.106+0000 7f41a659c640 1 --2- 172.21.15.98:0/3723859859 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f418806b250 0x7f418806d710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4198036000 tx=0x7f4198008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:54.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.110+0000 7f4196ffd640 1 -- 172.21.15.98:0/3723859859 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f419c054900 con 0x7f41a80c9420 2023-11-28T17:42:54.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.405+0000 7f41ac968640 1 -- 172.21.15.98:0/3723859859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f41a81f1b80 con 0x7f41a80c9420 2023-11-28T17:42:54.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.406+0000 7f4196ffd640 1 -- 172.21.15.98:0/3723859859 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f41a81f1b80 con 0x7f41a80c9420 2023-11-28T17:42:54.407 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:42:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.408+0000 7f41ac968640 1 -- 172.21.15.98:0/3723859859 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f418806b250 msgr2=0x7f418806d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.408+0000 7f41ac968640 1 --2- 172.21.15.98:0/3723859859 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f418806b250 0x7f418806d710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4198036000 tx=0x7f4198008030 comp rx=0 tx=0).stop 2023-11-28T17:42:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.408+0000 7f41ac968640 1 -- 172.21.15.98:0/3723859859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41a80c9420 msgr2=0x7f41a8152c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.408+0000 7f41ac968640 1 --2- 172.21.15.98:0/3723859859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41a80c9420 0x7f41a8152c30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f419c00b550 tx=0x7f419c00ba10 comp rx=0 tx=0).stop 2023-11-28T17:42:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.408+0000 7f41ac968640 1 -- 172.21.15.98:0/3723859859 shutdown_connections 2023-11-28T17:42:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.408+0000 7f41ac968640 1 --2- 172.21.15.98:0/3723859859 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f418806b250 0x7f418806d710 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.408+0000 7f41ac968640 1 --2- 172.21.15.98:0/3723859859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41a8154040 0x7f41a81f1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.408+0000 7f41ac968640 1 --2- 172.21.15.98:0/3723859859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41a80c9420 0x7f41a8152c30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.408+0000 7f41ac968640 1 --2- 172.21.15.98:0/3723859859 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41a80c9000 0x7f41a80c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.408+0000 7f41ac968640 1 -- 172.21.15.98:0/3723859859 >> 172.21.15.98:0/3723859859 conn(0x7f41a806a4d0 msgr2=0x7f41a80706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.408+0000 7f41ac968640 1 -- 172.21.15.98:0/3723859859 shutdown_connections 2023-11-28T17:42:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:54.408+0000 7f41ac968640 1 -- 172.21.15.98:0/3723859859 wait complete. 2023-11-28T17:42:57.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:42:57.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.605+0000 7f1297fff640 1 Processor -- start 2023-11-28T17:42:57.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.606+0000 7f1297fff640 1 -- start start 2023-11-28T17:42:57.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.608+0000 7f1297fff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1298159c50 0x7f129815a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:57.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.608+0000 7f1297fff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129815a620 con 0x7f12980b0fc0 2023-11-28T17:42:57.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.608+0000 7f1297fff640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1298071f70 con 0x7f129806bb20 2023-11-28T17:42:57.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.608+0000 7f1297fff640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1298072530 con 0x7f1298159c50 2023-11-28T17:42:57.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.608+0000 7f12977fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f1298159c50 msgr2=0x7f129815a0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:42:57.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.608+0000 7f12967fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f129806bb20 legacy=0x7f129806a4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:42:57.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.608+0000 7f12977fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1298159c50 0x7f129815a0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:57.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.609+0000 7f1296ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f12980b0fc0 0x7f12980c6a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45104/0 (socket says 172.21.15.98:45104) 2023-11-28T17:42:57.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.609+0000 7f1296ffd640 1 -- 172.21.15.98:0/1014965750 learned_addr learned my addr 172.21.15.98:0/1014965750 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:42:57.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.609+0000 7f1295ffb640 1 -- 172.21.15.98:0/1014965750 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076551559 0 0) 0x7f129815a620 con 0x7f12980b0fc0 2023-11-28T17:42:57.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.609+0000 7f1295ffb640 1 -- 172.21.15.98:0/1014965750 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1280003520 con 0x7f12980b0fc0 2023-11-28T17:42:57.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.609+0000 7f1295ffb640 1 -- 172.21.15.98:0/1014965750 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1534951301 0 0) 0x7f1280003520 con 0x7f12980b0fc0 2023-11-28T17:42:57.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.610+0000 7f1295ffb640 1 -- 172.21.15.98:0/1014965750 >> v2:172.21.15.113:3300/0 conn(0x7f1298159c50 msgr2=0x7f129815a0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:57.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.610+0000 7f1295ffb640 1 --2- 172.21.15.98:0/1014965750 >> v2:172.21.15.113:3300/0 conn(0x7f1298159c50 0x7f129815a0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:57.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.610+0000 7f1295ffb640 1 -- 172.21.15.98:0/1014965750 >> v1:172.21.15.113:6789/0 conn(0x7f129806bb20 legacy=0x7f129806a4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:57.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.610+0000 7f1295ffb640 1 -- 172.21.15.98:0/1014965750 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1298072250 con 0x7f12980b0fc0 2023-11-28T17:42:57.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.610+0000 7f1295ffb640 1 -- 172.21.15.98:0/1014965750 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f1288001670 con 0x7f12980b0fc0 2023-11-28T17:42:57.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.610+0000 7f1295ffb640 1 -- 172.21.15.98:0/1014965750 >> v1:172.21.15.98:6789/0 conn(0x7f12980b0fc0 legacy=0x7f12980c6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:57.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.610+0000 7f1295ffb640 1 --2- 172.21.15.98:0/1014965750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1280003970 0x7f1280003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:57.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.610+0000 7f1295ffb640 1 --2- 172.21.15.98:0/1014965750 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12800047c0 0x7f1280004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:57.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.610+0000 7f1296ffd640 1 --2- 172.21.15.98:0/1014965750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1280003970 0x7f1280003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.610+0000 7f1295ffb640 1 --2- 172.21.15.98:0/1014965750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12800052a0 0x7f128000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.610+0000 7f1295ffb640 1 -- 172.21.15.98:0/1014965750 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1298072530 con 0x7f1280003970 2023-11-28T17:42:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.610+0000 7f1295ffb640 1 -- 172.21.15.98:0/1014965750 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128000c1a0 con 0x7f12800052a0 2023-11-28T17:42:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.610+0000 7f1295ffb640 1 -- 172.21.15.98:0/1014965750 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128000c360 con 0x7f12800047c0 2023-11-28T17:42:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.611+0000 7f12977fe640 1 -- 172.21.15.98:0/1014965750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12800052a0 msgr2=0x7f128000b980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:42:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.611+0000 7f12977fe640 1 --2- 172.21.15.98:0/1014965750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12800052a0 0x7f128000b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.611+0000 7f12967fc640 1 --2- 172.21.15.98:0/1014965750 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12800047c0 0x7f1280004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.611+0000 7f1296ffd640 1 -- 172.21.15.98:0/1014965750 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12800047c0 msgr2=0x7f1280004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:42:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.611+0000 7f1296ffd640 1 --2- 172.21.15.98:0/1014965750 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f12800047c0 0x7f1280004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.611+0000 7f1296ffd640 1 -- 172.21.15.98:0/1014965750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12800052a0 msgr2=0x7f128000b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:42:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.611+0000 7f1296ffd640 1 --2- 172.21.15.98:0/1014965750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12800052a0 0x7f128000b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:57.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.611+0000 7f1296ffd640 1 -- 172.21.15.98:0/1014965750 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1280003520 con 0x7f1280003970 2023-11-28T17:42:57.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.611+0000 7f12977fe640 1 -- 172.21.15.98:0/1014965750 reap_dead start 2023-11-28T17:42:57.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.612+0000 7f1296ffd640 1 --2- 172.21.15.98:0/1014965750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1280003970 0x7f1280003d70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1288001eb0 tx=0x7f12800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa67a03868f50cd2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:57.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.612+0000 7f1295ffb640 1 -- 172.21.15.98:0/1014965750 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1288005ad0 con 0x7f1280003970 2023-11-28T17:42:57.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.612+0000 7f1295ffb640 1 -- 172.21.15.98:0/1014965750 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1288005c30 con 0x7f1280003970 2023-11-28T17:42:57.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.612+0000 7f1295ffb640 1 -- 172.21.15.98:0/1014965750 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128802c7c0 con 0x7f1280003970 2023-11-28T17:42:57.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.613+0000 7f1297fff640 1 -- 172.21.15.98:0/1014965750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1280003970 msgr2=0x7f1280003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:57.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.613+0000 7f1297fff640 1 --2- 172.21.15.98:0/1014965750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1280003970 0x7f1280003d70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1288001eb0 tx=0x7f12800057f0 comp rx=0 tx=0).stop 2023-11-28T17:42:57.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.613+0000 7f1297fff640 1 -- 172.21.15.98:0/1014965750 shutdown_connections 2023-11-28T17:42:57.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.613+0000 7f1297fff640 1 --2- 172.21.15.98:0/1014965750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1280003970 0x7f1280003d70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:57.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.613+0000 7f1297fff640 1 -- 172.21.15.98:0/1014965750 >> 172.21.15.98:0/1014965750 conn(0x7f1298066230 msgr2=0x7f1298066640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:57.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.613+0000 7f1297fff640 1 -- 172.21.15.98:0/1014965750 shutdown_connections 2023-11-28T17:42:57.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.613+0000 7f1297fff640 1 -- 172.21.15.98:0/1014965750 wait complete. 2023-11-28T17:42:57.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.615+0000 7f1297fff640 1 Processor -- start 2023-11-28T17:42:57.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.615+0000 7f1297fff640 1 -- start start 2023-11-28T17:42:57.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.616+0000 7f1297fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1280003970 0x7f1298159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:57.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.616+0000 7f1297fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1298152e70 0x7f12981551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:57.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.616+0000 7f1297fff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1298153870 0x7f1298155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:57.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.616+0000 7f1297fff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1298154aa0 con 0x7f1298152e70 2023-11-28T17:42:57.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.616+0000 7f1297fff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1298158c80 con 0x7f1280003970 2023-11-28T17:42:57.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.616+0000 7f1297fff640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1298159940 con 0x7f1298153870 2023-11-28T17:42:57.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.616+0000 7f1296ffd640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1280003970 msgr2=0x7f1298159400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:42:57.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.616+0000 7f1296ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1280003970 0x7f1298159400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:42:57.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.616+0000 7f12967fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1298152e70 0x7f12981551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:57.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.616+0000 7f12967fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1298152e70 0x7f12981551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44666/0 (socket says 172.21.15.98:44666) 2023-11-28T17:42:57.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.616+0000 7f12967fc640 1 -- 172.21.15.98:0/2221492167 learned_addr learned my addr 172.21.15.98:0/2221492167 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:42:57.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.617+0000 7f12967fc640 1 -- 172.21.15.98:0/2221492167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1298153870 msgr2=0x7f1298155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:57.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.617+0000 7f12977fe640 1 --2- 172.21.15.98:0/2221492167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1298153870 0x7f1298155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:57.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.617+0000 7f12967fc640 1 --2- 172.21.15.98:0/2221492167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1298153870 0x7f1298155d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:57.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.617+0000 7f12967fc640 1 -- 172.21.15.98:0/2221492167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1280003970 msgr2=0x7f1298159400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:42:57.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.617+0000 7f12967fc640 1 --2- 172.21.15.98:0/2221492167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1280003970 0x7f1298159400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:57.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.617+0000 7f12967fc640 1 -- 172.21.15.98:0/2221492167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1298152b70 con 0x7f1298152e70 2023-11-28T17:42:57.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.617+0000 7f12967fc640 1 --2- 172.21.15.98:0/2221492167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1298152e70 0x7f12981551e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f128c00d580 tx=0x7f128c00da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:57.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.617+0000 7f127ffff640 1 -- 172.21.15.98:0/2221492167 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c003aa0 con 0x7f1298152e70 2023-11-28T17:42:57.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.617+0000 7f127ffff640 1 -- 172.21.15.98:0/2221492167 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c003c00 con 0x7f1298152e70 2023-11-28T17:42:57.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.617+0000 7f127ffff640 1 -- 172.21.15.98:0/2221492167 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c0050d0 con 0x7f1298152e70 2023-11-28T17:42:57.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.617+0000 7f1297fff640 1 -- 172.21.15.98:0/2221492167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1298151910 con 0x7f1298152e70 2023-11-28T17:42:57.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.617+0000 7f1297fff640 1 -- 172.21.15.98:0/2221492167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1298151250 con 0x7f1298152e70 2023-11-28T17:42:57.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.619+0000 7f1297fff640 1 -- 172.21.15.98:0/2221492167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12980b2390 con 0x7f1298152e70 2023-11-28T17:42:57.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.621+0000 7f127ffff640 1 -- 172.21.15.98:0/2221492167 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f128c005230 con 0x7f1298152e70 2023-11-28T17:42:57.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.622+0000 7f127ffff640 1 --2- 172.21.15.98:0/2221492167 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f128406b2a0 0x7f128406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:42:57.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.622+0000 7f127ffff640 1 -- 172.21.15.98:0/2221492167 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f128c089920 con 0x7f1298152e70 2023-11-28T17:42:57.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.622+0000 7f1296ffd640 1 --2- 172.21.15.98:0/2221492167 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f128406b2a0 0x7f128406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:42:57.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.623+0000 7f1296ffd640 1 --2- 172.21.15.98:0/2221492167 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f128406b2a0 0x7f128406d760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1288034000 tx=0x7f1288009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:42:57.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:57.627+0000 7f127ffff640 1 -- 172.21.15.98:0/2221492167 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f128c054780 con 0x7f1298152e70 2023-11-28T17:42:58.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:58.063+0000 7f1297fff640 1 -- 172.21.15.98:0/2221492167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f129815aa40 con 0x7f1298152e70 2023-11-28T17:42:58.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:58.063+0000 7f127ffff640 1 -- 172.21.15.98:0/2221492167 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f128c016620 con 0x7f1298152e70 2023-11-28T17:42:58.065 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:42:58.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:58.066+0000 7f1297fff640 1 -- 172.21.15.98:0/2221492167 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f128406b2a0 msgr2=0x7f128406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:58.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:58.066+0000 7f1297fff640 1 --2- 172.21.15.98:0/2221492167 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f128406b2a0 0x7f128406d760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1288034000 tx=0x7f1288009030 comp rx=0 tx=0).stop 2023-11-28T17:42:58.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:58.066+0000 7f1297fff640 1 -- 172.21.15.98:0/2221492167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1298152e70 msgr2=0x7f12981551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:42:58.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:58.066+0000 7f1297fff640 1 --2- 172.21.15.98:0/2221492167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1298152e70 0x7f12981551e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f128c00d580 tx=0x7f128c00da40 comp rx=0 tx=0).stop 2023-11-28T17:42:58.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:58.067+0000 7f1297fff640 1 -- 172.21.15.98:0/2221492167 shutdown_connections 2023-11-28T17:42:58.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:58.067+0000 7f1297fff640 1 --2- 172.21.15.98:0/2221492167 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1298153870 0x7f1298155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:58.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:58.067+0000 7f1297fff640 1 --2- 172.21.15.98:0/2221492167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1298152e70 0x7f12981551e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:58.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:58.067+0000 7f1297fff640 1 --2- 172.21.15.98:0/2221492167 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f128406b2a0 0x7f128406d760 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:58.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:58.067+0000 7f1297fff640 1 --2- 172.21.15.98:0/2221492167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1280003970 0x7f1298159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:42:58.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:58.067+0000 7f1297fff640 1 -- 172.21.15.98:0/2221492167 >> 172.21.15.98:0/2221492167 conn(0x7f1298066230 msgr2=0x7f129806a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:42:58.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:58.067+0000 7f1297fff640 1 -- 172.21.15.98:0/2221492167 shutdown_connections 2023-11-28T17:42:58.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:42:58.067+0000 7f1297fff640 1 -- 172.21.15.98:0/2221492167 wait complete. 2023-11-28T17:43:01.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:43:01.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.287+0000 7f35c2bfc640 1 Processor -- start 2023-11-28T17:43:01.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.287+0000 7f35c2bfc640 1 -- start start 2023-11-28T17:43:01.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.289+0000 7f35c2bfc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35bc157b60 0x7f35bc15bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:01.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.289+0000 7f35c2bfc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f35bc0c4910 0x7f35bc0c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:01.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.289+0000 7f35c2bfc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc15b150 con 0x7f35bc151d80 2023-11-28T17:43:01.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.289+0000 7f35c2bfc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35bc157f90 con 0x7f35bc0c4910 2023-11-28T17:43:01.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.289+0000 7f35c2bfc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc15c150 con 0x7f35bc157b60 2023-11-28T17:43:01.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.289+0000 7f35c0998640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f35bc151d80 legacy=0x7f35bc0c5c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:43:01.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.289+0000 7f35c1199640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f35bc0c4910 0x7f35bc0c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:01.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.289+0000 7f35c1199640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f35bc0c4910 0x7f35bc0c4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44132/0 (socket says 172.21.15.98:44132) 2023-11-28T17:43:01.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.289+0000 7f35c1199640 1 -- 172.21.15.98:0/3724495529 learned_addr learned my addr 172.21.15.98:0/3724495529 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:01.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.289+0000 7f35bbfff640 1 --2- 172.21.15.98:0/3724495529 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35bc157b60 0x7f35bc15bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:01.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.290+0000 7f35c1199640 1 -- 172.21.15.98:0/3724495529 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35bc157b60 msgr2=0x7f35bc15bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:01.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.290+0000 7f35c1199640 1 --2- 172.21.15.98:0/3724495529 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35bc157b60 0x7f35bc15bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:01.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.290+0000 7f35c1199640 1 -- 172.21.15.98:0/3724495529 >> v1:172.21.15.113:6789/0 conn(0x7f35bc151d80 legacy=0x7f35bc0c5c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:43:01.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.290+0000 7f35c1199640 1 -- 172.21.15.98:0/3724495529 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc06e1f0 con 0x7f35bc0c4910 2023-11-28T17:43:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.290+0000 7f35bbfff640 1 --2- 172.21.15.98:0/3724495529 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35bc157b60 0x7f35bc15bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:43:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.291+0000 7f35c1199640 1 --2- 172.21.15.98:0/3724495529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc0c4910 0x7f35bc0c4da0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009e50 tx=0x7f35b402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a8a3a5b87d25fe2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.291+0000 7f35bb7fe640 1 -- 172.21.15.98:0/3724495529 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b402cb00 con 0x7f35bc0c4910 2023-11-28T17:43:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.291+0000 7f35bb7fe640 1 -- 172.21.15.98:0/3724495529 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b402cc60 con 0x7f35bc0c4910 2023-11-28T17:43:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.291+0000 7f35bb7fe640 1 -- 172.21.15.98:0/3724495529 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b40343f0 con 0x7f35bc0c4910 2023-11-28T17:43:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.292+0000 7f35c2bfc640 1 -- 172.21.15.98:0/3724495529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc0c4910 msgr2=0x7f35bc0c4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.292+0000 7f35c2bfc640 1 --2- 172.21.15.98:0/3724495529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc0c4910 0x7f35bc0c4da0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009e50 tx=0x7f35b402bf60 comp rx=0 tx=0).stop 2023-11-28T17:43:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.292+0000 7f35c2bfc640 1 -- 172.21.15.98:0/3724495529 shutdown_connections 2023-11-28T17:43:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.292+0000 7f35c2bfc640 1 --2- 172.21.15.98:0/3724495529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc0c4910 0x7f35bc0c4da0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.292+0000 7f35c2bfc640 1 --2- 172.21.15.98:0/3724495529 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35bc157b60 0x7f35bc15bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:01.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.292+0000 7f35c2bfc640 1 -- 172.21.15.98:0/3724495529 >> 172.21.15.98:0/3724495529 conn(0x7f35bc06a4d0 msgr2=0x7f35bc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:01.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.292+0000 7f35c2bfc640 1 -- 172.21.15.98:0/3724495529 shutdown_connections 2023-11-28T17:43:01.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.292+0000 7f35c2bfc640 1 -- 172.21.15.98:0/3724495529 wait complete. 2023-11-28T17:43:01.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.294+0000 7f35c2bfc640 1 Processor -- start 2023-11-28T17:43:01.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.294+0000 7f35c2bfc640 1 -- start start 2023-11-28T17:43:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.295+0000 7f35c2bfc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35bc0c4910 0x7f35bc0c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.295+0000 7f35c2bfc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc151d80 0x7f35bc0c2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.295+0000 7f35c2bfc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35bc157b60 0x7f35bc1465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.295+0000 7f35c2bfc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc072130 con 0x7f35bc151d80 2023-11-28T17:43:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.295+0000 7f35c2bfc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc072410 con 0x7f35bc157b60 2023-11-28T17:43:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.295+0000 7f35c2bfc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc071ad0 con 0x7f35bc0c4910 2023-11-28T17:43:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.295+0000 7f35c1199640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35bc157b60 msgr2=0x7f35bc1465d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:43:01.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.295+0000 7f35c1199640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35bc157b60 0x7f35bc1465d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:43:01.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.295+0000 7f35bbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc151d80 0x7f35bc0c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:01.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.295+0000 7f35bbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc151d80 0x7f35bc0c2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44136/0 (socket says 172.21.15.98:44136) 2023-11-28T17:43:01.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.295+0000 7f35bbfff640 1 -- 172.21.15.98:0/4111933971 learned_addr learned my addr 172.21.15.98:0/4111933971 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:01.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.295+0000 7f35c0998640 1 --2- 172.21.15.98:0/4111933971 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35bc0c4910 0x7f35bc0c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:01.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.296+0000 7f35bbfff640 1 -- 172.21.15.98:0/4111933971 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35bc0c4910 msgr2=0x7f35bc0c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:01.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.296+0000 7f35bbfff640 1 --2- 172.21.15.98:0/4111933971 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35bc0c4910 0x7f35bc0c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:01.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.296+0000 7f35bbfff640 1 -- 172.21.15.98:0/4111933971 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35bc157b60 msgr2=0x7f35bc1465d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:43:01.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.296+0000 7f35bbfff640 1 --2- 172.21.15.98:0/4111933971 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35bc157b60 0x7f35bc1465d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:01.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.296+0000 7f35bbfff640 1 -- 172.21.15.98:0/4111933971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b4009b30 con 0x7f35bc151d80 2023-11-28T17:43:01.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.296+0000 7f35bbfff640 1 --2- 172.21.15.98:0/4111933971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc151d80 0x7f35bc0c2700 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f35a800e810 tx=0x7f35a800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:01.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.296+0000 7f35b97fa640 1 -- 172.21.15.98:0/4111933971 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8004030 con 0x7f35bc151d80 2023-11-28T17:43:01.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.296+0000 7f35b97fa640 1 -- 172.21.15.98:0/4111933971 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a8004190 con 0x7f35bc151d80 2023-11-28T17:43:01.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.296+0000 7f35c2bfc640 1 -- 172.21.15.98:0/4111933971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35bc071570 con 0x7f35bc151d80 2023-11-28T17:43:01.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.296+0000 7f35c2bfc640 1 -- 172.21.15.98:0/4111933971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35bc070fb0 con 0x7f35bc151d80 2023-11-28T17:43:01.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.296+0000 7f35b97fa640 1 -- 172.21.15.98:0/4111933971 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8004520 con 0x7f35bc151d80 2023-11-28T17:43:01.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.298+0000 7f35c2bfc640 1 -- 172.21.15.98:0/4111933971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359c005250 con 0x7f35bc151d80 2023-11-28T17:43:01.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.299+0000 7f35b97fa640 1 -- 172.21.15.98:0/4111933971 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f35a800c6d0 con 0x7f35bc151d80 2023-11-28T17:43:01.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.300+0000 7f35b97fa640 1 --2- 172.21.15.98:0/4111933971 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f35a006b280 0x7f35a006d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:01.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.300+0000 7f35b97fa640 1 -- 172.21.15.98:0/4111933971 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f35a80898c0 con 0x7f35bc151d80 2023-11-28T17:43:01.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.300+0000 7f35c0998640 1 --2- 172.21.15.98:0/4111933971 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f35a006b280 0x7f35a006d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:01.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.301+0000 7f35c0998640 1 --2- 172.21.15.98:0/4111933971 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f35a006b280 0x7f35a006d740 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f35b0003930 tx=0x7f35b0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:01.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.305+0000 7f35b97fa640 1 -- 172.21.15.98:0/4111933971 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35a80546b0 con 0x7f35bc151d80 2023-11-28T17:43:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.559+0000 7f35c2bfc640 1 -- 172.21.15.98:0/4111933971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f359c0050c0 con 0x7f35bc151d80 2023-11-28T17:43:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.559+0000 7f35b97fa640 1 -- 172.21.15.98:0/4111933971 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f35a8020620 con 0x7f35bc151d80 2023-11-28T17:43:01.561 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:43:01.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.562+0000 7f35c2bfc640 1 -- 172.21.15.98:0/4111933971 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f35a006b280 msgr2=0x7f35a006d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:01.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.562+0000 7f35c2bfc640 1 --2- 172.21.15.98:0/4111933971 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f35a006b280 0x7f35a006d740 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f35b0003930 tx=0x7f35b0008040 comp rx=0 tx=0).stop 2023-11-28T17:43:01.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.562+0000 7f35c2bfc640 1 -- 172.21.15.98:0/4111933971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc151d80 msgr2=0x7f35bc0c2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:01.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.562+0000 7f35c2bfc640 1 --2- 172.21.15.98:0/4111933971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc151d80 0x7f35bc0c2700 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f35a800e810 tx=0x7f35a800ecd0 comp rx=0 tx=0).stop 2023-11-28T17:43:01.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.562+0000 7f35c2bfc640 1 -- 172.21.15.98:0/4111933971 shutdown_connections 2023-11-28T17:43:01.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.562+0000 7f35c2bfc640 1 --2- 172.21.15.98:0/4111933971 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f35a006b280 0x7f35a006d740 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:01.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.562+0000 7f35c2bfc640 1 --2- 172.21.15.98:0/4111933971 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35bc157b60 0x7f35bc1465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:01.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.562+0000 7f35c2bfc640 1 --2- 172.21.15.98:0/4111933971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc151d80 0x7f35bc0c2700 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:01.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.562+0000 7f35c2bfc640 1 --2- 172.21.15.98:0/4111933971 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f35bc0c4910 0x7f35bc0c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.562+0000 7f35c2bfc640 1 -- 172.21.15.98:0/4111933971 >> 172.21.15.98:0/4111933971 conn(0x7f35bc06a4d0 msgr2=0x7f35bc14df20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.562+0000 7f35c2bfc640 1 -- 172.21.15.98:0/4111933971 shutdown_connections 2023-11-28T17:43:01.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:01.562+0000 7f35c2bfc640 1 -- 172.21.15.98:0/4111933971 wait complete. 2023-11-28T17:43:04.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:43:04.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.870+0000 7fae5595c640 1 Processor -- start 2023-11-28T17:43:04.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.871+0000 7fae5595c640 1 -- start start 2023-11-28T17:43:04.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.872+0000 7fae5595c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50158240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:04.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.872+0000 7fae5595c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 0x7fae5015b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:04.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.872+0000 7fae5595c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5006ade0 con 0x7fae501518d0 2023-11-28T17:43:04.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.872+0000 7fae5595c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae50151cb0 con 0x7fae5015ac70 2023-11-28T17:43:04.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.872+0000 7fae5595c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5006b030 con 0x7fae500728a0 2023-11-28T17:43:04.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.872+0000 7fae5515b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 0x7fae5015b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:04.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.872+0000 7fae5515b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 0x7fae5015b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44146/0 (socket says 172.21.15.98:44146) 2023-11-28T17:43:04.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.872+0000 7fae5515b640 1 -- 172.21.15.98:0/2302354213 learned_addr learned my addr 172.21.15.98:0/2302354213 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:04.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.873+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:04.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.873+0000 7fae5515b640 1 -- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 msgr2=0x7fae5015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:43:04.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.873+0000 7fae5515b640 1 -- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 msgr2=0x7fae5015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:04.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.873+0000 7fae5515b640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 0x7fae5015b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:04.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.873+0000 7fae5515b640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 0x7fae5015b0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:43:04.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.873+0000 7fae4ffff640 1 -- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 msgr2=0x7fae50158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:43:04.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.873+0000 7fae4ffff640 1 -- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 msgr2=0x7fae50158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:04.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.873+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:04.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:04.873+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50158240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:43:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.074+0000 7fae5515b640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 0x7fae5015b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.074+0000 7fae5515b640 1 -- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 msgr2=0x7fae5015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:43:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.074+0000 7fae5515b640 1 -- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 msgr2=0x7fae5015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.074+0000 7fae5515b640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 0x7fae5015b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.074+0000 7fae5515b640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 0x7fae5015b0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:43:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.074+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.075+0000 7fae4ffff640 1 -- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 msgr2=0x7fae50158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:43:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.075+0000 7fae4ffff640 1 -- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 msgr2=0x7fae50158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.075+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.075+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50158240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:43:05.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.475+0000 7fae5515b640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 0x7fae5015b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:05.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.475+0000 7fae5515b640 1 -- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 msgr2=0x7fae5015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:43:05.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.475+0000 7fae5515b640 1 -- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 msgr2=0x7fae5015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.475+0000 7fae5515b640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 0x7fae5015b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.475+0000 7fae5515b640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 0x7fae5015b0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:43:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.476+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.477+0000 7fae4ffff640 1 -- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 msgr2=0x7fae50158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:43:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.477+0000 7fae4ffff640 1 -- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 msgr2=0x7fae50158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.477+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:05.477+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50158240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:43:06.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:06.276+0000 7fae5515b640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 0x7fae5015b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:06.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:06.276+0000 7fae5515b640 1 -- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 msgr2=0x7fae5015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:43:06.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:06.276+0000 7fae5515b640 1 -- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 msgr2=0x7fae5015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:06.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:06.276+0000 7fae5515b640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 0x7fae5015b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:06.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:06.276+0000 7fae5515b640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 0x7fae5015b0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:43:06.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:06.278+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:06.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:06.279+0000 7fae4ffff640 1 -- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 msgr2=0x7fae50158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:43:06.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:06.279+0000 7fae4ffff640 1 -- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 msgr2=0x7fae50158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:06.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:06.279+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:06.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:06.279+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50158240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:43:07.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:43:07.724 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:43:07.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:07.872+0000 7fae4e7fc640 1 -- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 msgr2=0x7fae50158240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:43:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:07.872+0000 7fae4e7fc640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50158240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:07.872+0000 7fae4e7fc640 1 -- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 msgr2=0x7fae5015b0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:43:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:07.872+0000 7fae4e7fc640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.98:3300/0 conn(0x7fae5015ac70 0x7fae5015b0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:07.872+0000 7fae4e7fc640 1 -- 172.21.15.98:0/2302354213 >> v1:172.21.15.98:6789/0 conn(0x7fae501518d0 legacy=0x7fae50070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:07.872+0000 7fae4e7fc640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.113:3300/0 conn(0x7fae34007370 0x7fae34009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:07.872+0000 7fae4e7fc640 1 -- 172.21.15.98:0/2302354213 --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae500700e0 con 0x7fae340036e0 2023-11-28T17:43:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:07.872+0000 7fae4e7fc640 1 -- 172.21.15.98:0/2302354213 --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae50151cb0 con 0x7fae34007370 2023-11-28T17:43:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:07.873+0000 7fae5495a640 1 -- 172.21.15.98:0/2302354213 >> v1:172.21.15.113:6789/0 conn(0x7fae340036e0 legacy=0x7fae34005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:43:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:07.873+0000 7fae4ffff640 1 -- 172.21.15.98:0/2302354213 >> v2:172.21.15.113:3300/0 conn(0x7fae34007370 msgr2=0x7fae34009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:43:07.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:07.873+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.113:3300/0 conn(0x7fae34007370 0x7fae34009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:43:08.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:08.074+0000 7fae5495a640 1 -- 172.21.15.98:0/2302354213 >> v1:172.21.15.113:6789/0 conn(0x7fae340036e0 legacy=0x7fae34005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:43:08.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:08.074+0000 7fae4ffff640 1 -- 172.21.15.98:0/2302354213 >> v2:172.21.15.113:3300/0 conn(0x7fae34007370 msgr2=0x7fae34009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:43:08.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:08.074+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.113:3300/0 conn(0x7fae34007370 0x7fae34009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:43:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:08.475+0000 7fae4ffff640 1 -- 172.21.15.98:0/2302354213 >> v2:172.21.15.113:3300/0 conn(0x7fae34007370 msgr2=0x7fae34009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:43:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:08.475+0000 7fae5495a640 1 -- 172.21.15.98:0/2302354213 >> v1:172.21.15.113:6789/0 conn(0x7fae340036e0 legacy=0x7fae34005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:43:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:08.475+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.113:3300/0 conn(0x7fae34007370 0x7fae34009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:43:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:09.276+0000 7fae5495a640 1 -- 172.21.15.98:0/2302354213 >> v1:172.21.15.113:6789/0 conn(0x7fae340036e0 legacy=0x7fae34005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:43:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:09.276+0000 7fae4ffff640 1 -- 172.21.15.98:0/2302354213 >> v2:172.21.15.113:3300/0 conn(0x7fae34007370 msgr2=0x7fae34009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:43:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:09.276+0000 7fae4ffff640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.113:3300/0 conn(0x7fae34007370 0x7fae34009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:43:10.310 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 63% (7705/12117) Updating files: 64% (7755/12117) Updating files: 65% (7877/12117) Updating files: 66% (7998/12117) Updating files: 67% (8119/12117) Updating files: 68% (8240/12117) Updating files: 69% (8361/12117) Updating files: 70% (8482/12117) Updating files: 71% (8604/12117) Updating files: 72% (8725/12117) Updating files: 73% (8846/12117) Updating files: 74% (8967/12117) Updating files: 75% (9088/12117) Updating files: 76% (9209/12117) Updating files: 77% (9331/12117) Updating files: 78% (9452/12117) Updating files: 79% (9573/12117) Updating files: 80% (9694/12117) Updating files: 81% (9815/12117) Updating files: 82% (9936/12117) Updating files: 83% (10058/12117) Updating files: 84% (10179/12117) Updating files: 85% (10300/12117) Updating files: 86% (10421/12117) Updating files: 87% (10542/12117) Updating files: 88% (10663/12117) Updating files: 89% (10785/12117) Updating files: 90% (10906/12117) Updating files: 91% (11027/12117) Updating files: 92% (11148/12117) Updating files: 93% (11269/12117) Updating files: 94% (11390/12117) Updating files: 95% (11512/12117) Updating files: 96% (11633/12117) Updating files: 97% (11754/12117) Updating files: 98% (11875/12117) Updating files: 99% (11996/12117) Updating files: 100% (12117/12117) Updating files: 100% (12117/12117), done. 2023-11-28T17:43:10.573 INFO:tasks.workunit.client.0.smithi098.stderr:Note: switching to '47c1c547372ed5bff7984647dc200bac0e019caf'. 2023-11-28T17:43:10.573 INFO:tasks.workunit.client.0.smithi098.stderr: 2023-11-28T17:43:10.573 INFO:tasks.workunit.client.0.smithi098.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-11-28T17:43:10.573 INFO:tasks.workunit.client.0.smithi098.stderr:changes and commit them, and you can discard any commits you make in this 2023-11-28T17:43:10.573 INFO:tasks.workunit.client.0.smithi098.stderr:state without impacting any branches by switching back to a branch. 2023-11-28T17:43:10.574 INFO:tasks.workunit.client.0.smithi098.stderr: 2023-11-28T17:43:10.574 INFO:tasks.workunit.client.0.smithi098.stderr:If you want to create a new branch to retain commits you create, you may 2023-11-28T17:43:10.574 INFO:tasks.workunit.client.0.smithi098.stderr:do so (now or later) by using -c with the switch command. Example: 2023-11-28T17:43:10.574 INFO:tasks.workunit.client.0.smithi098.stderr: 2023-11-28T17:43:10.574 INFO:tasks.workunit.client.0.smithi098.stderr: git switch -c 2023-11-28T17:43:10.574 INFO:tasks.workunit.client.0.smithi098.stderr: 2023-11-28T17:43:10.574 INFO:tasks.workunit.client.0.smithi098.stderr:Or undo this operation with: 2023-11-28T17:43:10.574 INFO:tasks.workunit.client.0.smithi098.stderr: 2023-11-28T17:43:10.574 INFO:tasks.workunit.client.0.smithi098.stderr: git switch - 2023-11-28T17:43:10.574 INFO:tasks.workunit.client.0.smithi098.stderr: 2023-11-28T17:43:10.575 INFO:tasks.workunit.client.0.smithi098.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-11-28T17:43:10.575 INFO:tasks.workunit.client.0.smithi098.stderr: 2023-11-28T17:43:10.575 INFO:tasks.workunit.client.0.smithi098.stderr:HEAD is now at 47c1c547372 Merge PR #51931 into wip-vshankar-testing-20231127.102654 2023-11-28T17:43:10.577 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-11-28T17:43:10.689 INFO:tasks.workunit.client.0.smithi098.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-11-28T17:43:10.696 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-28T17:43:10.697 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-11-28T17:43:10.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.872+0000 7fae4e7fc640 1 -- 172.21.15.98:0/2302354213 >> v2:172.21.15.113:3300/0 conn(0x7fae34007370 msgr2=0x7fae34009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:43:10.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.872+0000 7fae4e7fc640 1 --2- 172.21.15.98:0/2302354213 >> v2:172.21.15.113:3300/0 conn(0x7fae34007370 0x7fae34009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:10.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.872+0000 7fae4e7fc640 1 -- 172.21.15.98:0/2302354213 >> v1:172.21.15.113:6789/0 conn(0x7fae340036e0 legacy=0x7fae34005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:43:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.873+0000 7fae5515b640 1 -- 172.21.15.98:0/2302354213 reap_dead start 2023-11-28T17:43:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.873+0000 7fae4e7fc640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae3400a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.873+0000 7fae4e7fc640 1 -- 172.21.15.98:0/2302354213 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3400af20 con 0x7fae501518d0 2023-11-28T17:43:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.873+0000 7fae4e7fc640 1 -- 172.21.15.98:0/2302354213 --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae500700e0 con 0x7fae340036e0 2023-11-28T17:43:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.873+0000 7fae4e7fc640 1 -- 172.21.15.98:0/2302354213 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae50151cb0 con 0x7fae500728a0 2023-11-28T17:43:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.873+0000 7fae4ffff640 1 -- 172.21.15.98:0/2302354213 >> v1:172.21.15.113:6789/0 conn(0x7fae340036e0 legacy=0x7fae34005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:43:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.874+0000 7fae5515b640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae3400a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.874+0000 7fae4f7fe640 1 -- 172.21.15.98:0/2302354213 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838096604 0 0) 0x7fae3400af20 con 0x7fae501518d0 2023-11-28T17:43:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.875+0000 7fae4f7fe640 1 -- 172.21.15.98:0/2302354213 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae40003520 con 0x7fae501518d0 2023-11-28T17:43:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.875+0000 7fae5515b640 1 -- 172.21.15.98:0/2302354213 >> v1:172.21.15.113:6789/0 conn(0x7fae340036e0 legacy=0x7fae34005b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:43:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.875+0000 7fae5515b640 1 -- 172.21.15.98:0/2302354213 >> v1:172.21.15.98:6789/0 conn(0x7fae501518d0 legacy=0x7fae34009ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.875+0000 7fae5515b640 1 -- 172.21.15.98:0/2302354213 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3400ac60 con 0x7fae500728a0 2023-11-28T17:43:10.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.878+0000 7fae5515b640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae3400a650 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fae50155400 tx=0x7fae4802c790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3de5e8ea8acf5f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.879+0000 7fae4f7fe640 1 -- 172.21.15.98:0/2302354213 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae48032050 con 0x7fae500728a0 2023-11-28T17:43:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.879+0000 7fae4f7fe640 1 -- 172.21.15.98:0/2302354213 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4800f040 con 0x7fae500728a0 2023-11-28T17:43:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.879+0000 7fae4f7fe640 1 -- 172.21.15.98:0/2302354213 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae48033660 con 0x7fae500728a0 2023-11-28T17:43:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.880+0000 7fae5595c640 1 -- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 msgr2=0x7fae3400a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.880+0000 7fae5595c640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae3400a650 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fae50155400 tx=0x7fae4802c790 comp rx=0 tx=0).stop 2023-11-28T17:43:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.880+0000 7fae5595c640 1 -- 172.21.15.98:0/2302354213 shutdown_connections 2023-11-28T17:43:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.880+0000 7fae5595c640 1 --2- 172.21.15.98:0/2302354213 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae3400a650 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.880+0000 7fae5595c640 1 -- 172.21.15.98:0/2302354213 >> 172.21.15.98:0/2302354213 conn(0x7fae50066230 msgr2=0x7fae5006a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.880+0000 7fae5595c640 1 -- 172.21.15.98:0/2302354213 shutdown_connections 2023-11-28T17:43:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.880+0000 7fae5595c640 1 -- 172.21.15.98:0/2302354213 wait complete. 2023-11-28T17:43:10.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.883+0000 7fae5595c640 1 Processor -- start 2023-11-28T17:43:10.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.883+0000 7fae5595c640 1 -- start start 2023-11-28T17:43:10.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.885+0000 7fae5595c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae340036e0 0x7fae5014a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:10.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.885+0000 7fae5595c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50145580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:10.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.885+0000 7fae5595c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae501518d0 0x7fae50145ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:10.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.885+0000 7fae5595c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae500c39e0 con 0x7fae501518d0 2023-11-28T17:43:10.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.885+0000 7fae5595c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae501499f0 con 0x7fae340036e0 2023-11-28T17:43:10.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.885+0000 7fae5595c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae500c2b30 con 0x7fae500728a0 2023-11-28T17:43:10.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.885+0000 7fae5495a640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae340036e0 msgr2=0x7fae5014a3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:43:10.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.885+0000 7fae5495a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae340036e0 0x7fae5014a3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:43:10.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.885+0000 7fae5515b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae501518d0 0x7fae50145ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:10.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.885+0000 7fae5515b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae501518d0 0x7fae50145ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37802/0 (socket says 172.21.15.98:37802) 2023-11-28T17:43:10.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.885+0000 7fae5515b640 1 -- 172.21.15.98:0/1260309470 learned_addr learned my addr 172.21.15.98:0/1260309470 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:10.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.885+0000 7fae4ffff640 1 --2- 172.21.15.98:0/1260309470 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50145580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:10.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.886+0000 7fae5515b640 1 -- 172.21.15.98:0/1260309470 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 msgr2=0x7fae50145580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:10.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.886+0000 7fae5515b640 1 --2- 172.21.15.98:0/1260309470 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50145580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:10.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.886+0000 7fae5515b640 1 -- 172.21.15.98:0/1260309470 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae340036e0 msgr2=0x7fae5014a3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:43:10.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.886+0000 7fae5515b640 1 --2- 172.21.15.98:0/1260309470 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae340036e0 0x7fae5014a3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:10.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.886+0000 7fae5515b640 1 -- 172.21.15.98:0/1260309470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae48009820 con 0x7fae501518d0 2023-11-28T17:43:10.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.886+0000 7fae5515b640 1 --2- 172.21.15.98:0/1260309470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae501518d0 0x7fae50145ac0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fae4802c7d0 tx=0x7fae48004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:10.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.886+0000 7fae4d7fa640 1 -- 172.21.15.98:0/1260309470 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae480130b0 con 0x7fae501518d0 2023-11-28T17:43:10.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.886+0000 7fae4d7fa640 1 -- 172.21.15.98:0/1260309470 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae48041450 con 0x7fae501518d0 2023-11-28T17:43:10.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.886+0000 7fae4d7fa640 1 -- 172.21.15.98:0/1260309470 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae480417e0 con 0x7fae501518d0 2023-11-28T17:43:10.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.886+0000 7fae5595c640 1 -- 172.21.15.98:0/1260309470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae501493e0 con 0x7fae501518d0 2023-11-28T17:43:10.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.887+0000 7fae5595c640 1 -- 172.21.15.98:0/1260309470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae500c3700 con 0x7fae501518d0 2023-11-28T17:43:10.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.888+0000 7fae5595c640 1 -- 172.21.15.98:0/1260309470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae50149240 con 0x7fae501518d0 2023-11-28T17:43:10.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.889+0000 7fae4d7fa640 1 -- 172.21.15.98:0/1260309470 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fae4803d070 con 0x7fae501518d0 2023-11-28T17:43:10.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.890+0000 7fae4d7fa640 1 --2- 172.21.15.98:0/1260309470 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fae3406c140 0x7fae3406e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:10.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.890+0000 7fae4d7fa640 1 -- 172.21.15.98:0/1260309470 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fae480ac420 con 0x7fae501518d0 2023-11-28T17:43:10.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.890+0000 7fae5495a640 1 --2- 172.21.15.98:0/1260309470 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fae3406c140 0x7fae3406e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:10.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.891+0000 7fae5495a640 1 --2- 172.21.15.98:0/1260309470 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fae3406c140 0x7fae3406e600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae440011d0 tx=0x7fae44007860 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:10.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:10.895+0000 7fae4d7fa640 1 -- 172.21.15.98:0/1260309470 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae50149240 con 0x7fae501518d0 2023-11-28T17:43:11.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.160+0000 7fae5595c640 1 -- 172.21.15.98:0/1260309470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fae50149240 con 0x7fae501518d0 2023-11-28T17:43:11.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.161+0000 7fae4d7fa640 1 -- 172.21.15.98:0/1260309470 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7fae4803b620 con 0x7fae501518d0 2023-11-28T17:43:11.162 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":18,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:43:11.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.164+0000 7fae5595c640 1 -- 172.21.15.98:0/1260309470 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fae3406c140 msgr2=0x7fae3406e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.164+0000 7fae5595c640 1 --2- 172.21.15.98:0/1260309470 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fae3406c140 0x7fae3406e600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae440011d0 tx=0x7fae44007860 comp rx=0 tx=0).stop 2023-11-28T17:43:11.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.164+0000 7fae5595c640 1 -- 172.21.15.98:0/1260309470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae501518d0 msgr2=0x7fae50145ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.164+0000 7fae5595c640 1 --2- 172.21.15.98:0/1260309470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae501518d0 0x7fae50145ac0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fae4802c7d0 tx=0x7fae48004ab0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.164+0000 7fae5595c640 1 -- 172.21.15.98:0/1260309470 shutdown_connections 2023-11-28T17:43:11.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.164+0000 7fae5595c640 1 --2- 172.21.15.98:0/1260309470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae501518d0 0x7fae50145ac0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.164+0000 7fae5595c640 1 --2- 172.21.15.98:0/1260309470 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fae500728a0 0x7fae50145580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.164+0000 7fae5595c640 1 --2- 172.21.15.98:0/1260309470 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fae3406c140 0x7fae3406e600 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.164+0000 7fae5595c640 1 --2- 172.21.15.98:0/1260309470 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae340036e0 0x7fae5014a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.164+0000 7fae5595c640 1 -- 172.21.15.98:0/1260309470 >> 172.21.15.98:0/1260309470 conn(0x7fae50066230 msgr2=0x7fae5006dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:11.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.164+0000 7fae5595c640 1 -- 172.21.15.98:0/1260309470 shutdown_connections 2023-11-28T17:43:11.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.164+0000 7fae5595c640 1 -- 172.21.15.98:0/1260309470 wait complete. 2023-11-28T17:43:11.183 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-11-28T17:43:11.184 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-11-28T17:43:11.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:43:11.374 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-11-28T17:43:11.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.386+0000 7feaedf97640 1 Processor -- start 2023-11-28T17:43:11.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.386+0000 7feaedf97640 1 -- start start 2023-11-28T17:43:11.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.387+0000 7feaedf97640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feae80b5270 0x7feae80c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.388+0000 7feaedf97640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae815db80 con 0x7feae80c76f0 2023-11-28T17:43:11.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.388+0000 7feaedf97640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae815e0c0 con 0x7feae8153fb0 2023-11-28T17:43:11.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.388+0000 7feaedf97640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae80cb730 con 0x7feae80b5270 2023-11-28T17:43:11.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.388+0000 7feae77fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7feae8153fb0 legacy=0x7feae80ca300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:43:11.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.388+0000 7feae7fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7feae80c76f0 0x7feae80c7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50924/0 (socket says 172.21.15.98:50924) 2023-11-28T17:43:11.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.388+0000 7feae7fff640 1 -- 172.21.15.98:0/3888057491 learned_addr learned my addr 172.21.15.98:0/3888057491 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:43:11.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.388+0000 7feadeffd640 1 --2- 172.21.15.98:0/3888057491 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feae80b5270 0x7feae80c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.388+0000 7feae6ffd640 1 -- 172.21.15.98:0/3888057491 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813256277 0 0) 0x7feae815db80 con 0x7feae80c76f0 2023-11-28T17:43:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.388+0000 7feae6ffd640 1 -- 172.21.15.98:0/3888057491 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead4003520 con 0x7feae80c76f0 2023-11-28T17:43:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae6ffd640 1 -- 172.21.15.98:0/3888057491 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2371481639 0 0) 0x7fead4003520 con 0x7feae80c76f0 2023-11-28T17:43:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae6ffd640 1 -- 172.21.15.98:0/3888057491 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feae80b5270 msgr2=0x7feae80c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae6ffd640 1 --2- 172.21.15.98:0/3888057491 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feae80b5270 0x7feae80c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae6ffd640 1 -- 172.21.15.98:0/3888057491 >> v1:172.21.15.113:6789/0 conn(0x7feae8153fb0 legacy=0x7feae80ca300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:43:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae6ffd640 1 -- 172.21.15.98:0/3888057491 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae80d00a0 con 0x7feae80c76f0 2023-11-28T17:43:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae6ffd640 1 -- 172.21.15.98:0/3888057491 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7feae0001670 con 0x7feae80c76f0 2023-11-28T17:43:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae6ffd640 1 -- 172.21.15.98:0/3888057491 >> v1:172.21.15.98:6789/0 conn(0x7feae80c76f0 legacy=0x7feae80c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae6ffd640 1 --2- 172.21.15.98:0/3888057491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fead40039a0 0x7fead4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae6ffd640 1 --2- 172.21.15.98:0/3888057491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead4004370 0x7fead4004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae6ffd640 1 --2- 172.21.15.98:0/3888057491 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fead40050e0 0x7fead400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae6ffd640 1 -- 172.21.15.98:0/3888057491 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae80cb730 con 0x7fead4004370 2023-11-28T17:43:11.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae6ffd640 1 -- 172.21.15.98:0/3888057491 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead400bec0 con 0x7fead40039a0 2023-11-28T17:43:11.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae6ffd640 1 -- 172.21.15.98:0/3888057491 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead400c000 con 0x7fead40050e0 2023-11-28T17:43:11.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae77fe640 1 -- 172.21.15.98:0/3888057491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fead40039a0 msgr2=0x7fead4003da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:43:11.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae77fe640 1 --2- 172.21.15.98:0/3888057491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fead40039a0 0x7fead4003da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:43:11.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feadeffd640 1 --2- 172.21.15.98:0/3888057491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead4004370 0x7fead4004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:11.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.389+0000 7feae7fff640 1 --2- 172.21.15.98:0/3888057491 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fead40050e0 0x7fead400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:11.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.390+0000 7feadeffd640 1 -- 172.21.15.98:0/3888057491 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fead40050e0 msgr2=0x7fead400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:11.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.390+0000 7feadeffd640 1 --2- 172.21.15.98:0/3888057491 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fead40050e0 0x7fead400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.390+0000 7feadeffd640 1 -- 172.21.15.98:0/3888057491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fead40039a0 msgr2=0x7fead4003da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:43:11.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.390+0000 7feadeffd640 1 --2- 172.21.15.98:0/3888057491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fead40039a0 0x7fead4003da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.390+0000 7feadeffd640 1 -- 172.21.15.98:0/3888057491 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead4003520 con 0x7fead4004370 2023-11-28T17:43:11.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.391+0000 7feadeffd640 1 --2- 172.21.15.98:0/3888057491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead4004370 0x7fead4004840 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fead0009fc0 tx=0x7fead40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36213bd3a7394075 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:11.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.391+0000 7feae6ffd640 1 -- 172.21.15.98:0/3888057491 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead002cc20 con 0x7fead4004370 2023-11-28T17:43:11.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.391+0000 7feae6ffd640 1 -- 172.21.15.98:0/3888057491 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead000f040 con 0x7fead4004370 2023-11-28T17:43:11.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.391+0000 7feae6ffd640 1 -- 172.21.15.98:0/3888057491 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead0034440 con 0x7fead4004370 2023-11-28T17:43:11.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.391+0000 7feaedf97640 1 -- 172.21.15.98:0/3888057491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead4004370 msgr2=0x7fead4004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.391+0000 7feaedf97640 1 --2- 172.21.15.98:0/3888057491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead4004370 0x7fead4004840 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fead0009fc0 tx=0x7fead40057f0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.391+0000 7feae7fff640 1 -- 172.21.15.98:0/3888057491 reap_dead start 2023-11-28T17:43:11.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.392+0000 7feaedf97640 1 -- 172.21.15.98:0/3888057491 shutdown_connections 2023-11-28T17:43:11.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.392+0000 7feaedf97640 1 -- 172.21.15.98:0/3888057491 >> 172.21.15.98:0/3888057491 conn(0x7feae80cf4a0 msgr2=0x7feae80cf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:11.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.392+0000 7feaedf97640 1 -- 172.21.15.98:0/3888057491 shutdown_connections 2023-11-28T17:43:11.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.392+0000 7feaedf97640 1 -- 172.21.15.98:0/3888057491 wait complete. 2023-11-28T17:43:11.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.393+0000 7feaedf97640 1 Processor -- start 2023-11-28T17:43:11.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.393+0000 7feaedf97640 1 -- start start 2023-11-28T17:43:11.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.394+0000 7feaedf97640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead4004370 0x7feae815b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.394+0000 7feaedf97640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feae815cd00 0x7feae80ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.394+0000 7feaedf97640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feae80be4a0 0x7feae80cea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.394+0000 7feaedf97640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae80bd350 con 0x7fead4004370 2023-11-28T17:43:11.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.394+0000 7feaedf97640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae81eff60 con 0x7feae80be4a0 2023-11-28T17:43:11.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.394+0000 7feaedf97640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae81ef0f0 con 0x7feae815cd00 2023-11-28T17:43:11.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.395+0000 7feae7fff640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feae80be4a0 msgr2=0x7feae80cea00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:43:11.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.395+0000 7feae77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead4004370 0x7feae815b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:11.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.395+0000 7feae7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feae80be4a0 0x7feae80cea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:43:11.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.395+0000 7feae77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead4004370 0x7feae815b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37830/0 (socket says 172.21.15.98:37830) 2023-11-28T17:43:11.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.395+0000 7feae77fe640 1 -- 172.21.15.98:0/1907414219 learned_addr learned my addr 172.21.15.98:0/1907414219 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:11.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.395+0000 7feadeffd640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feae815cd00 0x7feae80ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:11.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.395+0000 7feae77fe640 1 -- 172.21.15.98:0/1907414219 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feae815cd00 msgr2=0x7feae80ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.395+0000 7feae77fe640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7feae815cd00 0x7feae80ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.395+0000 7feae77fe640 1 -- 172.21.15.98:0/1907414219 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feae80be4a0 msgr2=0x7feae80cea00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:43:11.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.395+0000 7feae77fe640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feae80be4a0 0x7feae80cea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.395+0000 7feae77fe640 1 -- 172.21.15.98:0/1907414219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae81ef2e0 con 0x7fead4004370 2023-11-28T17:43:11.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.395+0000 7feae77fe640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead4004370 0x7feae815b9b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fead800b570 tx=0x7fead800ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:11.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.396+0000 7feae4ff9640 1 -- 172.21.15.98:0/1907414219 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8003e50 con 0x7fead4004370 2023-11-28T17:43:11.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.396+0000 7feae4ff9640 1 -- 172.21.15.98:0/1907414219 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead8005070 con 0x7fead4004370 2023-11-28T17:43:11.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.396+0000 7feaedf97640 1 -- 172.21.15.98:0/1907414219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae81ef5c0 con 0x7fead4004370 2023-11-28T17:43:11.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.396+0000 7feae4ff9640 1 -- 172.21.15.98:0/1907414219 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8005400 con 0x7fead4004370 2023-11-28T17:43:11.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.396+0000 7feaedf97640 1 -- 172.21.15.98:0/1907414219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae81efb80 con 0x7fead4004370 2023-11-28T17:43:11.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.397+0000 7feaedf97640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feae80bbb60 0x7feae80bbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.397+0000 7feaedf97640 1 -- 172.21.15.98:0/1907414219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae8074110 con 0x7feae80bbb60 2023-11-28T17:43:11.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.397+0000 7feadeffd640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feae80bbb60 0x7feae80bbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:11.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.398+0000 7feadeffd640 1 -- 172.21.15.98:0/1907414219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feae8150210 con 0x7feae80bbb60 2023-11-28T17:43:11.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.398+0000 7feadeffd640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feae80bbb60 0x7feae80bbf40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fead0009f90 tx=0x7fead0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:11.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.398+0000 7feae4ff9640 1 -- 172.21.15.98:0/1907414219 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fead8005560 con 0x7fead4004370 2023-11-28T17:43:11.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.399+0000 7feae4ff9640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7feac806b250 0x7feac806d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.399+0000 7feae4ff9640 1 -- 172.21.15.98:0/1907414219 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fead8089c00 con 0x7fead4004370 2023-11-28T17:43:11.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.399+0000 7feae7fff640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7feac806b250 0x7feac806d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.400+0000 7feae4ff9640 1 -- 172.21.15.98:0/1907414219 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead0034da0 con 0x7feae80bbb60 2023-11-28T17:43:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.400+0000 7feae4ff9640 1 -- 172.21.15.98:0/1907414219 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fead003f480 con 0x7feae80bbb60 2023-11-28T17:43:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.400+0000 7feae4ff9640 1 -- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feae80bbb60 msgr2=0x7feae80bbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.400+0000 7feae4ff9640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feae80bbb60 0x7feae80bbf40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fead0009f90 tx=0x7fead0009750 comp rx=0 tx=0).stop 2023-11-28T17:43:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.400+0000 7feae4ff9640 1 -- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feae80bbb60 msgr2=0x7feae80bbf40 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.400+0000 7feae4ff9640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feae80bbb60 0x7feae80bbf40 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fead0009f90 tx=0x7fead0009750 comp rx=0 tx=0).stop 2023-11-28T17:43:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.400+0000 7feae7fff640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7feac806b250 0x7feac806d710 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feae80c7ad0 tx=0x7feae0000fe0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:11.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.406+0000 7feaedf97640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feae81f2660 0x7feae81f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.407+0000 7feaedf97640 1 -- 172.21.15.98:0/1907414219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae815d0e0 con 0x7feae81f2660 2023-11-28T17:43:11.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.407+0000 7feadeffd640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feae81f2660 0x7feae81f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:11.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.407+0000 7feadeffd640 1 -- 172.21.15.98:0/1907414219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7feae80cb9a0 con 0x7feae81f2660 2023-11-28T17:43:11.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.408+0000 7feadeffd640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feae81f2660 0x7feae81f2a70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fead0009e60 tx=0x7fead0030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.408+0000 7feae4ff9640 1 -- 172.21.15.98:0/1907414219 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead004e070 con 0x7feae81f2660 2023-11-28T17:43:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.408+0000 7feae4ff9640 1 -- 172.21.15.98:0/1907414219 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4796 (secure 0 0 0) 0x7fead0052410 con 0x7feae81f2660 2023-11-28T17:43:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.408+0000 7feae4ff9640 1 -- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feae81f2660 msgr2=0x7feae81f2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.408+0000 7feae4ff9640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feae81f2660 0x7feae81f2a70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fead0009e60 tx=0x7fead0030000 comp rx=0 tx=0).stop 2023-11-28T17:43:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.408+0000 7feae4ff9640 1 -- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feae81f2660 msgr2=0x7feae81f2a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.408+0000 7feae4ff9640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feae81f2660 0x7feae81f2a70 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fead0009e60 tx=0x7fead0030000 comp rx=0 tx=0).stop 2023-11-28T17:43:11.410 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:43:11.410 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:43:11.410 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:43:11.410 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:43:11.410 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 18, 2023-11-28T17:43:11.410 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:43:11.410 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:43:11.410 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:43:11.410 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:11.411 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2023-11-28T17:43:11.411 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:11.411 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:43:11.411 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:43:11.411 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:11.411 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:11.411 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:11.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:11.411 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:11.411 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:11.411 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:11.412 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:11.412 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:11.412 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:11.412 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:11.412 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:43:11.412 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:43:11.412 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:11.412 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:11.412 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:11.413 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:11.413 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:11.413 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:11.413 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:11.413 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:11.413 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:11.413 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:11.413 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:11.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.413 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:43:11.413 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:43:11.414 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:43:11.414 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:43:11.414 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:43:11.414 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:43:11.414 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:11.414 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:11.414 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:43:11.414 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:43:11.414 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:43:11.414 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:43:11.414 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:43:11.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:43:11.415 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:43:11.415 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:11.415 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:43:11.415 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:11.415 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:11.415 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:11.415 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:11.415 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:11.415 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:11.415 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:11.416 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:11.416 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:11.416 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:11.416 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:11.416 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:43:11.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.416 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:43:11.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.416 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:43:11.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:43:11.417 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:11.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:11.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:11.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:43:11.417 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:11.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:11.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:43:11.417 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:11.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.418 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:11.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:43:11.418 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:43:11.418 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:11.418 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:11.418 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:11.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.419 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:43:11.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:43:11.419 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:11.419 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:11.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:11.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:43:11.420 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:11.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:11.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:43:11.420 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:11.420 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.420 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:11.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:43:11.421 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:43:11.421 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:11.421 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:11.421 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:11.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.421 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:43:11.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:43:11.421 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:11.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:11.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:11.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:43:11.422 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:11.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:11.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:43:11.422 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:11.422 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.422 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:11.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:43:11.423 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:43:11.423 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:11.423 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:11.423 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:11.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:11.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.423 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:43:11.423 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:43:11.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.423 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:11.423 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:11.424 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:43:11.424 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.424 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:11.424 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:43:11.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.424 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:11.424 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:11.424 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:43:11.424 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.424 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:11.425 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:43:11.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.425 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:11.425 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:11.425 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:43:11.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.425 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:11.425 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:43:11.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.426 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:11.426 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:11.426 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:43:11.426 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.426 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:11.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.426 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:43:11.426 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:43:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.408+0000 7feaedf97640 1 -- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7feac806b250 msgr2=0x7feac806d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.408+0000 7feaedf97640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7feac806b250 0x7feac806d710 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feae80c7ad0 tx=0x7feae0000fe0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.408+0000 7feaedf97640 1 -- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead4004370 msgr2=0x7feae815b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.408+0000 7feaedf97640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead4004370 0x7feae815b9b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fead800b570 tx=0x7fead800ba30 comp rx=0 tx=0).stop 2023-11-28T17:43:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.408+0000 7feae7fff640 1 -- 172.21.15.98:0/1907414219 reap_dead start 2023-11-28T17:43:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.409+0000 7feaedf97640 1 -- 172.21.15.98:0/1907414219 shutdown_connections 2023-11-28T17:43:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.409+0000 7feaedf97640 1 --2- 172.21.15.98:0/1907414219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fead4004370 0x7feae815b9b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.409+0000 7feaedf97640 1 -- 172.21.15.98:0/1907414219 >> 172.21.15.98:0/1907414219 conn(0x7feae80cf4a0 msgr2=0x7feae814fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.409+0000 7feaedf97640 1 -- 172.21.15.98:0/1907414219 shutdown_connections 2023-11-28T17:43:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.409+0000 7feaedf97640 1 -- 172.21.15.98:0/1907414219 wait complete. 2023-11-28T17:43:11.430 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:43:11.431 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-11-28T17:43:11.431 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:43:11.487 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-11-28T17:43:11.559 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-28T17:43:11.564 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-28T17:43:11.564 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-11-28T17:43:11.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.616+0000 7f9ebc47c640 1 Processor -- start 2023-11-28T17:43:11.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.616+0000 7f9ebc47c640 1 -- start start 2023-11-28T17:43:11.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.617+0000 7f9ebc47c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9eb40c5a60 0x7f9eb415e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.617+0000 7f9ebc47c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb40c6630 con 0x7f9eb4071b50 2023-11-28T17:43:11.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.617+0000 7f9ebc47c640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb406e710 con 0x7f9eb4071770 2023-11-28T17:43:11.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.617+0000 7f9ebc47c640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb406de70 con 0x7f9eb40c5a60 2023-11-28T17:43:11.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.617+0000 7f9eba218640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9eb4071770 legacy=0x7f9eb4157fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:43:11.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.617+0000 7f9ebaa19640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9eb40c5a60 msgr2=0x7f9eb415e110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:43:11.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.617+0000 7f9ebaa19640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9eb40c5a60 0x7f9eb415e110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:43:11.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.617+0000 7f9eb9a17640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9eb4071b50 0x7f9eb4166a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50926/0 (socket says 172.21.15.98:50926) 2023-11-28T17:43:11.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9a17640 1 -- 172.21.15.98:0/1824282999 learned_addr learned my addr 172.21.15.98:0/1824282999 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:43:11.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9216640 1 -- 172.21.15.98:0/1824282999 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988364633 0 0) 0x7f9eb40c6630 con 0x7f9eb4071b50 2023-11-28T17:43:11.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9216640 1 -- 172.21.15.98:0/1824282999 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ea0003520 con 0x7f9eb4071b50 2023-11-28T17:43:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9216640 1 -- 172.21.15.98:0/1824282999 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3584579997 0 0) 0x7f9ea0003520 con 0x7f9eb4071b50 2023-11-28T17:43:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9216640 1 -- 172.21.15.98:0/1824282999 >> v2:172.21.15.113:3300/0 conn(0x7f9eb40c5a60 msgr2=0x7f9eb415e110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:43:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9216640 1 --2- 172.21.15.98:0/1824282999 >> v2:172.21.15.113:3300/0 conn(0x7f9eb40c5a60 0x7f9eb415e110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9216640 1 -- 172.21.15.98:0/1824282999 >> v1:172.21.15.113:6789/0 conn(0x7f9eb4071770 legacy=0x7f9eb4157fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:43:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9216640 1 -- 172.21.15.98:0/1824282999 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb406e150 con 0x7f9eb4071b50 2023-11-28T17:43:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9216640 1 -- 172.21.15.98:0/1824282999 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f9ea4001670 con 0x7f9eb4071b50 2023-11-28T17:43:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9216640 1 -- 172.21.15.98:0/1824282999 >> v1:172.21.15.98:6789/0 conn(0x7f9eb4071b50 legacy=0x7f9eb4166a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9216640 1 --2- 172.21.15.98:0/1824282999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ea0003970 0x7f9ea0003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9216640 1 --2- 172.21.15.98:0/1824282999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ea00047c0 0x7f9ea0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9216640 1 --2- 172.21.15.98:0/1824282999 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ea00052a0 0x7f9ea000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9216640 1 -- 172.21.15.98:0/1824282999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb406de70 con 0x7f9ea0003970 2023-11-28T17:43:11.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9216640 1 -- 172.21.15.98:0/1824282999 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea000c1f0 con 0x7f9ea00047c0 2023-11-28T17:43:11.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.618+0000 7f9eb9216640 1 -- 172.21.15.98:0/1824282999 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea000c3b0 con 0x7f9ea00052a0 2023-11-28T17:43:11.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.619+0000 7f9eb9a17640 1 -- 172.21.15.98:0/1824282999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ea00047c0 msgr2=0x7f9ea0004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:43:11.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.619+0000 7f9eba218640 1 --2- 172.21.15.98:0/1824282999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ea0003970 0x7f9ea0003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:11.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.619+0000 7f9eb9a17640 1 --2- 172.21.15.98:0/1824282999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ea00047c0 0x7f9ea0004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:43:11.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.619+0000 7f9eba218640 1 -- 172.21.15.98:0/1824282999 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ea00052a0 msgr2=0x7f9ea000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:11.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.619+0000 7f9ebaa19640 1 --2- 172.21.15.98:0/1824282999 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ea00052a0 0x7f9ea000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:11.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.619+0000 7f9eba218640 1 --2- 172.21.15.98:0/1824282999 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9ea00052a0 0x7f9ea000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.619+0000 7f9eba218640 1 -- 172.21.15.98:0/1824282999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ea00047c0 msgr2=0x7f9ea0004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:43:11.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.619+0000 7f9eba218640 1 --2- 172.21.15.98:0/1824282999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ea00047c0 0x7f9ea0004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.619+0000 7f9eba218640 1 -- 172.21.15.98:0/1824282999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea0003520 con 0x7f9ea0003970 2023-11-28T17:43:11.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.619+0000 7f9ebaa19640 1 -- 172.21.15.98:0/1824282999 reap_dead start 2023-11-28T17:43:11.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.620+0000 7f9eba218640 1 --2- 172.21.15.98:0/1824282999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ea0003970 0x7f9ea0003d70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0003cb0 tx=0x7f9ea00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2e6909652dab96 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:11.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.620+0000 7f9eb9216640 1 -- 172.21.15.98:0/1824282999 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb002ccf0 con 0x7f9ea0003970 2023-11-28T17:43:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.620+0000 7f9eb9216640 1 -- 172.21.15.98:0/1824282999 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb002ce50 con 0x7f9ea0003970 2023-11-28T17:43:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.620+0000 7f9eb9216640 1 -- 172.21.15.98:0/1824282999 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb00356b0 con 0x7f9ea0003970 2023-11-28T17:43:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.621+0000 7f9ebc47c640 1 -- 172.21.15.98:0/1824282999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ea0003970 msgr2=0x7f9ea0003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.621+0000 7f9ebc47c640 1 --2- 172.21.15.98:0/1824282999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ea0003970 0x7f9ea0003d70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0003cb0 tx=0x7f9ea00057f0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.621+0000 7f9ebc47c640 1 -- 172.21.15.98:0/1824282999 shutdown_connections 2023-11-28T17:43:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.621+0000 7f9ebc47c640 1 --2- 172.21.15.98:0/1824282999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9ea0003970 0x7f9ea0003d70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.621+0000 7f9ebc47c640 1 -- 172.21.15.98:0/1824282999 >> 172.21.15.98:0/1824282999 conn(0x7f9eb406c0f0 msgr2=0x7f9eb406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.621+0000 7f9ebc47c640 1 -- 172.21.15.98:0/1824282999 shutdown_connections 2023-11-28T17:43:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.621+0000 7f9ebc47c640 1 -- 172.21.15.98:0/1824282999 wait complete. 2023-11-28T17:43:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.623+0000 7f9ebc47c640 1 Processor -- start 2023-11-28T17:43:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.623+0000 7f9ebc47c640 1 -- start start 2023-11-28T17:43:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.624+0000 7f9ebc47c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ea0003970 0x7f9eb415ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.624+0000 7f9ebc47c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4160520 0x7f9eb414a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.624+0000 7f9ebc47c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb414a8f0 0x7f9eb414fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.624+0000 7f9ebc47c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb415fbc0 con 0x7f9eb414a8f0 2023-11-28T17:43:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.624+0000 7f9ebc47c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb415f2b0 con 0x7f9ea0003970 2023-11-28T17:43:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.624+0000 7f9ebc47c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb4160980 con 0x7f9eb4160520 2023-11-28T17:43:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.624+0000 7f9eba218640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ea0003970 msgr2=0x7f9eb415ffe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:43:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.624+0000 7f9eba218640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ea0003970 0x7f9eb415ffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:43:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.624+0000 7f9ebaa19640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb414a8f0 0x7f9eb414fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.624+0000 7f9ebaa19640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb414a8f0 0x7f9eb414fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37860/0 (socket says 172.21.15.98:37860) 2023-11-28T17:43:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.624+0000 7f9ebaa19640 1 -- 172.21.15.98:0/1456559254 learned_addr learned my addr 172.21.15.98:0/1456559254 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.625+0000 7f9eb9a17640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4160520 0x7f9eb414a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.625+0000 7f9ebaa19640 1 -- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4160520 msgr2=0x7f9eb414a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.625+0000 7f9ebaa19640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4160520 0x7f9eb414a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.625+0000 7f9ebaa19640 1 -- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ea0003970 msgr2=0x7f9eb415ffe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:43:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.625+0000 7f9ebaa19640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9ea0003970 0x7f9eb415ffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.625+0000 7f9ebaa19640 1 -- 172.21.15.98:0/1456559254 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb41f6f80 con 0x7f9eb414a8f0 2023-11-28T17:43:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.625+0000 7f9ebaa19640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb414a8f0 0x7f9eb414fb40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9eb40c76e0 tx=0x7f9eac00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.625+0000 7f9eaaffd640 1 -- 172.21.15.98:0/1456559254 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eac003cf0 con 0x7f9eb414a8f0 2023-11-28T17:43:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.625+0000 7f9eaaffd640 1 -- 172.21.15.98:0/1456559254 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eac003e50 con 0x7f9eb414a8f0 2023-11-28T17:43:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.625+0000 7f9eaaffd640 1 -- 172.21.15.98:0/1456559254 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eac0052b0 con 0x7f9eb414a8f0 2023-11-28T17:43:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.626+0000 7f9ebc47c640 1 -- 172.21.15.98:0/1456559254 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb41f7260 con 0x7f9eb414a8f0 2023-11-28T17:43:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.626+0000 7f9ebc47c640 1 -- 172.21.15.98:0/1456559254 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb415f7e0 con 0x7f9eb414a8f0 2023-11-28T17:43:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.627+0000 7f9ebc47c640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4150080 0x7f9eb41f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.627+0000 7f9ebc47c640 1 -- 172.21.15.98:0/1456559254 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb40744d0 con 0x7f9eb4150080 2023-11-28T17:43:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.628+0000 7f9eba218640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4150080 0x7f9eb41f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.628+0000 7f9eaaffd640 1 -- 172.21.15.98:0/1456559254 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f9eac005410 con 0x7f9eb414a8f0 2023-11-28T17:43:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.628+0000 7f9eba218640 1 -- 172.21.15.98:0/1456559254 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9eb40711a0 con 0x7f9eb4150080 2023-11-28T17:43:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.629+0000 7f9eba218640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4150080 0x7f9eb41f2e90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9ea00057a0 tx=0x7f9eb0005d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.629+0000 7f9eaaffd640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9e9806b250 0x7f9e9806d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.629+0000 7f9eaaffd640 1 -- 172.21.15.98:0/1456559254 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9eac0891c0 con 0x7f9eb414a8f0 2023-11-28T17:43:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.629+0000 7f9eb9a17640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9e9806b250 0x7f9e9806d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.629+0000 7f9eaaffd640 1 -- 172.21.15.98:0/1456559254 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb003c040 con 0x7f9eb4150080 2023-11-28T17:43:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.630+0000 7f9eaaffd640 1 -- 172.21.15.98:0/1456559254 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f9eb00092d0 con 0x7f9eb4150080 2023-11-28T17:43:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.630+0000 7f9eaaffd640 1 -- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4150080 msgr2=0x7f9eb41f2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.630+0000 7f9eaaffd640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4150080 0x7f9eb41f2e90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9ea00057a0 tx=0x7f9eb0005d90 comp rx=0 tx=0).stop 2023-11-28T17:43:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.630+0000 7f9eaaffd640 1 -- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4150080 msgr2=0x7f9eb41f2e90 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.630+0000 7f9eaaffd640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4150080 0x7f9eb41f2e90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.630+0000 7f9eb9a17640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9e9806b250 0x7f9e9806d710 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9ea40018a0 tx=0x7f9ea40016f0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.637+0000 7f9ebc47c640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4150460 0x7f9eb41f33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.637+0000 7f9ebc47c640 1 -- 172.21.15.98:0/1456559254 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb415f970 con 0x7f9eb4150460 2023-11-28T17:43:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.638+0000 7f9eba218640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4150460 0x7f9eb41f33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.639+0000 7f9eba218640 1 -- 172.21.15.98:0/1456559254 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9eb415fd00 con 0x7f9eb4150460 2023-11-28T17:43:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.639+0000 7f9eba218640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4150460 0x7f9eb41f33d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9eb002dfd0 tx=0x7f9eb0009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.640+0000 7f9eaaffd640 1 -- 172.21.15.98:0/1456559254 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb004e070 con 0x7f9eb4150460 2023-11-28T17:43:11.640 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-28T17:43:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.640+0000 7f9eaaffd640 1 -- 172.21.15.98:0/1456559254 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4964 (secure 0 0 0) 0x7f9eb0003b30 con 0x7f9eb4150460 2023-11-28T17:43:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.640+0000 7f9eaaffd640 1 -- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4150460 msgr2=0x7f9eb41f33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.640+0000 7f9eaaffd640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4150460 0x7f9eb41f33d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9eb002dfd0 tx=0x7f9eb0009520 comp rx=0 tx=0).stop 2023-11-28T17:43:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.640+0000 7f9eaaffd640 1 -- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4150460 msgr2=0x7f9eb41f33d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.640+0000 7f9eaaffd640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9eb4150460 0x7f9eb41f33d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.642 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:43:11.642 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:43:11.642 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:43:11.642 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:43:11.642 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 18, 2023-11-28T17:43:11.642 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:43:11.642 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:43:11.642 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:43:11.642 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:11.642 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2023-11-28T17:43:11.642 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:11.642 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:43:11.643 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:43:11.643 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:11.643 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:11.643 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:11.643 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:11.643 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:11.643 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:11.643 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:11.643 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:11.643 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:11.643 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:11.643 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:11.643 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:43:11.644 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:43:11.644 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:11.644 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:11.644 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:11.644 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:11.644 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:11.644 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:11.644 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:11.644 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:11.644 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:11.644 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:11.644 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:11.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.644 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:43:11.645 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:43:11.645 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:43:11.645 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:43:11.645 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:43:11.645 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:43:11.645 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:11.645 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:11.645 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:43:11.645 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:43:11.645 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:43:11.645 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:43:11.646 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:43:11.646 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:43:11.646 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:43:11.646 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:11.646 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:43:11.646 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:11.646 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:11.646 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:11.646 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:11.646 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:11.647 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:11.647 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:11.647 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:11.647 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:11.647 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:11.647 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:11.647 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:43:11.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.647 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:43:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:43:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:43:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:43:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:43:11.648 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:11.649 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.649 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:11.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.649 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:43:11.649 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:43:11.649 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:11.649 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:11.649 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:11.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.649 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:43:11.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:43:11.649 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:11.649 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:11.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:11.650 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:43:11.650 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:11.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:11.650 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:43:11.650 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:11.650 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.650 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:11.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.651 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:43:11.651 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:43:11.651 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:11.651 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:11.651 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:11.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.651 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:43:11.651 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:43:11.651 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:11.651 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:11.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:11.651 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:43:11.652 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:11.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.652 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:11.652 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:43:11.652 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:11.652 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.652 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:11.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.652 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:43:11.652 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:43:11.652 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:11.652 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:11.652 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:11.653 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.653 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:11.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.653 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:43:11.653 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:43:11.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.653 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:11.653 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:11.653 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:43:11.653 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.653 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:11.653 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:43:11.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.654 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:11.654 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:11.654 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:43:11.654 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.654 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:11.654 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:43:11.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.654 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:11.654 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:11.654 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 6 2023-11-28T17:43:11.654 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.654 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:11.655 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:43:11.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.655 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:11.655 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:11.655 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:43:11.655 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.655 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:11.655 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:43:11.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:11.655 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:11.655 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:11.655 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:43:11.655 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:11.656 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:11.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:11.656 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:43:11.656 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:43:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.641+0000 7f9ebc47c640 1 -- 172.21.15.98:0/1456559254 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9e9806b250 msgr2=0x7f9e9806d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.641+0000 7f9ebc47c640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9e9806b250 0x7f9e9806d710 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9ea40018a0 tx=0x7f9ea40016f0 comp rx=0 tx=0).stop 2023-11-28T17:43:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.641+0000 7f9ebc47c640 1 -- 172.21.15.98:0/1456559254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb414a8f0 msgr2=0x7f9eb414fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.641+0000 7f9ebc47c640 1 --2- 172.21.15.98:0/1456559254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb414a8f0 0x7f9eb414fb40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9eb40c76e0 tx=0x7f9eac00c890 comp rx=0 tx=0).stop 2023-11-28T17:43:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.641+0000 7f9ebaa19640 1 -- 172.21.15.98:0/1456559254 reap_dead start 2023-11-28T17:43:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.641+0000 7f9ebaa19640 1 -- 172.21.15.98:0/1456559254 reap_dead start 2023-11-28T17:43:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.641+0000 7f9ebc47c640 1 -- 172.21.15.98:0/1456559254 shutdown_connections 2023-11-28T17:43:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.641+0000 7f9ebc47c640 1 -- 172.21.15.98:0/1456559254 >> 172.21.15.98:0/1456559254 conn(0x7f9eb406c0f0 msgr2=0x7f9eb415e250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.641+0000 7f9ebc47c640 1 -- 172.21.15.98:0/1456559254 shutdown_connections 2023-11-28T17:43:11.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:11.641+0000 7f9ebc47c640 1 -- 172.21.15.98:0/1456559254 wait complete. 2023-11-28T17:43:11.658 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:43:11.658 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-11-28T17:43:11.659 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:43:11.659 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-11-28T17:43:11.755 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2023-11-28T17:43:11.756 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2023-11-28T17:43:11.756 DEBUG:teuthology.orchestra.run.smithi098:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2023-11-28T17:43:11.849 INFO:tasks.workunit.client.0.smithi098.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2023-11-28T17:43:11.852 INFO:tasks.workunit.client.0.smithi098.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2023-11-28T17:43:11.852 INFO:tasks.workunit.client.0.smithi098.stderr:+ trap -- retry EXIT 2023-11-28T17:43:11.852 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm -rf ffsb 2023-11-28T17:43:11.856 INFO:tasks.workunit.client.0.smithi098.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2023-11-28T17:43:11.871 INFO:tasks.workunit.client.0.smithi098.stderr:Cloning into 'ffsb'... 2023-11-28T17:43:29.665 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 50% (30/59) Updating files: 52% (31/59) Updating files: 54% (32/59) Updating files: 55% (33/59) Updating files: 57% (34/59) Updating files: 59% (35/59) Updating files: 61% (36/59) Updating files: 62% (37/59) Updating files: 64% (38/59) Updating files: 66% (39/59) Updating files: 67% (40/59) Updating files: 69% (41/59) Updating files: 71% (42/59) Updating files: 72% (43/59) Updating files: 74% (44/59) Updating files: 76% (45/59) Updating files: 77% (46/59) Updating files: 79% (47/59) Updating files: 81% (48/59) Updating files: 83% (49/59) Updating files: 84% (50/59) Updating files: 86% (51/59) Updating files: 88% (52/59) Updating files: 89% (53/59) Updating files: 91% (54/59) Updating files: 93% (55/59) Updating files: 94% (56/59) Updating files: 96% (57/59) Updating files: 98% (58/59) Updating files: 100% (59/59) Updating files: 100% (59/59), done. 2023-11-28T17:43:30.004 INFO:tasks.workunit.client.0.smithi098.stderr:+ trap - EXIT 2023-11-28T17:43:30.004 INFO:tasks.workunit.client.0.smithi098.stderr:+ cd ffsb 2023-11-28T17:43:30.010 INFO:tasks.workunit.client.0.smithi098.stderr:+ ./configure 2023-11-28T17:43:30.572 INFO:tasks.workunit.client.0.smithi098.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2023-11-28T17:43:31.621 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether build environment is sane... yes 2023-11-28T17:43:31.660 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-11-28T17:43:31.660 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-11-28T17:43:31.661 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-28T17:43:31.661 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-28T17:43:31.670 INFO:tasks.ceph.mon.b:Started 2023-11-28T17:43:31.671 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:43:31.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:43:31.695 INFO:tasks.workunit.client.0.smithi098.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2023-11-28T17:43:31.697 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gawk... gawk 2023-11-28T17:43:31.744 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether make sets $(MAKE)... yes 2023-11-28T17:43:31.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.876+0000 7f13b72f1640 1 Processor -- start 2023-11-28T17:43:31.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.876+0000 7f13b72f1640 1 -- start start 2023-11-28T17:43:31.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.878+0000 7f13b72f1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f13b006dbc0 0x7f13b006b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:31.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.878+0000 7f13b72f1640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f13b0070d30 0x7f13b0151660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:31.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.878+0000 7f13b72f1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b0070450 con 0x7f13b00708e0 2023-11-28T17:43:31.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.878+0000 7f13b72f1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b0071180 con 0x7f13b006dbc0 2023-11-28T17:43:31.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.878+0000 7f13b72f1640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b0072a80 con 0x7f13b0070d30 2023-11-28T17:43:31.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.878+0000 7f13b62ef640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f13b006dbc0 0x7f13b006b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:31.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.878+0000 7f13b5aee640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f13b00708e0 0x7f13b006dfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33950/0 (socket says 172.21.15.98:33950) 2023-11-28T17:43:31.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.878+0000 7f13b5aee640 1 -- 172.21.15.98:0/1245775749 learned_addr learned my addr 172.21.15.98:0/1245775749 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:43:31.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.879+0000 7f13b62ef640 1 -- 172.21.15.98:0/1245775749 >> v2:172.21.15.113:3300/0 conn(0x7f13b0070d30 msgr2=0x7f13b0151660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:31.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.879+0000 7f13b62ef640 1 --2- 172.21.15.98:0/1245775749 >> v2:172.21.15.113:3300/0 conn(0x7f13b0070d30 0x7f13b0151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:31.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.879+0000 7f13b62ef640 1 -- 172.21.15.98:0/1245775749 >> v1:172.21.15.98:6789/0 conn(0x7f13b00708e0 legacy=0x7f13b006dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:31.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.879+0000 7f13b62ef640 1 -- 172.21.15.98:0/1245775749 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b0158660 con 0x7f13b006dbc0 2023-11-28T17:43:31.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.880+0000 7f13b62ef640 1 --2- 172.21.15.98:0/1245775749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b006dbc0 0x7f13b006b020 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f13b00703c0 tx=0x7f13ac02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b71ece49b65c3e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:31.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.880+0000 7f13b52ed640 1 -- 172.21.15.98:0/1245775749 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac030610 con 0x7f13b006dbc0 2023-11-28T17:43:31.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.880+0000 7f13b52ed640 1 -- 172.21.15.98:0/1245775749 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ac030770 con 0x7f13b006dbc0 2023-11-28T17:43:31.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.880+0000 7f13b52ed640 1 -- 172.21.15.98:0/1245775749 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac030b00 con 0x7f13b006dbc0 2023-11-28T17:43:31.881 INFO:tasks.workunit.client.0.smithi098.stdout:checking build system type... x86_64-unknown-linux-gnu 2023-11-28T17:43:31.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.881+0000 7f13b72f1640 1 -- 172.21.15.98:0/1245775749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b006dbc0 msgr2=0x7f13b006b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:31.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.881+0000 7f13b72f1640 1 --2- 172.21.15.98:0/1245775749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b006dbc0 0x7f13b006b020 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f13b00703c0 tx=0x7f13ac02b960 comp rx=0 tx=0).stop 2023-11-28T17:43:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.881+0000 7f13b72f1640 1 -- 172.21.15.98:0/1245775749 shutdown_connections 2023-11-28T17:43:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.881+0000 7f13b72f1640 1 --2- 172.21.15.98:0/1245775749 >> v2:172.21.15.113:3300/0 conn(0x7f13b0070d30 0x7f13b0151660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.881+0000 7f13b72f1640 1 --2- 172.21.15.98:0/1245775749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b006dbc0 0x7f13b006b020 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.881+0000 7f13b72f1640 1 -- 172.21.15.98:0/1245775749 >> 172.21.15.98:0/1245775749 conn(0x7f13b0066230 msgr2=0x7f13b006a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.881+0000 7f13b72f1640 1 -- 172.21.15.98:0/1245775749 shutdown_connections 2023-11-28T17:43:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.881+0000 7f13b72f1640 1 -- 172.21.15.98:0/1245775749 wait complete. 2023-11-28T17:43:31.884 INFO:tasks.workunit.client.0.smithi098.stdout:checking host system type... x86_64-unknown-linux-gnu 2023-11-28T17:43:31.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.883+0000 7f13b72f1640 1 Processor -- start 2023-11-28T17:43:31.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.883+0000 7f13b72f1640 1 -- start start 2023-11-28T17:43:31.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.884+0000 7f13b72f1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b006dbc0 0x7f13b01ec360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:31.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.884+0000 7f13b72f1640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13b00708e0 0x7f13b0141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.884+0000 7f13b72f1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13b0070d30 0x7f13b0139a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.884+0000 7f13b72f1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b014dc20 con 0x7f13b006dbc0 2023-11-28T17:43:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.884+0000 7f13b72f1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b01ee400 con 0x7f13b0070d30 2023-11-28T17:43:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.884+0000 7f13b72f1640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b01eec50 con 0x7f13b00708e0 2023-11-28T17:43:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b62ef640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b006dbc0 0x7f13b01ec360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b5aee640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13b00708e0 0x7f13b0141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b6af0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13b0070d30 0x7f13b0139a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b62ef640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b006dbc0 0x7f13b01ec360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36564/0 (socket says 172.21.15.98:36564) 2023-11-28T17:43:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b62ef640 1 -- 172.21.15.98:0/1443692946 learned_addr learned my addr 172.21.15.98:0/1443692946 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b6af0640 1 -- 172.21.15.98:0/1443692946 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13b0070d30 msgr2=0x7f13b0139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-11-28T17:43:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b6af0640 1 -- 172.21.15.98:0/1443692946 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13b0070d30 msgr2=0x7f13b0139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:43:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b6af0640 1 --2- 172.21.15.98:0/1443692946 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13b0070d30 0x7f13b0139a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b6af0640 1 --2- 172.21.15.98:0/1443692946 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13b0070d30 0x7f13b0139a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:43:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b5aee640 1 -- 172.21.15.98:0/1443692946 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13b0070d30 msgr2=0x7f13b0139a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:43:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b5aee640 1 --2- 172.21.15.98:0/1443692946 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13b0070d30 0x7f13b0139a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b5aee640 1 -- 172.21.15.98:0/1443692946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b006dbc0 msgr2=0x7f13b01ec360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b5aee640 1 --2- 172.21.15.98:0/1443692946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b006dbc0 0x7f13b01ec360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b5aee640 1 -- 172.21.15.98:0/1443692946 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ac009400 con 0x7f13b00708e0 2023-11-28T17:43:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b62ef640 1 --2- 172.21.15.98:0/1443692946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b006dbc0 0x7f13b01ec360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:43:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.885+0000 7f13b5aee640 1 --2- 172.21.15.98:0/1443692946 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13b00708e0 0x7f13b0141380 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f13a000c470 tx=0x7f13a0001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.886+0000 7f13a6ffd640 1 -- 172.21.15.98:0/1443692946 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a0005ce0 con 0x7f13b00708e0 2023-11-28T17:43:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.886+0000 7f13a6ffd640 1 -- 172.21.15.98:0/1443692946 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a0005e40 con 0x7f13b00708e0 2023-11-28T17:43:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.886+0000 7f13a6ffd640 1 -- 172.21.15.98:0/1443692946 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a00072d0 con 0x7f13b00708e0 2023-11-28T17:43:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.886+0000 7f13b72f1640 1 -- 172.21.15.98:0/1443692946 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b01420c0 con 0x7f13b00708e0 2023-11-28T17:43:31.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.886+0000 7f13b72f1640 1 -- 172.21.15.98:0/1443692946 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b014d940 con 0x7f13b00708e0 2023-11-28T17:43:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.887+0000 7f13b72f1640 1 -- 172.21.15.98:0/1443692946 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b0071a50 con 0x7f13b00708e0 2023-11-28T17:43:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.888+0000 7f13a6ffd640 1 -- 172.21.15.98:0/1443692946 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f13a0007430 con 0x7f13b00708e0 2023-11-28T17:43:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.889+0000 7f13a6ffd640 1 --2- 172.21.15.98:0/1443692946 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f139406b2a0 0x7f139406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.889+0000 7f13a6ffd640 1 -- 172.21.15.98:0/1443692946 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f13a0089200 con 0x7f13b00708e0 2023-11-28T17:43:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.889+0000 7f13b62ef640 1 --2- 172.21.15.98:0/1443692946 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f139406b2a0 0x7f139406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.890+0000 7f13b62ef640 1 --2- 172.21.15.98:0/1443692946 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f139406b2a0 0x7f139406d760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f13ac02b9a0 tx=0x7f13ac03b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:31.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:31.894+0000 7f13a6ffd640 1 -- 172.21.15.98:0/1443692946 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13a0053f70 con 0x7f13b00708e0 2023-11-28T17:43:32.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:32.153+0000 7f13b72f1640 1 -- 172.21.15.98:0/1443692946 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f13b0074db0 con 0x7f13b00708e0 2023-11-28T17:43:32.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:32.154+0000 7f13a6ffd640 1 -- 172.21.15.98:0/1443692946 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7f13a0017620 con 0x7f13b00708e0 2023-11-28T17:43:32.155 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":18,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:43:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:32.157+0000 7f13b72f1640 1 -- 172.21.15.98:0/1443692946 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f139406b2a0 msgr2=0x7f139406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:32.157+0000 7f13b72f1640 1 --2- 172.21.15.98:0/1443692946 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f139406b2a0 0x7f139406d760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f13ac02b9a0 tx=0x7f13ac03b040 comp rx=0 tx=0).stop 2023-11-28T17:43:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:32.157+0000 7f13b72f1640 1 -- 172.21.15.98:0/1443692946 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13b00708e0 msgr2=0x7f13b0141380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:32.157+0000 7f13b72f1640 1 --2- 172.21.15.98:0/1443692946 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13b00708e0 0x7f13b0141380 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f13a000c470 tx=0x7f13a0001e70 comp rx=0 tx=0).stop 2023-11-28T17:43:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:32.157+0000 7f13b72f1640 1 -- 172.21.15.98:0/1443692946 shutdown_connections 2023-11-28T17:43:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:32.157+0000 7f13b72f1640 1 --2- 172.21.15.98:0/1443692946 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f139406b2a0 0x7f139406d760 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:32.157+0000 7f13b72f1640 1 --2- 172.21.15.98:0/1443692946 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13b0070d30 0x7f13b0139a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:32.157+0000 7f13b72f1640 1 --2- 172.21.15.98:0/1443692946 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13b00708e0 0x7f13b0141380 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:32.157+0000 7f13b72f1640 1 --2- 172.21.15.98:0/1443692946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b006dbc0 0x7f13b01ec360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:32.157+0000 7f13b72f1640 1 -- 172.21.15.98:0/1443692946 >> 172.21.15.98:0/1443692946 conn(0x7f13b0066230 msgr2=0x7f13b006dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:32.157+0000 7f13b72f1640 1 -- 172.21.15.98:0/1443692946 shutdown_connections 2023-11-28T17:43:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:32.157+0000 7f13b72f1640 1 -- 172.21.15.98:0/1443692946 wait complete. 2023-11-28T17:43:32.358 INFO:tasks.workunit.client.0.smithi098.stdout:checking for style of include used by make... GNU 2023-11-28T17:43:32.384 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gcc... gcc 2023-11-28T17:43:32.632 INFO:tasks.workunit.client.0.smithi098.stdout:checking for C compiler default output file name... a.out 2023-11-28T17:43:32.645 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether the C compiler works... yes 2023-11-28T17:43:33.234 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether we are cross compiling... no 2023-11-28T17:43:33.404 INFO:tasks.workunit.client.0.smithi098.stdout:checking for suffix of executables... 2023-11-28T17:43:33.623 INFO:tasks.workunit.client.0.smithi098.stdout:checking for suffix of object files... o 2023-11-28T17:43:33.851 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether we are using the GNU C compiler... yes 2023-11-28T17:43:34.074 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether gcc accepts -g... yes 2023-11-28T17:43:34.372 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gcc option to accept ISO C89... none needed 2023-11-28T17:43:35.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:43:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.392+0000 7efdde5db640 1 Processor -- start 2023-11-28T17:43:35.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.392+0000 7efdde5db640 1 -- start start 2023-11-28T17:43:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.393+0000 7efdde5db640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 0x7efdd815a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.393+0000 7efdde5db640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 0x7efdd80ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.393+0000 7efdde5db640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd815d330 con 0x7efdd80cf280 2023-11-28T17:43:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.393+0000 7efdde5db640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdd815d120 con 0x7efdd80ca390 2023-11-28T17:43:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.393+0000 7efdde5db640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdd815d610 con 0x7efdd8153bb0 2023-11-28T17:43:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.394+0000 7efddcdd8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 0x7efdd80ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.394+0000 7efddcdd8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 0x7efdd80ca7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36566/0 (socket says 172.21.15.98:36566) 2023-11-28T17:43:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.394+0000 7efddcdd8640 1 -- 172.21.15.98:0/2122317461 learned_addr learned my addr 172.21.15.98:0/2122317461 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.394+0000 7efddcdd8640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 msgr2=0x7efdd80ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:43:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.394+0000 7efddcdd8640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 msgr2=0x7efdd80ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:35.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.394+0000 7efddcdd8640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 0x7efdd80ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:35.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.394+0000 7efddcdd8640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 0x7efdd80ca7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:43:35.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.394+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 0x7efdd815a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:35.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.395+0000 7efddd5d9640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 msgr2=0x7efdd815a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:43:35.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.395+0000 7efddd5d9640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 msgr2=0x7efdd815a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:35.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.395+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 0x7efdd815a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:35.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.395+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 0x7efdd815a6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:43:35.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.595+0000 7efddcdd8640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 0x7efdd80ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.596+0000 7efddcdd8640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 msgr2=0x7efdd80ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:43:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.596+0000 7efddcdd8640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 msgr2=0x7efdd80ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.596+0000 7efddcdd8640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 0x7efdd80ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.596+0000 7efddcdd8640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 0x7efdd80ca7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:43:35.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.596+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 0x7efdd815a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.597+0000 7efddd5d9640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 msgr2=0x7efdd815a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:43:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.597+0000 7efddd5d9640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 msgr2=0x7efdd815a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.597+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 0x7efdd815a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.597+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 0x7efdd815a6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:43:35.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.997+0000 7efddcdd8640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 0x7efdd80ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:35.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.998+0000 7efddcdd8640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 msgr2=0x7efdd80ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:43:35.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.998+0000 7efddcdd8640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 msgr2=0x7efdd80ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:35.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.998+0000 7efddcdd8640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 0x7efdd80ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:35.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.998+0000 7efddcdd8640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 0x7efdd80ca7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:43:35.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.998+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 0x7efdd815a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:36.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.999+0000 7efddd5d9640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 msgr2=0x7efdd815a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:43:36.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.999+0000 7efddd5d9640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 msgr2=0x7efdd815a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:36.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.999+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 0x7efdd815a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:36.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:35.999+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 0x7efdd815a6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:43:36.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:36.799+0000 7efddcdd8640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 0x7efdd80ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:36.800+0000 7efddcdd8640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 msgr2=0x7efdd80ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:43:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:36.800+0000 7efddcdd8640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 msgr2=0x7efdd80ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:36.800+0000 7efddcdd8640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 0x7efdd80ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:36.800+0000 7efddcdd8640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 0x7efdd80ca7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:43:36.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:36.800+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 0x7efdd815a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:36.801+0000 7efddd5d9640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 msgr2=0x7efdd815a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:43:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:36.801+0000 7efddd5d9640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 msgr2=0x7efdd815a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:43:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:36.801+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 0x7efdd815a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:36.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:36.801+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 0x7efdd815a6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:43:37.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.525+0000 7efdcffff640 1 -- 172.21.15.98:0/2122317461 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943493647 0 0) 0x7efdd815d330 con 0x7efdd80cf280 2023-11-28T17:43:37.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.525+0000 7efdcffff640 1 -- 172.21.15.98:0/2122317461 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc4003520 con 0x7efdd80cf280 2023-11-28T17:43:37.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.544+0000 7efdcffff640 1 -- 172.21.15.98:0/2122317461 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3219857703 0 0) 0x7efdc4003520 con 0x7efdd80cf280 2023-11-28T17:43:37.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.544+0000 7efdcffff640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 msgr2=0x7efdd815a6d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:43:37.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.544+0000 7efdcffff640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.113:3300/0 conn(0x7efdd8153bb0 0x7efdd815a6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:37.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.545+0000 7efdcffff640 1 -- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 msgr2=0x7efdd80ca7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:43:37.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.545+0000 7efdcffff640 1 --2- 172.21.15.98:0/2122317461 >> v2:172.21.15.98:3300/0 conn(0x7efdd80ca390 0x7efdd80ca7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:37.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.545+0000 7efdcffff640 1 -- 172.21.15.98:0/2122317461 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd815d8f0 con 0x7efdd80cf280 2023-11-28T17:43:37.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.545+0000 7efdcffff640 1 -- 172.21.15.98:0/2122317461 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7efdd00019f0 con 0x7efdd80cf280 2023-11-28T17:43:37.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.545+0000 7efdcffff640 1 -- 172.21.15.98:0/2122317461 >> v1:172.21.15.98:6789/0 conn(0x7efdd80cf280 legacy=0x7efdd80b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:37.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.545+0000 7efdcffff640 1 --2- 172.21.15.98:0/2122317461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc4003970 0x7efdc4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:37.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.545+0000 7efdcffff640 1 --2- 172.21.15.98:0/2122317461 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efdc40047c0 0x7efdc4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:37.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.545+0000 7efdcffff640 1 --2- 172.21.15.98:0/2122317461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdc4005180 0x7efdc400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:37.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.545+0000 7efdcffff640 1 -- 172.21.15.98:0/2122317461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd815d120 con 0x7efdc4003970 2023-11-28T17:43:37.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.545+0000 7efdcffff640 1 -- 172.21.15.98:0/2122317461 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd815d610 con 0x7efdc4005180 2023-11-28T17:43:37.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.545+0000 7efdcffff640 1 -- 172.21.15.98:0/2122317461 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc400c080 con 0x7efdc40047c0 2023-11-28T17:43:37.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.546+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2122317461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc4003970 0x7efdc4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:37.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.546+0000 7efddddda640 1 --2- 172.21.15.98:0/2122317461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdc4005180 0x7efdc400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:37.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.547+0000 7efddddda640 1 -- 172.21.15.98:0/2122317461 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efdc40047c0 msgr2=0x7efdc4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:37.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.547+0000 7efddddda640 1 --2- 172.21.15.98:0/2122317461 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efdc40047c0 0x7efdc4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:37.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.547+0000 7efddddda640 1 -- 172.21.15.98:0/2122317461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc4003970 msgr2=0x7efdc4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:37.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.547+0000 7efddddda640 1 --2- 172.21.15.98:0/2122317461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc4003970 0x7efdc4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:37.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.547+0000 7efddddda640 1 -- 172.21.15.98:0/2122317461 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc4003520 con 0x7efdc4005180 2023-11-28T17:43:37.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.547+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2122317461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdc4003970 0x7efdc4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:43:37.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.549+0000 7efddddda640 1 -- 172.21.15.98:0/2122317461 reap_dead start 2023-11-28T17:43:37.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.549+0000 7efddddda640 1 --2- 172.21.15.98:0/2122317461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdc4005180 0x7efdc400b980 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efdd000c4f0 tx=0x7efdc40057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3220643110a2bfe1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:37.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.550+0000 7efdcffff640 1 -- 172.21.15.98:0/2122317461 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd002cd60 con 0x7efdc4005180 2023-11-28T17:43:37.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.550+0000 7efdcffff640 1 -- 172.21.15.98:0/2122317461 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd0034440 con 0x7efdc4005180 2023-11-28T17:43:37.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.550+0000 7efdcffff640 1 -- 172.21.15.98:0/2122317461 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd00347f0 con 0x7efdc4005180 2023-11-28T17:43:37.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.552+0000 7efdde5db640 1 -- 172.21.15.98:0/2122317461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdc4005180 msgr2=0x7efdc400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:37.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.552+0000 7efdde5db640 1 --2- 172.21.15.98:0/2122317461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdc4005180 0x7efdc400b980 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efdd000c4f0 tx=0x7efdc40057f0 comp rx=0 tx=0).stop 2023-11-28T17:43:37.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.552+0000 7efdde5db640 1 -- 172.21.15.98:0/2122317461 shutdown_connections 2023-11-28T17:43:37.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.552+0000 7efdde5db640 1 --2- 172.21.15.98:0/2122317461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdc4005180 0x7efdc400b980 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:37.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.552+0000 7efdde5db640 1 -- 172.21.15.98:0/2122317461 >> 172.21.15.98:0/2122317461 conn(0x7efdd8071c00 msgr2=0x7efdd8072010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:37.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.552+0000 7efdde5db640 1 -- 172.21.15.98:0/2122317461 shutdown_connections 2023-11-28T17:43:37.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.553+0000 7efdde5db640 1 -- 172.21.15.98:0/2122317461 wait complete. 2023-11-28T17:43:37.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.558+0000 7efdde5db640 1 Processor -- start 2023-11-28T17:43:37.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.559+0000 7efdde5db640 1 -- start start 2023-11-28T17:43:37.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.560+0000 7efdde5db640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efdc4005180 0x7efdd8153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:37.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.560+0000 7efdde5db640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdd8152170 0x7efdd8151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:37.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.560+0000 7efdde5db640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd81e2f10 0x7efdd81f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:37.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.560+0000 7efdde5db640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd81f03e0 con 0x7efdd81e2f10 2023-11-28T17:43:37.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.560+0000 7efdde5db640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd81e5ab0 con 0x7efdd8152170 2023-11-28T17:43:37.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.560+0000 7efdde5db640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd81e5d90 con 0x7efdc4005180 2023-11-28T17:43:37.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.561+0000 7efddddda640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd81e2f10 0x7efdd81f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:37.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.561+0000 7efddddda640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd81e2f10 0x7efdd81f2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36618/0 (socket says 172.21.15.98:36618) 2023-11-28T17:43:37.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.561+0000 7efddddda640 1 -- 172.21.15.98:0/2169554379 learned_addr learned my addr 172.21.15.98:0/2169554379 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:37.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.561+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2169554379 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efdc4005180 0x7efdd8153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:37.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.561+0000 7efddcdd8640 1 --2- 172.21.15.98:0/2169554379 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdd8152170 0x7efdd8151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:37.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.561+0000 7efddddda640 1 -- 172.21.15.98:0/2169554379 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efdc4005180 msgr2=0x7efdd8153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:37.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.561+0000 7efddddda640 1 --2- 172.21.15.98:0/2169554379 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efdc4005180 0x7efdd8153040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:37.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.561+0000 7efddddda640 1 -- 172.21.15.98:0/2169554379 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdd8152170 msgr2=0x7efdd8151600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:37.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.561+0000 7efddddda640 1 --2- 172.21.15.98:0/2169554379 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdd8152170 0x7efdd8151600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:37.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.561+0000 7efddddda640 1 -- 172.21.15.98:0/2169554379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd81ef620 con 0x7efdd81e2f10 2023-11-28T17:43:37.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.562+0000 7efddddda640 1 --2- 172.21.15.98:0/2169554379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd81e2f10 0x7efdd81f2a80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efdd0001cb0 tx=0x7efdd80b34b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:37.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.562+0000 7efdcdffb640 1 -- 172.21.15.98:0/2169554379 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd00050c0 con 0x7efdd81e2f10 2023-11-28T17:43:37.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.562+0000 7efdcdffb640 1 -- 172.21.15.98:0/2169554379 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd0005220 con 0x7efdd81e2f10 2023-11-28T17:43:37.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.562+0000 7efdde5db640 1 -- 172.21.15.98:0/2169554379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd81ef900 con 0x7efdd81e2f10 2023-11-28T17:43:37.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.562+0000 7efdcdffb640 1 -- 172.21.15.98:0/2169554379 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd0005590 con 0x7efdd81e2f10 2023-11-28T17:43:37.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.562+0000 7efdde5db640 1 -- 172.21.15.98:0/2169554379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd81591e0 con 0x7efdd81e2f10 2023-11-28T17:43:37.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.564+0000 7efdde5db640 1 -- 172.21.15.98:0/2169554379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd81f0100 con 0x7efdd81e2f10 2023-11-28T17:43:37.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.565+0000 7efdcdffb640 1 -- 172.21.15.98:0/2169554379 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7efdd003b040 con 0x7efdd81e2f10 2023-11-28T17:43:37.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.566+0000 7efdcdffb640 1 --2- 172.21.15.98:0/2169554379 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7efdbc06b2a0 0x7efdbc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:37.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.566+0000 7efdcdffb640 1 -- 172.21.15.98:0/2169554379 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7efdd00aaba0 con 0x7efdd81e2f10 2023-11-28T17:43:37.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.566+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2169554379 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7efdbc06b2a0 0x7efdbc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:37.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.567+0000 7efddd5d9640 1 --2- 172.21.15.98:0/2169554379 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7efdbc06b2a0 0x7efdbc06d760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efdd4006fd0 tx=0x7efdd4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:37.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.572+0000 7efdcdffb640 1 -- 172.21.15.98:0/2169554379 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efdd81f0100 con 0x7efdd81e2f10 2023-11-28T17:43:37.691 INFO:tasks.workunit.client.0.smithi098.stdout:checking dependency style of gcc... gcc3 2023-11-28T17:43:37.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.855+0000 7efdde5db640 1 -- 172.21.15.98:0/2169554379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efdd81f0100 con 0x7efdd81e2f10 2023-11-28T17:43:37.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.856+0000 7efdcdffb640 1 -- 172.21.15.98:0/2169554379 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7efdd81f0100 con 0x7efdd81e2f10 2023-11-28T17:43:37.858 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:43:37.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.859+0000 7efdde5db640 1 -- 172.21.15.98:0/2169554379 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7efdbc06b2a0 msgr2=0x7efdbc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:37.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.859+0000 7efdde5db640 1 --2- 172.21.15.98:0/2169554379 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7efdbc06b2a0 0x7efdbc06d760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efdd4006fd0 tx=0x7efdd4008040 comp rx=0 tx=0).stop 2023-11-28T17:43:37.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.859+0000 7efdde5db640 1 -- 172.21.15.98:0/2169554379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd81e2f10 msgr2=0x7efdd81f2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:37.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.859+0000 7efdde5db640 1 --2- 172.21.15.98:0/2169554379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd81e2f10 0x7efdd81f2a80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efdd0001cb0 tx=0x7efdd80b34b0 comp rx=0 tx=0).stop 2023-11-28T17:43:37.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.859+0000 7efdde5db640 1 -- 172.21.15.98:0/2169554379 shutdown_connections 2023-11-28T17:43:37.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.859+0000 7efdde5db640 1 --2- 172.21.15.98:0/2169554379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdd81e2f10 0x7efdd81f2a80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:37.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.859+0000 7efdde5db640 1 --2- 172.21.15.98:0/2169554379 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7efdbc06b2a0 0x7efdbc06d760 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:37.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.859+0000 7efdde5db640 1 --2- 172.21.15.98:0/2169554379 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdd8152170 0x7efdd8151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:37.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.859+0000 7efdde5db640 1 --2- 172.21.15.98:0/2169554379 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efdc4005180 0x7efdd8153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:37.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.859+0000 7efdde5db640 1 -- 172.21.15.98:0/2169554379 >> 172.21.15.98:0/2169554379 conn(0x7efdd8071c00 msgr2=0x7efdd80b4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:37.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.859+0000 7efdde5db640 1 -- 172.21.15.98:0/2169554379 shutdown_connections 2023-11-28T17:43:37.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:37.859+0000 7efdde5db640 1 -- 172.21.15.98:0/2169554379 wait complete. 2023-11-28T17:43:37.879 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:43:37.879 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:43:37.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:43:37.982 INFO:tasks.workunit.client.0.smithi098.stdout:checking how to run the C preprocessor... gcc -E 2023-11-28T17:43:38.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.077+0000 7fe638183640 1 Processor -- start 2023-11-28T17:43:38.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.077+0000 7fe638183640 1 -- start start 2023-11-28T17:43:38.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.078+0000 7fe638183640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe630071380 0x7fe630070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:38.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.078+0000 7fe638183640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6300c3920 con 0x7fe6301500f0 2023-11-28T17:43:38.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.078+0000 7fe638183640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6300cb560 con 0x7fe6300caff0 2023-11-28T17:43:38.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.078+0000 7fe638183640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6300cb820 con 0x7fe630071380 2023-11-28T17:43:38.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.079+0000 7fe635f1f640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe6301500f0 0x7fe630156b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33966/0 (socket says 172.21.15.98:33966) 2023-11-28T17:43:38.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.079+0000 7fe635f1f640 1 -- 172.21.15.98:0/3333366140 learned_addr learned my addr 172.21.15.98:0/3333366140 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:43:38.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.079+0000 7fe63571e640 1 --2- 172.21.15.98:0/3333366140 >> v2:172.21.15.98:3300/0 conn(0x7fe630071380 0x7fe630070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.079+0000 7fe63571e640 1 -- 172.21.15.98:0/3333366140 >> v1:172.21.15.113:6789/0 conn(0x7fe6300caff0 legacy=0x7fe6300c4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.079+0000 7fe63571e640 1 -- 172.21.15.98:0/3333366140 >> v1:172.21.15.98:6789/0 conn(0x7fe6301500f0 legacy=0x7fe630156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.079+0000 7fe63571e640 1 -- 172.21.15.98:0/3333366140 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6300c3b90 con 0x7fe630071380 2023-11-28T17:43:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.080+0000 7fe63571e640 1 --2- 172.21.15.98:0/3333366140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630071380 0x7fe630070520 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe6200097a0 tx=0x7fe62002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8cee6536baead1b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:38.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.081+0000 7fe634f1d640 1 -- 172.21.15.98:0/3333366140 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620030610 con 0x7fe630071380 2023-11-28T17:43:38.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.081+0000 7fe634f1d640 1 -- 172.21.15.98:0/3333366140 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe620030770 con 0x7fe630071380 2023-11-28T17:43:38.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.081+0000 7fe634f1d640 1 -- 172.21.15.98:0/3333366140 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620030b00 con 0x7fe630071380 2023-11-28T17:43:38.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.081+0000 7fe638183640 1 -- 172.21.15.98:0/3333366140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630071380 msgr2=0x7fe630070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:38.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.081+0000 7fe638183640 1 --2- 172.21.15.98:0/3333366140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630071380 0x7fe630070520 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe6200097a0 tx=0x7fe62002b960 comp rx=0 tx=0).stop 2023-11-28T17:43:38.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.081+0000 7fe638183640 1 -- 172.21.15.98:0/3333366140 shutdown_connections 2023-11-28T17:43:38.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.081+0000 7fe638183640 1 --2- 172.21.15.98:0/3333366140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630071380 0x7fe630070520 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:38.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.081+0000 7fe638183640 1 -- 172.21.15.98:0/3333366140 >> 172.21.15.98:0/3333366140 conn(0x7fe63006c0f0 msgr2=0x7fe63006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:38.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.082+0000 7fe638183640 1 -- 172.21.15.98:0/3333366140 shutdown_connections 2023-11-28T17:43:38.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.082+0000 7fe638183640 1 -- 172.21.15.98:0/3333366140 wait complete. 2023-11-28T17:43:38.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.083+0000 7fe638183640 1 Processor -- start 2023-11-28T17:43:38.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.084+0000 7fe638183640 1 -- start start 2023-11-28T17:43:38.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.084+0000 7fe638183640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630071380 0x7fe63013c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:38.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.084+0000 7fe638183640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6300caff0 0x7fe63014c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:38.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.084+0000 7fe638183640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6301500f0 0x7fe630141530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:38.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.084+0000 7fe638183640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe630142850 con 0x7fe630071380 2023-11-28T17:43:38.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.084+0000 7fe638183640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe630139560 con 0x7fe6300caff0 2023-11-28T17:43:38.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.084+0000 7fe638183640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63013a0e0 con 0x7fe6301500f0 2023-11-28T17:43:38.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.085+0000 7fe635f1f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630071380 0x7fe63013c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:38.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.085+0000 7fe635f1f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630071380 0x7fe63013c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36642/0 (socket says 172.21.15.98:36642) 2023-11-28T17:43:38.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.085+0000 7fe635f1f640 1 -- 172.21.15.98:0/2968479747 learned_addr learned my addr 172.21.15.98:0/2968479747 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:38.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.085+0000 7fe63571e640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6300caff0 0x7fe63014c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:38.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.085+0000 7fe635f1f640 1 -- 172.21.15.98:0/2968479747 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6301500f0 msgr2=0x7fe630141530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:38.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.085+0000 7fe635f1f640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6301500f0 0x7fe630141530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:38.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.085+0000 7fe635f1f640 1 -- 172.21.15.98:0/2968479747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6300caff0 msgr2=0x7fe63014c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:38.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.085+0000 7fe635f1f640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6300caff0 0x7fe63014c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:38.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.085+0000 7fe635f1f640 1 -- 172.21.15.98:0/2968479747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe620009400 con 0x7fe630071380 2023-11-28T17:43:38.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.085+0000 7fe63571e640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6300caff0 0x7fe63014c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:43:38.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.085+0000 7fe635f1f640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630071380 0x7fe63013c360 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00c4b0 tx=0x7fe62c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.086+0000 7fe6267fc640 1 -- 172.21.15.98:0/2968479747 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c005ce0 con 0x7fe630071380 2023-11-28T17:43:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.086+0000 7fe6267fc640 1 -- 172.21.15.98:0/2968479747 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62c005e40 con 0x7fe630071380 2023-11-28T17:43:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.086+0000 7fe6267fc640 1 -- 172.21.15.98:0/2968479747 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c007270 con 0x7fe630071380 2023-11-28T17:43:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.086+0000 7fe638183640 1 -- 172.21.15.98:0/2968479747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe63014caf0 con 0x7fe630071380 2023-11-28T17:43:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.086+0000 7fe638183640 1 -- 172.21.15.98:0/2968479747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe630139a90 con 0x7fe630071380 2023-11-28T17:43:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.087+0000 7fe638183640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630139c80 0x7fe63014b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.087+0000 7fe638183640 1 -- 172.21.15.98:0/2968479747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300744d0 con 0x7fe630139c80 2023-11-28T17:43:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.087+0000 7fe63571e640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630139c80 0x7fe63014b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.088+0000 7fe63571e640 1 -- 172.21.15.98:0/2968479747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe630164a30 con 0x7fe630139c80 2023-11-28T17:43:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.088+0000 7fe6267fc640 1 -- 172.21.15.98:0/2968479747 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe62c0073d0 con 0x7fe630071380 2023-11-28T17:43:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.088+0000 7fe63571e640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630139c80 0x7fe63014b0a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe630071fe0 tx=0x7fe620004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.089+0000 7fe6267fc640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe61406b2a0 0x7fe61406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.089+0000 7fe6267fc640 1 -- 172.21.15.98:0/2968479747 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe62c0890e0 con 0x7fe630071380 2023-11-28T17:43:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.089+0000 7fe636720640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe61406b2a0 0x7fe61406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.089+0000 7fe6267fc640 1 -- 172.21.15.98:0/2968479747 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62003e040 con 0x7fe630139c80 2023-11-28T17:43:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.089+0000 7fe6267fc640 1 -- 172.21.15.98:0/2968479747 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fe620039c30 con 0x7fe630139c80 2023-11-28T17:43:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.089+0000 7fe6267fc640 1 -- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630139c80 msgr2=0x7fe63014b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.089+0000 7fe6267fc640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630139c80 0x7fe63014b0a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe630071fe0 tx=0x7fe620004730 comp rx=0 tx=0).stop 2023-11-28T17:43:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.090+0000 7fe6267fc640 1 -- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630139c80 msgr2=0x7fe63014b0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.090+0000 7fe6267fc640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630139c80 0x7fe63014b0a0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe630071fe0 tx=0x7fe620004730 comp rx=0 tx=0).stop 2023-11-28T17:43:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.090+0000 7fe636720640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe61406b2a0 0x7fe61406d760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe628004340 tx=0x7fe628007d30 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.097+0000 7fe638183640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe63014b5e0 0x7fe6301f3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.097+0000 7fe638183640 1 -- 172.21.15.98:0/2968479747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe630072080 con 0x7fe63014b5e0 2023-11-28T17:43:38.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.097+0000 7fe63571e640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe63014b5e0 0x7fe6301f3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:38.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.098+0000 7fe63571e640 1 -- 172.21.15.98:0/2968479747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe6301504d0 con 0x7fe63014b5e0 2023-11-28T17:43:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.098+0000 7fe63571e640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe63014b5e0 0x7fe6301f3cd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe6200097a0 tx=0x7fe6200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.098+0000 7fe6267fc640 1 -- 172.21.15.98:0/2968479747 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62004b870 con 0x7fe63014b5e0 2023-11-28T17:43:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.099+0000 7fe6267fc640 1 -- 172.21.15.98:0/2968479747 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4807 (secure 0 0 0) 0x7fe62004b310 con 0x7fe63014b5e0 2023-11-28T17:43:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.099+0000 7fe6267fc640 1 -- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe63014b5e0 msgr2=0x7fe6301f3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.099+0000 7fe6267fc640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe63014b5e0 0x7fe6301f3cd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe6200097a0 tx=0x7fe6200046c0 comp rx=0 tx=0).stop 2023-11-28T17:43:38.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.099+0000 7fe6267fc640 1 -- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe63014b5e0 msgr2=0x7fe6301f3cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:38.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.099+0000 7fe6267fc640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe63014b5e0 0x7fe6301f3cd0 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe6200097a0 tx=0x7fe6200046c0 comp rx=0 tx=0).stop 2023-11-28T17:43:38.101 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:43:38.101 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:43:38.101 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:43:38.102 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:43:38.102 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2023-11-28T17:43:38.102 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:43:38.102 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:43:38.102 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:43:38.102 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:43:38.102 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:43:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:43:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:43:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:38.103 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:38.104 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:43:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:43:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:38.105 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:38.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:38.107 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:43:38.107 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:43:38.107 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:43:38.107 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:43:38.107 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:43:38.107 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:43:38.108 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:38.108 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:38.108 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:43:38.108 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:43:38.108 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:43:38.108 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:43:38.108 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:43:38.109 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:43:38.109 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:43:38.109 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:38.109 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:43:38.109 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:38.109 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:38.110 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:38.110 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:38.110 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:38.110 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:38.110 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:38.110 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:38.111 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:38.111 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:38.111 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:38.111 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:43:38.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:38.111 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:43:38.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:38.112 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:43:38.112 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:43:38.112 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:38.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:38.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:38.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:38.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:43:38.113 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:38.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:38.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:38.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:38.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:43:38.114 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:38.114 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:38.114 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:38.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:38.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:43:38.114 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:43:38.115 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:38.115 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:38.115 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:38.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:38.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:38.115 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:43:38.115 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:43:38.116 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:38.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:38.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:38.116 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:38.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:43:38.116 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:38.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:38.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:38.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:38.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:43:38.117 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:38.117 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:38.118 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:38.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:38.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:43:38.118 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:43:38.118 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:38.119 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:38.119 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:38.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:38.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:38.119 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:43:38.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:43:38.120 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:38.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:38.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:38.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:38.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:43:38.121 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:38.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:38.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:38.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:38.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:43:38.123 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:38.123 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:38.123 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:38.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:38.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:43:38.124 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:43:38.124 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:38.124 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:38.124 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:38.124 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:38.125 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:38.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:38.125 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:43:38.125 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:43:38.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:43:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:38.126 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:43:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:43:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:38.127 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:43:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:43:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:43:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:43:38.129 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:38.130 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:43:38.130 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:43:38.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.099+0000 7fe638183640 1 -- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe61406b2a0 msgr2=0x7fe61406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:38.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.099+0000 7fe638183640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe61406b2a0 0x7fe61406d760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe628004340 tx=0x7fe628007d30 comp rx=0 tx=0).stop 2023-11-28T17:43:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.099+0000 7fe638183640 1 -- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630071380 msgr2=0x7fe63013c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.099+0000 7fe638183640 1 --2- 172.21.15.98:0/2968479747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe630071380 0x7fe63013c360 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00c4b0 tx=0x7fe62c00c970 comp rx=0 tx=0).stop 2023-11-28T17:43:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.099+0000 7fe636720640 1 -- 172.21.15.98:0/2968479747 reap_dead start 2023-11-28T17:43:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.099+0000 7fe636720640 1 -- 172.21.15.98:0/2968479747 reap_dead start 2023-11-28T17:43:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.099+0000 7fe638183640 1 -- 172.21.15.98:0/2968479747 shutdown_connections 2023-11-28T17:43:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.099+0000 7fe638183640 1 -- 172.21.15.98:0/2968479747 >> 172.21.15.98:0/2968479747 conn(0x7fe63006c0f0 msgr2=0x7fe63006dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.099+0000 7fe638183640 1 -- 172.21.15.98:0/2968479747 shutdown_connections 2023-11-28T17:43:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.100+0000 7fe638183640 1 -- 172.21.15.98:0/2968479747 wait complete. 2023-11-28T17:43:38.135 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:43:38.135 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:43:38.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:43:38.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:43:38.198 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:43:38.228 INFO:tasks.workunit.client.0.smithi098.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2023-11-28T17:43:38.233 INFO:tasks.workunit.client.0.smithi098.stdout:checking for egrep... /usr/bin/grep -E 2023-11-28T17:43:38.269 INFO:tasks.workunit.client.0.smithi098.stdout:checking for AIX... no 2023-11-28T17:43:38.291 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gcc... (cached) gcc 2023-11-28T17:43:38.327 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether we are using the GNU C compiler... (cached) yes 2023-11-28T17:43:38.329 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether gcc accepts -g... (cached) yes 2023-11-28T17:43:38.330 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2023-11-28T17:43:38.331 INFO:tasks.workunit.client.0.smithi098.stdout:checking dependency style of gcc... (cached) gcc3 2023-11-28T17:43:38.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.333+0000 7fe552124640 1 Processor -- start 2023-11-28T17:43:38.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.334+0000 7fe552124640 1 -- start start 2023-11-28T17:43:38.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.335+0000 7fe552124640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe54c154030 0x7fe54c070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:38.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.335+0000 7fe552124640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe54c0c8600 0x7fe54c070e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:38.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.335+0000 7fe552124640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c154410 con 0x7fe54c06dcf0 2023-11-28T17:43:38.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.335+0000 7fe552124640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54c0c5da0 con 0x7fe54c154030 2023-11-28T17:43:38.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.335+0000 7fe552124640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c15d370 con 0x7fe54c0c8600 2023-11-28T17:43:38.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.335+0000 7fe54b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe54c154030 0x7fe54c070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:38.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.335+0000 7fe54b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe54c154030 0x7fe54c070660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36682/0 (socket says 172.21.15.98:36682) 2023-11-28T17:43:38.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.335+0000 7fe54b7fe640 1 -- 172.21.15.98:0/2012354198 learned_addr learned my addr 172.21.15.98:0/2012354198 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:38.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.336+0000 7fe54affd640 1 --2- 172.21.15.98:0/2012354198 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe54c0c8600 0x7fe54c070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:38.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.336+0000 7fe54b7fe640 1 -- 172.21.15.98:0/2012354198 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe54c0c8600 msgr2=0x7fe54c070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:38.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.336+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/2012354198 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe54c0c8600 0x7fe54c070e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:38.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.336+0000 7fe54b7fe640 1 -- 172.21.15.98:0/2012354198 >> v1:172.21.15.113:6789/0 conn(0x7fe54c06dcf0 legacy=0x7fe54c0caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:38.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.336+0000 7fe54b7fe640 1 -- 172.21.15.98:0/2012354198 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c15d650 con 0x7fe54c154030 2023-11-28T17:43:38.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.336+0000 7fe54affd640 1 --2- 172.21.15.98:0/2012354198 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe54c0c8600 0x7fe54c070e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:43:38.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.337+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/2012354198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c154030 0x7fe54c070660 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe54c06eba0 tx=0x7fe53c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25fe411c16d25a5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:38.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.337+0000 7fe54a7fc640 1 -- 172.21.15.98:0/2012354198 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c030610 con 0x7fe54c154030 2023-11-28T17:43:38.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.337+0000 7fe54a7fc640 1 -- 172.21.15.98:0/2012354198 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53c030770 con 0x7fe54c154030 2023-11-28T17:43:38.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.337+0000 7fe54a7fc640 1 -- 172.21.15.98:0/2012354198 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c030b00 con 0x7fe54c154030 2023-11-28T17:43:38.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.338+0000 7fe552124640 1 -- 172.21.15.98:0/2012354198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c154030 msgr2=0x7fe54c070660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:38.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.338+0000 7fe552124640 1 --2- 172.21.15.98:0/2012354198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c154030 0x7fe54c070660 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe54c06eba0 tx=0x7fe53c02b960 comp rx=0 tx=0).stop 2023-11-28T17:43:38.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.338+0000 7fe552124640 1 -- 172.21.15.98:0/2012354198 shutdown_connections 2023-11-28T17:43:38.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.338+0000 7fe552124640 1 --2- 172.21.15.98:0/2012354198 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe54c0c8600 0x7fe54c070e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:38.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.338+0000 7fe552124640 1 --2- 172.21.15.98:0/2012354198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c154030 0x7fe54c070660 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:38.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.338+0000 7fe552124640 1 -- 172.21.15.98:0/2012354198 >> 172.21.15.98:0/2012354198 conn(0x7fe54c06c0f0 msgr2=0x7fe54c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:38.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.338+0000 7fe552124640 1 -- 172.21.15.98:0/2012354198 shutdown_connections 2023-11-28T17:43:38.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.338+0000 7fe552124640 1 -- 172.21.15.98:0/2012354198 wait complete. 2023-11-28T17:43:38.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.340+0000 7fe552124640 1 Processor -- start 2023-11-28T17:43:38.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.340+0000 7fe552124640 1 -- start start 2023-11-28T17:43:38.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.341+0000 7fe552124640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c06dcf0 0x7fe54c149cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:38.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.341+0000 7fe552124640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe54c0c8600 0x7fe54c159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:38.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.341+0000 7fe552124640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c154030 0x7fe54c1eff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:38.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.341+0000 7fe552124640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c15ba40 con 0x7fe54c154030 2023-11-28T17:43:38.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.341+0000 7fe552124640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c14a380 con 0x7fe54c06dcf0 2023-11-28T17:43:38.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.341+0000 7fe552124640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c1efb00 con 0x7fe54c0c8600 2023-11-28T17:43:38.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.341+0000 7fe54bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c154030 0x7fe54c1eff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:38.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.341+0000 7fe54bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c154030 0x7fe54c1eff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36684/0 (socket says 172.21.15.98:36684) 2023-11-28T17:43:38.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.341+0000 7fe54bfff640 1 -- 172.21.15.98:0/3423656883 learned_addr learned my addr 172.21.15.98:0/3423656883 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:38.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.342+0000 7fe54affd640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe54c0c8600 0x7fe54c159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:38.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.342+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c06dcf0 0x7fe54c149cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:38.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.342+0000 7fe54bfff640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe54c0c8600 msgr2=0x7fe54c159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:38.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.342+0000 7fe54bfff640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe54c0c8600 0x7fe54c159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:38.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.342+0000 7fe54bfff640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c06dcf0 msgr2=0x7fe54c149cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:38.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.342+0000 7fe54bfff640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c06dcf0 0x7fe54c149cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:38.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.342+0000 7fe54bfff640 1 -- 172.21.15.98:0/3423656883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c0094d0 con 0x7fe54c154030 2023-11-28T17:43:38.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.342+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c06dcf0 0x7fe54c149cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:43:38.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.342+0000 7fe54bfff640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c154030 0x7fe54c1eff90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe54400c7b0 tx=0x7fe54400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:38.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.342+0000 7fe533fff640 1 -- 172.21.15.98:0/3423656883 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544004b60 con 0x7fe54c154030 2023-11-28T17:43:38.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.342+0000 7fe533fff640 1 -- 172.21.15.98:0/3423656883 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe544004cc0 con 0x7fe54c154030 2023-11-28T17:43:38.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.343+0000 7fe552124640 1 -- 172.21.15.98:0/3423656883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c14f3f0 con 0x7fe54c154030 2023-11-28T17:43:38.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.343+0000 7fe533fff640 1 -- 172.21.15.98:0/3423656883 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54400aa90 con 0x7fe54c154030 2023-11-28T17:43:38.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.343+0000 7fe552124640 1 -- 172.21.15.98:0/3423656883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c14f970 con 0x7fe54c154030 2023-11-28T17:43:38.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.344+0000 7fe552124640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:38.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.344+0000 7fe552124640 1 -- 172.21.15.98:0/3423656883 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c0744d0 con 0x7fe54c1f04d0 2023-11-28T17:43:38.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.344+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:38.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.345+0000 7fe54b7fe640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 msgr2=0x7fe54c1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2023-11-28T17:43:38.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.345+0000 7fe54b7fe640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 msgr2=0x7fe54c1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:43:38.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.345+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:38.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.345+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:43:38.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.345+0000 7fe533fff640 1 -- 172.21.15.98:0/3423656883 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe54400abf0 con 0x7fe54c154030 2023-11-28T17:43:38.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.346+0000 7fe533fff640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe53806b2d0 0x7fe53806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:38.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.346+0000 7fe533fff640 1 -- 172.21.15.98:0/3423656883 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe544089eb0 con 0x7fe54c154030 2023-11-28T17:43:38.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.346+0000 7fe54affd640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe53806b2d0 0x7fe53806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:38.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.347+0000 7fe54affd640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe53806b2d0 0x7fe53806d790 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe54c14f200 tx=0x7fe540008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:38.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.546+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.547+0000 7fe54b7fe640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 msgr2=0x7fe54c1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2023-11-28T17:43:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.547+0000 7fe54b7fe640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 msgr2=0x7fe54c1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:43:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.547+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:38.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.547+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:43:38.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.948+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:38.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.949+0000 7fe54b7fe640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 msgr2=0x7fe54c1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2023-11-28T17:43:38.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.949+0000 7fe54b7fe640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 msgr2=0x7fe54c1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:43:38.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.949+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:38.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:38.949+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:43:39.540 INFO:tasks.workunit.client.0.smithi098.stdout:checking for main in -lm... yes 2023-11-28T17:43:39.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:39.751+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:39.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:39.751+0000 7fe54b7fe640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 msgr2=0x7fe54c1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2023-11-28T17:43:39.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:39.751+0000 7fe54b7fe640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 msgr2=0x7fe54c1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:43:39.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:39.751+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:39.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:39.751+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:43:39.921 INFO:tasks.workunit.client.0.smithi098.stdout:checking for main in -lpthread... yes 2023-11-28T17:43:40.657 INFO:tasks.workunit.client.0.smithi098.stdout:checking for ANSI C header files... yes 2023-11-28T17:43:40.969 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2023-11-28T17:43:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:41.354+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:41.354+0000 7fe54b7fe640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 msgr2=0x7fe54c1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2023-11-28T17:43:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:41.354+0000 7fe54b7fe640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 msgr2=0x7fe54c1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:43:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:41.354+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:43:41.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:41.354+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:43:41.978 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/types.h... yes 2023-11-28T17:43:42.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.342+0000 7fe532ffd640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 msgr2=0x7fe54c1f2ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:43:42.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.342+0000 7fe532ffd640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.342+0000 7fe532ffd640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe524003540 0x7fe5240059d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.342+0000 7fe532ffd640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 msgr2=0x7fe54c1f2ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:42.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.342+0000 7fe532ffd640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f04d0 0x7fe54c1f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.342+0000 7fe532ffd640 1 -- 172.21.15.98:0/3423656883 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe524006080 con 0x7fe524003540 2023-11-28T17:43:42.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.343+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe524003540 0x7fe5240059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.344+0000 7fe54b7fe640 1 -- 172.21.15.98:0/3423656883 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5240061f0 con 0x7fe524003540 2023-11-28T17:43:42.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.345+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe524003540 0x7fe5240059d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe54c0c8f90 tx=0x7fe53c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:42.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.345+0000 7fe533fff640 1 -- 172.21.15.98:0/3423656883 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c039c30 con 0x7fe524003540 2023-11-28T17:43:42.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.347+0000 7fe533fff640 1 -- 172.21.15.98:0/3423656883 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fe53c039d90 con 0x7fe524003540 2023-11-28T17:43:42.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.347+0000 7fe533fff640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe524003540 msgr2=0x7fe5240059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.347+0000 7fe533fff640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe524003540 0x7fe5240059d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe54c0c8f90 tx=0x7fe53c0046c0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.347+0000 7fe533fff640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe524003540 msgr2=0x7fe5240059d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:42.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.347+0000 7fe533fff640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe524003540 0x7fe5240059d0 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe54c0c8f90 tx=0x7fe53c0046c0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.356+0000 7fe552124640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f3410 0x7fe54c1f3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.356+0000 7fe552124640 1 -- 172.21.15.98:0/3423656883 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c15dc10 con 0x7fe54c1f3410 2023-11-28T17:43:42.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.357+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f3410 0x7fe54c1f3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.358+0000 7fe54b7fe640 1 -- 172.21.15.98:0/3423656883 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe54c15e130 con 0x7fe54c1f3410 2023-11-28T17:43:42.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.358+0000 7fe54b7fe640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f3410 0x7fe54c1f3820 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe5240061c0 tx=0x7fe53c03b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:42.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.359+0000 7fe533fff640 1 -- 172.21.15.98:0/3423656883 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c043670 con 0x7fe54c1f3410 2023-11-28T17:43:42.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.360+0000 7fe533fff640 1 -- 172.21.15.98:0/3423656883 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4465 (secure 0 0 0) 0x7fe53c0437d0 con 0x7fe54c1f3410 2023-11-28T17:43:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.360+0000 7fe533fff640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f3410 msgr2=0x7fe54c1f3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.360+0000 7fe533fff640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f3410 0x7fe54c1f3820 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe5240061c0 tx=0x7fe53c03b040 comp rx=0 tx=0).stop 2023-11-28T17:43:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.360+0000 7fe533fff640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f3410 msgr2=0x7fe54c1f3820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.360+0000 7fe533fff640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe54c1f3410 0x7fe54c1f3820 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.360+0000 7fe54bfff640 1 -- 172.21.15.98:0/3423656883 reap_dead start 2023-11-28T17:43:42.362 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:43:42.362 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:43:42.362 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:43:42.363 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:43:42.363 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2023-11-28T17:43:42.363 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:43:42.363 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:43:42.363 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:43:42.363 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:43:42.363 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:42.363 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2023-11-28T17:43:42.363 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:42.363 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:43:42.363 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:43:42.363 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:42.363 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:42.364 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:42.364 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:42.364 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:42.364 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:42.364 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:42.364 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:42.364 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:42.364 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:42.364 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:42.364 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:43:42.364 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:43:42.364 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:42.364 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:42.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:42.365 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:42.365 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:42.365 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:42.365 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:42.365 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:42.365 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:42.365 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:42.365 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:42.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.365 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:43:42.365 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:43:42.365 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:43:42.365 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:43:42.366 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:43:42.366 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:43:42.366 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:42.366 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:42.366 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:43:42.366 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:43:42.366 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:43:42.366 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:43:42.367 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:43:42.367 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:43:42.367 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:43:42.367 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:42.367 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:43:42.367 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:42.367 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:42.367 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:42.368 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:42.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:42.368 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:42.368 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:42.368 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:42.368 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:42.368 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:42.368 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:42.368 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:43:42.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.368 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:43:42.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.369 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:43:42.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:43:42.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:42.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:42.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:42.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:43:42.369 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:42.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:42.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:43:42.369 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:42.369 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:42.370 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:42.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:43:42.370 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:43:42.370 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:42.370 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:42.370 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:42.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.370 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:43:42.370 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:43:42.370 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:42.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:42.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:42.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:43:42.371 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:42.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:42.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:43:42.371 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:42.371 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:42.371 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:42.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:43:42.372 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:43:42.372 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:42.372 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:42.372 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:42.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.372 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:43:42.372 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:43:42.372 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:42.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:42.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:42.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:43:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:43:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:43:42.373 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:43:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:43:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:43:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:43:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:42.374 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:43:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:43:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.375 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:43:42.375 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:43:42.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.361+0000 7fe552124640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe53806b2d0 msgr2=0x7fe53806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.361+0000 7fe552124640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe53806b2d0 0x7fe53806d790 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe54c14f200 tx=0x7fe540008040 comp rx=0 tx=0).stop 2023-11-28T17:43:42.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.361+0000 7fe552124640 1 -- 172.21.15.98:0/3423656883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c154030 msgr2=0x7fe54c1eff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.361+0000 7fe552124640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c154030 0x7fe54c1eff90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe54400c7b0 tx=0x7fe54400cc70 comp rx=0 tx=0).stop 2023-11-28T17:43:42.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.361+0000 7fe552124640 1 -- 172.21.15.98:0/3423656883 shutdown_connections 2023-11-28T17:43:42.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.361+0000 7fe552124640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe54c154030 0x7fe54c1eff90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.361+0000 7fe552124640 1 --2- 172.21.15.98:0/3423656883 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe53806b2d0 0x7fe53806d790 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.361+0000 7fe552124640 1 -- 172.21.15.98:0/3423656883 >> 172.21.15.98:0/3423656883 conn(0x7fe54c06c0f0 msgr2=0x7fe54c070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:42.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.361+0000 7fe552124640 1 -- 172.21.15.98:0/3423656883 shutdown_connections 2023-11-28T17:43:42.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.361+0000 7fe552124640 1 -- 172.21.15.98:0/3423656883 wait complete. 2023-11-28T17:43:42.385 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:43:42.385 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:43:42.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:43:42.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.649+0000 7f1199351640 1 Processor -- start 2023-11-28T17:43:42.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.650+0000 7f1199351640 1 -- start start 2023-11-28T17:43:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.651+0000 7f1199351640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f11940cb070 0x7f119406b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.651+0000 7f1199351640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f119406a660 con 0x7f11940ca280 2023-11-28T17:43:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.651+0000 7f1199351640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f119406a8b0 con 0x7f11941540d0 2023-11-28T17:43:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.651+0000 7f1199351640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11940cbdd0 con 0x7f11940cb070 2023-11-28T17:43:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.651+0000 7f11937fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f11940ca280 0x7f11941645f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56152/0 (socket says 172.21.15.98:56152) 2023-11-28T17:43:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.651+0000 7f11937fe640 1 -- 172.21.15.98:0/1125578745 learned_addr learned my addr 172.21.15.98:0/1125578745 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:43:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f1191ffb640 1 -- 172.21.15.98:0/1125578745 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363605097 0 0) 0x7f119406a660 con 0x7f11940ca280 2023-11-28T17:43:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f11927fc640 1 --2- 172.21.15.98:0/1125578745 >> v2:172.21.15.113:3300/0 conn(0x7f11940cb070 0x7f119406b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f1191ffb640 1 -- 172.21.15.98:0/1125578745 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1180003520 con 0x7f11940ca280 2023-11-28T17:43:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f1191ffb640 1 -- 172.21.15.98:0/1125578745 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3358770701 0 0) 0x7f1180003520 con 0x7f11940ca280 2023-11-28T17:43:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f1191ffb640 1 -- 172.21.15.98:0/1125578745 >> v2:172.21.15.113:3300/0 conn(0x7f11940cb070 msgr2=0x7f119406b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f1191ffb640 1 --2- 172.21.15.98:0/1125578745 >> v2:172.21.15.113:3300/0 conn(0x7f11940cb070 0x7f119406b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f1191ffb640 1 -- 172.21.15.98:0/1125578745 >> v1:172.21.15.113:6789/0 conn(0x7f11941540d0 legacy=0x7f119415dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f1191ffb640 1 -- 172.21.15.98:0/1125578745 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f119406ab90 con 0x7f11940ca280 2023-11-28T17:43:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f1191ffb640 1 -- 172.21.15.98:0/1125578745 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f118c001670 con 0x7f11940ca280 2023-11-28T17:43:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f1191ffb640 1 -- 172.21.15.98:0/1125578745 >> v1:172.21.15.98:6789/0 conn(0x7f11940ca280 legacy=0x7f11941645f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f1191ffb640 1 --2- 172.21.15.98:0/1125578745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1180003970 0x7f1180003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f1191ffb640 1 --2- 172.21.15.98:0/1125578745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11800047c0 0x7f1180004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f1191ffb640 1 --2- 172.21.15.98:0/1125578745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1180005180 0x7f118000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f1191ffb640 1 -- 172.21.15.98:0/1125578745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11940cbdd0 con 0x7f1180005180 2023-11-28T17:43:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f1191ffb640 1 -- 172.21.15.98:0/1125578745 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118000c020 con 0x7f1180003970 2023-11-28T17:43:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.652+0000 7f1191ffb640 1 -- 172.21.15.98:0/1125578745 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118000c1e0 con 0x7f11800047c0 2023-11-28T17:43:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.653+0000 7f11937fe640 1 --2- 172.21.15.98:0/1125578745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1180005180 0x7f118000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.653+0000 7f11927fc640 1 --2- 172.21.15.98:0/1125578745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11800047c0 0x7f1180004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.653+0000 7f11937fe640 1 -- 172.21.15.98:0/1125578745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11800047c0 msgr2=0x7f1180004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.653+0000 7f11937fe640 1 --2- 172.21.15.98:0/1125578745 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11800047c0 0x7f1180004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.653+0000 7f11937fe640 1 -- 172.21.15.98:0/1125578745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1180003970 msgr2=0x7f1180003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.653+0000 7f11937fe640 1 --2- 172.21.15.98:0/1125578745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1180003970 0x7f1180003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.653+0000 7f11937fe640 1 -- 172.21.15.98:0/1125578745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1180003520 con 0x7f1180005180 2023-11-28T17:43:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.654+0000 7f11937fe640 1 -- 172.21.15.98:0/1125578745 reap_dead start 2023-11-28T17:43:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.654+0000 7f11937fe640 1 --2- 172.21.15.98:0/1125578745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1180005180 0x7f118000b980 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f118c00c240 tx=0x7f11800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f8eec42824ad63 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.655+0000 7f1191ffb640 1 -- 172.21.15.98:0/1125578745 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c005ad0 con 0x7f1180005180 2023-11-28T17:43:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.655+0000 7f1191ffb640 1 -- 172.21.15.98:0/1125578745 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118c005c70 con 0x7f1180005180 2023-11-28T17:43:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.655+0000 7f1191ffb640 1 -- 172.21.15.98:0/1125578745 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c02c8b0 con 0x7f1180005180 2023-11-28T17:43:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.655+0000 7f1199351640 1 -- 172.21.15.98:0/1125578745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1180005180 msgr2=0x7f118000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.655+0000 7f1199351640 1 --2- 172.21.15.98:0/1125578745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1180005180 0x7f118000b980 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f118c00c240 tx=0x7f11800057f0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.655+0000 7f1199351640 1 -- 172.21.15.98:0/1125578745 shutdown_connections 2023-11-28T17:43:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.655+0000 7f1199351640 1 --2- 172.21.15.98:0/1125578745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1180005180 0x7f118000b980 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.655+0000 7f1199351640 1 -- 172.21.15.98:0/1125578745 >> 172.21.15.98:0/1125578745 conn(0x7f11940cf490 msgr2=0x7f11940cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.655+0000 7f1199351640 1 -- 172.21.15.98:0/1125578745 shutdown_connections 2023-11-28T17:43:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.655+0000 7f1199351640 1 -- 172.21.15.98:0/1125578745 wait complete. 2023-11-28T17:43:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.657+0000 7f1199351640 1 Processor -- start 2023-11-28T17:43:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.657+0000 7f1199351640 1 -- start start 2023-11-28T17:43:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.658+0000 7f1199351640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1180005180 0x7f11941f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.658+0000 7f1199351640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119414a290 0x7f11941518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.658+0000 7f1199351640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1194149720 0x7f1194149bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.658+0000 7f1199351640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119414aec0 con 0x7f119414a290 2023-11-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.658+0000 7f1199351640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119414bba0 con 0x7f1180005180 2023-11-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.658+0000 7f1199351640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f119414a110 con 0x7f1194149720 2023-11-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.658+0000 7f11927fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119414a290 0x7f11941518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.658+0000 7f11927fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119414a290 0x7f11941518f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41888/0 (socket says 172.21.15.98:41888) 2023-11-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.658+0000 7f11927fc640 1 -- 172.21.15.98:0/1108103331 learned_addr learned my addr 172.21.15.98:0/1108103331 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.659+0000 7f11927fc640 1 -- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1194149720 msgr2=0x7f1194149bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.659+0000 7f11937fe640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1194149720 0x7f1194149bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.659+0000 7f11927fc640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1194149720 0x7f1194149bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.659+0000 7f11927fc640 1 -- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1180005180 msgr2=0x7f11941f2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.659+0000 7f11927fc640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1180005180 0x7f11941f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.659+0000 7f11927fc640 1 -- 172.21.15.98:0/1108103331 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11941e5920 con 0x7f119414a290 2023-11-28T17:43:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.659+0000 7f11927fc640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119414a290 0x7f11941518f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f117c010890 tx=0x7f117c010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:42.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.659+0000 7f118b7fe640 1 -- 172.21.15.98:0/1108103331 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c009610 con 0x7f119414a290 2023-11-28T17:43:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.659+0000 7f118b7fe640 1 -- 172.21.15.98:0/1108103331 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117c00eb60 con 0x7f119414a290 2023-11-28T17:43:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.659+0000 7f1199351640 1 -- 172.21.15.98:0/1108103331 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11941e5070 con 0x7f119414a290 2023-11-28T17:43:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.659+0000 7f118b7fe640 1 -- 172.21.15.98:0/1108103331 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c01c400 con 0x7f119414a290 2023-11-28T17:43:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.659+0000 7f1199351640 1 -- 172.21.15.98:0/1108103331 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f119414abe0 con 0x7f119414a290 2023-11-28T17:43:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.660+0000 7f1199351640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11941e4ab0 0x7f11940c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.660+0000 7f1199351640 1 -- 172.21.15.98:0/1108103331 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1194074110 con 0x7f11941e4ab0 2023-11-28T17:43:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.661+0000 7f1192ffd640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11941e4ab0 0x7f11940c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.662+0000 7f1192ffd640 1 -- 172.21.15.98:0/1108103331 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f119406b430 con 0x7f11941e4ab0 2023-11-28T17:43:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.662+0000 7f118b7fe640 1 -- 172.21.15.98:0/1108103331 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f117c01c560 con 0x7f119414a290 2023-11-28T17:43:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.662+0000 7f1192ffd640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11941e4ab0 0x7f11940c4070 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f118400c6b0 tx=0x7f118400cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.662+0000 7f118b7fe640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f117406b2d0 0x7f117406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.663+0000 7f118b7fe640 1 -- 172.21.15.98:0/1108103331 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f117c08a270 con 0x7f119414a290 2023-11-28T17:43:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.663+0000 7f11937fe640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f117406b2d0 0x7f117406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.663+0000 7f118b7fe640 1 -- 172.21.15.98:0/1108103331 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184006d50 con 0x7f11941e4ab0 2023-11-28T17:43:42.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.663+0000 7f118b7fe640 1 -- 172.21.15.98:0/1108103331 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f1184003990 con 0x7f11941e4ab0 2023-11-28T17:43:42.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.663+0000 7f118b7fe640 1 -- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11941e4ab0 msgr2=0x7f11940c4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.663+0000 7f118b7fe640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11941e4ab0 0x7f11940c4070 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f118400c6b0 tx=0x7f118400cb70 comp rx=0 tx=0).stop 2023-11-28T17:43:42.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.663+0000 7f118b7fe640 1 -- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11941e4ab0 msgr2=0x7f11940c4070 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:42.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.663+0000 7f118b7fe640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11941e4ab0 0x7f11940c4070 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.664+0000 7f11937fe640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f117406b2d0 0x7f117406d790 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f118000bff0 tx=0x7f118c039040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:42.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.670+0000 7f1199351640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11940c45b0 0x7f11940c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.670+0000 7f1199351640 1 -- 172.21.15.98:0/1108103331 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f119406b150 con 0x7f11940c45b0 2023-11-28T17:43:42.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.671+0000 7f1192ffd640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11940c45b0 0x7f11940c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.672+0000 7f1192ffd640 1 -- 172.21.15.98:0/1108103331 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f11940c6890 con 0x7f11940c45b0 2023-11-28T17:43:42.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.672+0000 7f1192ffd640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11940c45b0 0x7f11940c49c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1184014000 tx=0x7f1184001510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:42.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.673+0000 7f118b7fe640 1 -- 172.21.15.98:0/1108103331 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184020070 con 0x7f11940c45b0 2023-11-28T17:43:42.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.673+0000 7f118b7fe640 1 -- 172.21.15.98:0/1108103331 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4975 (secure 0 0 0) 0x7f11840050d0 con 0x7f11940c45b0 2023-11-28T17:43:42.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.673+0000 7f118b7fe640 1 -- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11940c45b0 msgr2=0x7f11940c49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.673+0000 7f118b7fe640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11940c45b0 0x7f11940c49c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1184014000 tx=0x7f1184001510 comp rx=0 tx=0).stop 2023-11-28T17:43:42.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.673+0000 7f118b7fe640 1 -- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11940c45b0 msgr2=0x7f11940c49c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:42.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.673+0000 7f118b7fe640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f11940c45b0 0x7f11940c49c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.675 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:43:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:43:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:43:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:43:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2023-11-28T17:43:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:43:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:43:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:43:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:43:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2023-11-28T17:43:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:43:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:43:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:43:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:43:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:43:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:43:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:43:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:43:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:43:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:43:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:43:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:43:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:43:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:43:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:43:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:43:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:43:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:43:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:43:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:43:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:43:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:43:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:43:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:43:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:43:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:43:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:43:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:43:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:43:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:43:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:43:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:43:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:43:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:43:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:43:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:43:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:43:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:43:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:43:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:43:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:43:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:43:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:43:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:43:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 6 2023-11-28T17:43:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:43:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:43:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:43:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:43:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:42.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:42.689 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:43:42.689 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:43:42.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.674+0000 7f1199351640 1 -- 172.21.15.98:0/1108103331 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f117406b2d0 msgr2=0x7f117406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.674+0000 7f1199351640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f117406b2d0 0x7f117406d790 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f118000bff0 tx=0x7f118c039040 comp rx=0 tx=0).stop 2023-11-28T17:43:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.674+0000 7f1199351640 1 -- 172.21.15.98:0/1108103331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119414a290 msgr2=0x7f11941518f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.674+0000 7f1199351640 1 --2- 172.21.15.98:0/1108103331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119414a290 0x7f11941518f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f117c010890 tx=0x7f117c010d50 comp rx=0 tx=0).stop 2023-11-28T17:43:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.674+0000 7f11937fe640 1 -- 172.21.15.98:0/1108103331 reap_dead start 2023-11-28T17:43:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.674+0000 7f11937fe640 1 -- 172.21.15.98:0/1108103331 reap_dead start 2023-11-28T17:43:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.674+0000 7f1199351640 1 -- 172.21.15.98:0/1108103331 shutdown_connections 2023-11-28T17:43:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.674+0000 7f1199351640 1 -- 172.21.15.98:0/1108103331 >> 172.21.15.98:0/1108103331 conn(0x7f11940cf490 msgr2=0x7f11940c6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.674+0000 7f1199351640 1 -- 172.21.15.98:0/1108103331 shutdown_connections 2023-11-28T17:43:42.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.674+0000 7f1199351640 1 -- 172.21.15.98:0/1108103331 wait complete. 2023-11-28T17:43:42.691 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:43:42.691 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-11-28T17:43:42.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-11-28T17:43:42.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.900+0000 7f69e8b05640 1 Processor -- start 2023-11-28T17:43:42.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.901+0000 7f69e8b05640 1 -- start start 2023-11-28T17:43:42.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.902+0000 7f69e8b05640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f69e40cf6d0 0x7f69e40c7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.902+0000 7f69e8b05640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e415aa10 con 0x7f69e415a590 2023-11-28T17:43:42.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.902+0000 7f69e8b05640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e415acf0 con 0x7f69e40cfab0 2023-11-28T17:43:42.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.902+0000 7f69e8b05640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e415d430 con 0x7f69e40cf6d0 2023-11-28T17:43:42.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.902+0000 7f69e2d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f69e415a590 0x7f69e4153cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56164/0 (socket says 172.21.15.98:56164) 2023-11-28T17:43:42.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.902+0000 7f69e2d9d640 1 -- 172.21.15.98:0/4023755458 learned_addr learned my addr 172.21.15.98:0/4023755458 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:43:42.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e1d9b640 1 -- 172.21.15.98:0/4023755458 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699100047 0 0) 0x7f69e415aa10 con 0x7f69e415a590 2023-11-28T17:43:42.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e259c640 1 --2- 172.21.15.98:0/4023755458 >> v2:172.21.15.113:3300/0 conn(0x7f69e40cf6d0 0x7f69e40c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e1d9b640 1 -- 172.21.15.98:0/4023755458 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69d0003520 con 0x7f69e415a590 2023-11-28T17:43:42.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e1d9b640 1 -- 172.21.15.98:0/4023755458 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1301402854 0 0) 0x7f69d0003520 con 0x7f69e415a590 2023-11-28T17:43:42.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e1d9b640 1 -- 172.21.15.98:0/4023755458 >> v2:172.21.15.113:3300/0 conn(0x7f69e40cf6d0 msgr2=0x7f69e40c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:42.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e1d9b640 1 --2- 172.21.15.98:0/4023755458 >> v2:172.21.15.113:3300/0 conn(0x7f69e40cf6d0 0x7f69e40c7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e1d9b640 1 -- 172.21.15.98:0/4023755458 >> v1:172.21.15.113:6789/0 conn(0x7f69e40cfab0 legacy=0x7f69e4159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:42.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e1d9b640 1 -- 172.21.15.98:0/4023755458 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e415d710 con 0x7f69e415a590 2023-11-28T17:43:42.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e1d9b640 1 -- 172.21.15.98:0/4023755458 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f69dc0019f0 con 0x7f69e415a590 2023-11-28T17:43:42.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e1d9b640 1 -- 172.21.15.98:0/4023755458 >> v1:172.21.15.98:6789/0 conn(0x7f69e415a590 legacy=0x7f69e4153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e1d9b640 1 --2- 172.21.15.98:0/4023755458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69d0003970 0x7f69d0003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e1d9b640 1 --2- 172.21.15.98:0/4023755458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69d00047c0 0x7f69d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e1d9b640 1 --2- 172.21.15.98:0/4023755458 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69d0005180 0x7f69d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e1d9b640 1 -- 172.21.15.98:0/4023755458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e415d430 con 0x7f69d0003970 2023-11-28T17:43:42.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e1d9b640 1 -- 172.21.15.98:0/4023755458 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d000c0b0 con 0x7f69d00047c0 2023-11-28T17:43:42.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.903+0000 7f69e1d9b640 1 -- 172.21.15.98:0/4023755458 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69d000c270 con 0x7f69d0005180 2023-11-28T17:43:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.904+0000 7f69e259c640 1 --2- 172.21.15.98:0/4023755458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69d0003970 0x7f69d0003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.904+0000 7f69e2d9d640 1 --2- 172.21.15.98:0/4023755458 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69d0005180 0x7f69d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.904+0000 7f69dbfff640 1 --2- 172.21.15.98:0/4023755458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69d00047c0 0x7f69d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.904+0000 7f69e259c640 1 -- 172.21.15.98:0/4023755458 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69d0005180 msgr2=0x7f69d000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.904+0000 7f69e259c640 1 --2- 172.21.15.98:0/4023755458 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69d0005180 0x7f69d000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.904+0000 7f69e259c640 1 -- 172.21.15.98:0/4023755458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69d00047c0 msgr2=0x7f69d0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.904+0000 7f69e259c640 1 --2- 172.21.15.98:0/4023755458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69d00047c0 0x7f69d0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.904+0000 7f69e259c640 1 -- 172.21.15.98:0/4023755458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d0003520 con 0x7f69d0003970 2023-11-28T17:43:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.904+0000 7f69dbfff640 1 --2- 172.21.15.98:0/4023755458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69d00047c0 0x7f69d0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:43:42.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.904+0000 7f69e2d9d640 1 -- 172.21.15.98:0/4023755458 reap_dead start 2023-11-28T17:43:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.905+0000 7f69e259c640 1 --2- 172.21.15.98:0/4023755458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69d0003970 0x7f69d0003d70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f69d400ba00 tx=0x7f69d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc26e93b4c9cb27b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:42.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.905+0000 7f69e1d9b640 1 -- 172.21.15.98:0/4023755458 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4033680 con 0x7f69d0003970 2023-11-28T17:43:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.905+0000 7f69e1d9b640 1 -- 172.21.15.98:0/4023755458 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d40337e0 con 0x7f69d0003970 2023-11-28T17:43:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.905+0000 7f69e1d9b640 1 -- 172.21.15.98:0/4023755458 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4033b70 con 0x7f69d0003970 2023-11-28T17:43:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.906+0000 7f69e8b05640 1 -- 172.21.15.98:0/4023755458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69d0003970 msgr2=0x7f69d0003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.906+0000 7f69e8b05640 1 --2- 172.21.15.98:0/4023755458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69d0003970 0x7f69d0003d70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f69d400ba00 tx=0x7f69d00057f0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.906+0000 7f69e8b05640 1 -- 172.21.15.98:0/4023755458 shutdown_connections 2023-11-28T17:43:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.906+0000 7f69e8b05640 1 --2- 172.21.15.98:0/4023755458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69d0003970 0x7f69d0003d70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.906+0000 7f69e8b05640 1 -- 172.21.15.98:0/4023755458 >> 172.21.15.98:0/4023755458 conn(0x7f69e406af10 msgr2=0x7f69e406b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.906+0000 7f69e8b05640 1 -- 172.21.15.98:0/4023755458 shutdown_connections 2023-11-28T17:43:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.906+0000 7f69e8b05640 1 -- 172.21.15.98:0/4023755458 wait complete. 2023-11-28T17:43:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.908+0000 7f69e8b05640 1 Processor -- start 2023-11-28T17:43:42.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.908+0000 7f69e8b05640 1 -- start start 2023-11-28T17:43:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.909+0000 7f69e8b05640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69d0003970 0x7f69e4151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.909+0000 7f69e8b05640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69e40c3330 0x7f69e40c37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.909+0000 7f69e8b05640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e4149650 0x7f69e4149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.909+0000 7f69e8b05640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e4151eb0 con 0x7f69e4149650 2023-11-28T17:43:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.909+0000 7f69e8b05640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e40c5a70 con 0x7f69d0003970 2023-11-28T17:43:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.909+0000 7f69e8b05640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e40c57b0 con 0x7f69e40c3330 2023-11-28T17:43:42.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.909+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e4149650 0x7f69e4149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.909+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e4149650 0x7f69e4149ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41912/0 (socket says 172.21.15.98:41912) 2023-11-28T17:43:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.909+0000 7f69e2d9d640 1 -- 172.21.15.98:0/4170485525 learned_addr learned my addr 172.21.15.98:0/4170485525 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.910+0000 7f69e259c640 1 --2- 172.21.15.98:0/4170485525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69d0003970 0x7f69e4151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.910+0000 7f69dbfff640 1 --2- 172.21.15.98:0/4170485525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69e40c3330 0x7f69e40c37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.910+0000 7f69e2d9d640 1 -- 172.21.15.98:0/4170485525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69e40c3330 msgr2=0x7f69e40c37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.910+0000 7f69e2d9d640 1 --2- 172.21.15.98:0/4170485525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69e40c3330 0x7f69e40c37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.910+0000 7f69e2d9d640 1 -- 172.21.15.98:0/4170485525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69d0003970 msgr2=0x7f69e4151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.910+0000 7f69e2d9d640 1 --2- 172.21.15.98:0/4170485525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69d0003970 0x7f69e4151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.910+0000 7f69e2d9d640 1 -- 172.21.15.98:0/4170485525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e415e370 con 0x7f69e4149650 2023-11-28T17:43:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.910+0000 7f69e2d9d640 1 --2- 172.21.15.98:0/4170485525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e4149650 0x7f69e4149ad0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f69e406bcb0 tx=0x7f69dc00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.910+0000 7f69daffd640 1 -- 172.21.15.98:0/4170485525 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69dc004d60 con 0x7f69e4149650 2023-11-28T17:43:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.910+0000 7f69daffd640 1 -- 172.21.15.98:0/4170485525 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69dc00bb80 con 0x7f69e4149650 2023-11-28T17:43:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.910+0000 7f69daffd640 1 -- 172.21.15.98:0/4170485525 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69dc007070 con 0x7f69e4149650 2023-11-28T17:43:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.910+0000 7f69e8b05640 1 -- 172.21.15.98:0/4170485525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e40c3cf0 con 0x7f69e4149650 2023-11-28T17:43:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.910+0000 7f69e8b05640 1 -- 172.21.15.98:0/4170485525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e40cea60 con 0x7f69e4149650 2023-11-28T17:43:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.911+0000 7f69e8b05640 1 -- 172.21.15.98:0/4170485525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e4074f60 con 0x7f69e4149650 2023-11-28T17:43:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.912+0000 7f69daffd640 1 -- 172.21.15.98:0/4170485525 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f69dc0071d0 con 0x7f69e4149650 2023-11-28T17:43:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.913+0000 7f69daffd640 1 --2- 172.21.15.98:0/4170485525 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f69c406b2a0 0x7f69c406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.913+0000 7f69daffd640 1 -- 172.21.15.98:0/4170485525 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f69dc089cf0 con 0x7f69e4149650 2023-11-28T17:43:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.913+0000 7f69e259c640 1 --2- 172.21.15.98:0/4170485525 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f69c406b2a0 0x7f69c406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.914+0000 7f69e259c640 1 --2- 172.21.15.98:0/4170485525 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f69c406b2a0 0x7f69c406d760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f69d00057a0 tx=0x7f69d403e040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:42.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:42.918+0000 7f69daffd640 1 -- 172.21.15.98:0/4170485525 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f69dc054ad0 con 0x7f69e4149650 2023-11-28T17:43:43.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:43.182+0000 7f69e8b05640 1 -- 172.21.15.98:0/4170485525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f69e40c3cf0 con 0x7f69e4149650 2023-11-28T17:43:43.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:43.184+0000 7f69daffd640 1 -- 172.21.15.98:0/4170485525 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f69e40c3cf0 con 0x7f69e4149650 2023-11-28T17:43:43.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:43.187+0000 7f69e8b05640 1 -- 172.21.15.98:0/4170485525 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f69c406b2a0 msgr2=0x7f69c406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:43.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:43.187+0000 7f69e8b05640 1 --2- 172.21.15.98:0/4170485525 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f69c406b2a0 0x7f69c406d760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f69d00057a0 tx=0x7f69d403e040 comp rx=0 tx=0).stop 2023-11-28T17:43:43.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:43.187+0000 7f69e8b05640 1 -- 172.21.15.98:0/4170485525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e4149650 msgr2=0x7f69e4149ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:43.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:43.187+0000 7f69e8b05640 1 --2- 172.21.15.98:0/4170485525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e4149650 0x7f69e4149ad0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f69e406bcb0 tx=0x7f69dc00b4a0 comp rx=0 tx=0).stop 2023-11-28T17:43:43.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:43.187+0000 7f69e8b05640 1 -- 172.21.15.98:0/4170485525 shutdown_connections 2023-11-28T17:43:43.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:43.187+0000 7f69e8b05640 1 --2- 172.21.15.98:0/4170485525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69e4149650 0x7f69e4149ad0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:43.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:43.187+0000 7f69e8b05640 1 --2- 172.21.15.98:0/4170485525 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f69e40c3330 0x7f69e40c37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:43.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:43.187+0000 7f69e8b05640 1 --2- 172.21.15.98:0/4170485525 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f69c406b2a0 0x7f69c406d760 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:43.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:43.187+0000 7f69e8b05640 1 --2- 172.21.15.98:0/4170485525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f69d0003970 0x7f69e4151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:43.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:43.187+0000 7f69e8b05640 1 -- 172.21.15.98:0/4170485525 >> 172.21.15.98:0/4170485525 conn(0x7f69e406af10 msgr2=0x7f69e4150370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:43.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:43.187+0000 7f69e8b05640 1 -- 172.21.15.98:0/4170485525 shutdown_connections 2023-11-28T17:43:43.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:43.187+0000 7f69e8b05640 1 -- 172.21.15.98:0/4170485525 wait complete. 2023-11-28T17:43:43.207 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2023-11-28T17:43:43.414 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/stat.h... yes 2023-11-28T17:43:43.676 INFO:tasks.workunit.client.0.smithi098.stdout:checking for stdlib.h... yes 2023-11-28T17:43:43.930 INFO:tasks.workunit.client.0.smithi098.stdout:checking for string.h... yes 2023-11-28T17:43:44.984 INFO:tasks.workunit.client.0.smithi098.stdout:checking for memory.h... yes 2023-11-28T17:43:45.918 INFO:tasks.workunit.client.0.smithi098.stdout:checking for strings.h... yes 2023-11-28T17:43:46.495 INFO:tasks.workunit.client.0.smithi098.stdout:checking for inttypes.h... yes 2023-11-28T17:43:46.764 INFO:tasks.workunit.client.0.smithi098.stdout:checking for stdint.h... yes 2023-11-28T17:43:47.842 INFO:tasks.workunit.client.0.smithi098.stdout:checking for unistd.h... yes 2023-11-28T17:43:48.136 INFO:tasks.workunit.client.0.smithi098.stdout:checking pthread.h usability... yes 2023-11-28T17:43:48.294 INFO:tasks.workunit.client.0.smithi098.stdout:checking pthread.h presence... yes 2023-11-28T17:43:48.297 INFO:tasks.workunit.client.0.smithi098.stdout:checking for pthread.h... yes 2023-11-28T17:43:49.533 INFO:tasks.workunit.client.0.smithi098.stdout:checking fcntl.h usability... yes 2023-11-28T17:43:49.689 INFO:tasks.workunit.client.0.smithi098.stdout:checking fcntl.h presence... yes 2023-11-28T17:43:49.694 INFO:tasks.workunit.client.0.smithi098.stdout:checking for fcntl.h... yes 2023-11-28T17:43:50.234 INFO:tasks.workunit.client.0.smithi098.stdout:checking limits.h usability... yes 2023-11-28T17:43:50.379 INFO:tasks.workunit.client.0.smithi098.stdout:checking limits.h presence... yes 2023-11-28T17:43:50.384 INFO:tasks.workunit.client.0.smithi098.stdout:checking for limits.h... yes 2023-11-28T17:43:50.431 INFO:tasks.workunit.client.0.smithi098.stdout:checking for stdint.h... (cached) yes 2023-11-28T17:43:50.710 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/time.h usability... yes 2023-11-28T17:43:50.866 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/time.h presence... yes 2023-11-28T17:43:50.869 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/time.h... yes 2023-11-28T17:43:51.288 INFO:tasks.workunit.client.0.smithi098.stdout:checking for unistd.h... (cached) yes 2023-11-28T17:43:51.552 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/vfs.h usability... yes 2023-11-28T17:43:51.699 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/vfs.h presence... yes 2023-11-28T17:43:51.704 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/vfs.h... yes 2023-11-28T17:43:51.951 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/limits.h usability... no 2023-11-28T17:43:52.112 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/limits.h presence... no 2023-11-28T17:43:52.117 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/limits.h... no 2023-11-28T17:43:52.335 INFO:tasks.workunit.client.0.smithi098.stdout:checking for an ANSI C-conforming const... yes 2023-11-28T17:43:52.566 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether time.h and sys/time.h may both be included... yes 2023-11-28T17:43:52.586 INFO:tasks.workunit.client.0.smithi098.stdout:checking for function prototypes... yes 2023-11-28T17:43:52.787 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether setvbuf arguments are reversed... no 2023-11-28T17:43:53.145 INFO:tasks.workunit.client.0.smithi098.stdout:checking for system... yes 2023-11-28T17:43:53.208 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:43:53.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:43:53.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.409+0000 7f610c6e0640 1 Processor -- start 2023-11-28T17:43:53.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.409+0000 7f610c6e0640 1 -- start start 2023-11-28T17:43:53.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.410+0000 7f610c6e0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6104150050 0x7f61040c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:53.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.410+0000 7f610c6e0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61040c9540 con 0x7f61040c8590 2023-11-28T17:43:53.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.410+0000 7f610c6e0640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61040c6450 con 0x7f61040c77b0 2023-11-28T17:43:53.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.410+0000 7f610c6e0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f610406b3d0 con 0x7f6104150050 2023-11-28T17:43:53.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.411+0000 7f610a47c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6104150050 0x7f61040c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:53.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.411+0000 7f610a47c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6104150050 0x7f61040c9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43256/0 (socket says 172.21.15.98:43256) 2023-11-28T17:43:53.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.411+0000 7f610a47c640 1 -- 172.21.15.98:0/3781864452 learned_addr learned my addr 172.21.15.98:0/3781864452 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:53.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.411+0000 7f6109c7b640 1 --1- 172.21.15.98:0/3781864452 >> v1:172.21.15.98:6789/0 conn(0x7f61040c8590 0x7f6104156b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40554/0 (socket says 172.21.15.98:40554) 2023-11-28T17:43:53.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.411+0000 7f610a47c640 1 -- 172.21.15.98:0/3781864452 >> v1:172.21.15.113:6789/0 conn(0x7f61040c77b0 legacy=0x7f610406ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:53.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.411+0000 7f610a47c640 1 -- 172.21.15.98:0/3781864452 >> v1:172.21.15.98:6789/0 conn(0x7f61040c8590 legacy=0x7f6104156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:53.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.411+0000 7f610a47c640 1 -- 172.21.15.98:0/3781864452 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61040c6730 con 0x7f6104150050 2023-11-28T17:43:53.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.411+0000 7f610947a640 1 -- 172.21.15.98:0/3781864452 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446182254 0 0) 0x7f61040c9540 con 0x7f61040c8590 2023-11-28T17:43:53.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.412+0000 7f610a47c640 1 --2- 172.21.15.98:0/3781864452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6104150050 0x7f61040c9000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f610000bb00 tx=0x7f610002dfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba351cd6811ee5b2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:53.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.413+0000 7f610947a640 1 -- 172.21.15.98:0/3781864452 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610002ed70 con 0x7f6104150050 2023-11-28T17:43:53.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.413+0000 7f610947a640 1 -- 172.21.15.98:0/3781864452 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6100032070 con 0x7f6104150050 2023-11-28T17:43:53.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.413+0000 7f610947a640 1 -- 172.21.15.98:0/3781864452 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6100038620 con 0x7f6104150050 2023-11-28T17:43:53.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.413+0000 7f610c6e0640 1 -- 172.21.15.98:0/3781864452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6104150050 msgr2=0x7f61040c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:53.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.413+0000 7f610c6e0640 1 --2- 172.21.15.98:0/3781864452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6104150050 0x7f61040c9000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f610000bb00 tx=0x7f610002dfc0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.413+0000 7f610c6e0640 1 -- 172.21.15.98:0/3781864452 shutdown_connections 2023-11-28T17:43:53.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.413+0000 7f610c6e0640 1 --2- 172.21.15.98:0/3781864452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6104150050 0x7f61040c9000 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.413+0000 7f610c6e0640 1 -- 172.21.15.98:0/3781864452 >> 172.21.15.98:0/3781864452 conn(0x7f610406a4d0 msgr2=0x7f610406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:53.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.414+0000 7f610c6e0640 1 -- 172.21.15.98:0/3781864452 shutdown_connections 2023-11-28T17:43:53.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.414+0000 7f610c6e0640 1 -- 172.21.15.98:0/3781864452 wait complete. 2023-11-28T17:43:53.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.415+0000 7f610c6e0640 1 Processor -- start 2023-11-28T17:43:53.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.416+0000 7f610c6e0640 1 -- start start 2023-11-28T17:43:53.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.416+0000 7f610c6e0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61040c77b0 0x7f610414ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:53.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.416+0000 7f610c6e0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61040c8590 0x7f6104152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:53.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.417+0000 7f610c6e0640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6104150050 0x7f6104152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:53.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.417+0000 7f610c6e0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61041e0760 con 0x7f61040c8590 2023-11-28T17:43:53.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.417+0000 7f610c6e0640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61041e1690 con 0x7f61040c77b0 2023-11-28T17:43:53.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.417+0000 7f610c6e0640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6104153830 con 0x7f6104150050 2023-11-28T17:43:53.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.417+0000 7f6109c7b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61040c8590 0x7f6104152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:53.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.417+0000 7f6109c7b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61040c8590 0x7f6104152340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43270/0 (socket says 172.21.15.98:43270) 2023-11-28T17:43:53.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.417+0000 7f6109c7b640 1 -- 172.21.15.98:0/2518844265 learned_addr learned my addr 172.21.15.98:0/2518844265 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:53.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.417+0000 7f610a47c640 1 --2- 172.21.15.98:0/2518844265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61040c77b0 0x7f610414ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:53.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.417+0000 7f6109c7b640 1 -- 172.21.15.98:0/2518844265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6104150050 msgr2=0x7f6104152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:53.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.417+0000 7f6109c7b640 1 --2- 172.21.15.98:0/2518844265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6104150050 0x7f6104152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.417+0000 7f6109c7b640 1 -- 172.21.15.98:0/2518844265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61040c77b0 msgr2=0x7f610414ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:53.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.417+0000 7f6109c7b640 1 --2- 172.21.15.98:0/2518844265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61040c77b0 0x7f610414ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.417+0000 7f6109c7b640 1 -- 172.21.15.98:0/2518844265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610000b760 con 0x7f61040c8590 2023-11-28T17:43:53.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.418+0000 7f6109c7b640 1 --2- 172.21.15.98:0/2518844265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61040c8590 0x7f6104152340 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f60f400c500 tx=0x7f60f400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:53.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.418+0000 7f60faffd640 1 -- 172.21.15.98:0/2518844265 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f4007560 con 0x7f61040c8590 2023-11-28T17:43:53.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.418+0000 7f60faffd640 1 -- 172.21.15.98:0/2518844265 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f40047e0 con 0x7f61040c8590 2023-11-28T17:43:53.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.418+0000 7f610c6e0640 1 -- 172.21.15.98:0/2518844265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61041e1c90 con 0x7f61040c8590 2023-11-28T17:43:53.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.418+0000 7f60faffd640 1 -- 172.21.15.98:0/2518844265 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f4004b90 con 0x7f61040c8590 2023-11-28T17:43:53.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.418+0000 7f610c6e0640 1 -- 172.21.15.98:0/2518844265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f610415a4c0 con 0x7f61040c8590 2023-11-28T17:43:53.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.419+0000 7f610c6e0640 1 -- 172.21.15.98:0/2518844265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61041e0a60 con 0x7f61040c8590 2023-11-28T17:43:53.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.421+0000 7f60faffd640 1 -- 172.21.15.98:0/2518844265 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f60f4004cf0 con 0x7f61040c8590 2023-11-28T17:43:53.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.421+0000 7f60faffd640 1 --2- 172.21.15.98:0/2518844265 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f60e806b2a0 0x7f60e806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:53.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.422+0000 7f60faffd640 1 -- 172.21.15.98:0/2518844265 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f60f4088fd0 con 0x7f61040c8590 2023-11-28T17:43:53.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.422+0000 7f610a47c640 1 --2- 172.21.15.98:0/2518844265 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f60e806b2a0 0x7f60e806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:53.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.422+0000 7f610a47c640 1 --2- 172.21.15.98:0/2518844265 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f60e806b2a0 0x7f60e806d760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6100039000 tx=0x7f6100034000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:53.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.427+0000 7f60faffd640 1 -- 172.21.15.98:0/2518844265 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f60f4053db0 con 0x7f61040c8590 2023-11-28T17:43:53.460 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gettimeofday... yes 2023-11-28T17:43:53.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.698+0000 7f610c6e0640 1 -- 172.21.15.98:0/2518844265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6104067bf0 con 0x7f61040c8590 2023-11-28T17:43:53.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.699+0000 7f60faffd640 1 -- 172.21.15.98:0/2518844265 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f60f4018020 con 0x7f61040c8590 2023-11-28T17:43:53.701 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:43:53.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.702+0000 7f610c6e0640 1 -- 172.21.15.98:0/2518844265 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f60e806b2a0 msgr2=0x7f60e806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:53.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.702+0000 7f610c6e0640 1 --2- 172.21.15.98:0/2518844265 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f60e806b2a0 0x7f60e806d760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6100039000 tx=0x7f6100034000 comp rx=0 tx=0).stop 2023-11-28T17:43:53.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.702+0000 7f610c6e0640 1 -- 172.21.15.98:0/2518844265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61040c8590 msgr2=0x7f6104152340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:53.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.702+0000 7f610c6e0640 1 --2- 172.21.15.98:0/2518844265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61040c8590 0x7f6104152340 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f60f400c500 tx=0x7f60f400c9c0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.702+0000 7f610c6e0640 1 -- 172.21.15.98:0/2518844265 shutdown_connections 2023-11-28T17:43:53.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.702+0000 7f610c6e0640 1 --2- 172.21.15.98:0/2518844265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6104150050 0x7f6104152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.702+0000 7f610c6e0640 1 --2- 172.21.15.98:0/2518844265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61040c8590 0x7f6104152340 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.702+0000 7f610c6e0640 1 --2- 172.21.15.98:0/2518844265 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f60e806b2a0 0x7f60e806d760 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.702+0000 7f610c6e0640 1 --2- 172.21.15.98:0/2518844265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61040c77b0 0x7f610414ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.702+0000 7f610c6e0640 1 -- 172.21.15.98:0/2518844265 >> 172.21.15.98:0/2518844265 conn(0x7f610406a4d0 msgr2=0x7f610414cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:53.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.702+0000 7f610c6e0640 1 -- 172.21.15.98:0/2518844265 shutdown_connections 2023-11-28T17:43:53.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.702+0000 7f610c6e0640 1 -- 172.21.15.98:0/2518844265 wait complete. 2023-11-28T17:43:53.722 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:43:53.722 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-11-28T17:43:53.722 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:43:53.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:43:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.929+0000 7f8938c2e640 1 Processor -- start 2023-11-28T17:43:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.929+0000 7f8938c2e640 1 -- start start 2023-11-28T17:43:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.931+0000 7f8938c2e640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f893406a4d0 0x7f893415dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.931+0000 7f8938c2e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89340b03b0 con 0x7f8934070a90 2023-11-28T17:43:53.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.931+0000 7f8938c2e640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893406d1e0 con 0x7f893406b160 2023-11-28T17:43:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.931+0000 7f8938c2e640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f893406c2f0 con 0x7f893406a4d0 2023-11-28T17:43:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.931+0000 7f8933fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8934070a90 0x7f8934164590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40570/0 (socket says 172.21.15.98:40570) 2023-11-28T17:43:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.931+0000 7f8933fff640 1 -- 172.21.15.98:0/1603634073 learned_addr learned my addr 172.21.15.98:0/1603634073 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:43:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f89337fe640 1 --2- 172.21.15.98:0/1603634073 >> v2:172.21.15.113:3300/0 conn(0x7f893406a4d0 0x7f893415dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8932ffd640 1 -- 172.21.15.98:0/1603634073 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251791636 0 0) 0x7f89340b03b0 con 0x7f8934070a90 2023-11-28T17:43:53.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8932ffd640 1 -- 172.21.15.98:0/1603634073 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8920003520 con 0x7f8934070a90 2023-11-28T17:43:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8932ffd640 1 -- 172.21.15.98:0/1603634073 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 984348663 0 0) 0x7f8920003520 con 0x7f8934070a90 2023-11-28T17:43:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8932ffd640 1 -- 172.21.15.98:0/1603634073 >> v2:172.21.15.113:3300/0 conn(0x7f893406a4d0 msgr2=0x7f893415dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8932ffd640 1 --2- 172.21.15.98:0/1603634073 >> v2:172.21.15.113:3300/0 conn(0x7f893406a4d0 0x7f893415dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8932ffd640 1 -- 172.21.15.98:0/1603634073 >> v1:172.21.15.113:6789/0 conn(0x7f893406b160 legacy=0x7f8934070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8932ffd640 1 -- 172.21.15.98:0/1603634073 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893406c5d0 con 0x7f8934070a90 2023-11-28T17:43:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8932ffd640 1 -- 172.21.15.98:0/1603634073 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f892c0019f0 con 0x7f8934070a90 2023-11-28T17:43:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8932ffd640 1 -- 172.21.15.98:0/1603634073 >> v1:172.21.15.98:6789/0 conn(0x7f8934070a90 legacy=0x7f8934164590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:53.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8932ffd640 1 --2- 172.21.15.98:0/1603634073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8920003970 0x7f8920003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:53.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8932ffd640 1 --2- 172.21.15.98:0/1603634073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89200047c0 0x7f8920004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:53.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8932ffd640 1 --2- 172.21.15.98:0/1603634073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8920005180 0x7f892000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:53.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8932ffd640 1 -- 172.21.15.98:0/1603634073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893406c2f0 con 0x7f8920005180 2023-11-28T17:43:53.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8932ffd640 1 -- 172.21.15.98:0/1603634073 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892000c000 con 0x7f89200047c0 2023-11-28T17:43:53.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8932ffd640 1 -- 172.21.15.98:0/1603634073 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892000c1c0 con 0x7f8920003970 2023-11-28T17:43:53.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.932+0000 7f8933fff640 1 --2- 172.21.15.98:0/1603634073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8920005180 0x7f892000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:53.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.933+0000 7f8933fff640 1 -- 172.21.15.98:0/1603634073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8920003970 msgr2=0x7f8920003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:53.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.933+0000 7f8933fff640 1 --2- 172.21.15.98:0/1603634073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8920003970 0x7f8920003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.933+0000 7f8933fff640 1 -- 172.21.15.98:0/1603634073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89200047c0 msgr2=0x7f8920004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:53.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.933+0000 7f8933fff640 1 --2- 172.21.15.98:0/1603634073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f89200047c0 0x7f8920004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.933+0000 7f8933fff640 1 -- 172.21.15.98:0/1603634073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8920003520 con 0x7f8920005180 2023-11-28T17:43:53.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.934+0000 7f8933fff640 1 -- 172.21.15.98:0/1603634073 reap_dead start 2023-11-28T17:43:53.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.934+0000 7f8933fff640 1 --2- 172.21.15.98:0/1603634073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8920005180 0x7f892000b980 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f892c00a810 tx=0x7f89200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748f17ab6a7b59e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.934+0000 7f8932ffd640 1 -- 172.21.15.98:0/1603634073 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c034630 con 0x7f8920005180 2023-11-28T17:43:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.934+0000 7f8932ffd640 1 -- 172.21.15.98:0/1603634073 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892c02f070 con 0x7f8920005180 2023-11-28T17:43:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.934+0000 7f8932ffd640 1 -- 172.21.15.98:0/1603634073 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c0349d0 con 0x7f8920005180 2023-11-28T17:43:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.935+0000 7f8938c2e640 1 -- 172.21.15.98:0/1603634073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8920005180 msgr2=0x7f892000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.935+0000 7f8938c2e640 1 --2- 172.21.15.98:0/1603634073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8920005180 0x7f892000b980 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f892c00a810 tx=0x7f89200057f0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.935+0000 7f8938c2e640 1 -- 172.21.15.98:0/1603634073 shutdown_connections 2023-11-28T17:43:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.935+0000 7f8938c2e640 1 --2- 172.21.15.98:0/1603634073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8920005180 0x7f892000b980 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.935+0000 7f8938c2e640 1 -- 172.21.15.98:0/1603634073 >> 172.21.15.98:0/1603634073 conn(0x7f89340cafe0 msgr2=0x7f89340cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.935+0000 7f8938c2e640 1 -- 172.21.15.98:0/1603634073 shutdown_connections 2023-11-28T17:43:53.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.935+0000 7f8938c2e640 1 -- 172.21.15.98:0/1603634073 wait complete. 2023-11-28T17:43:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.937+0000 7f8938c2e640 1 Processor -- start 2023-11-28T17:43:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.937+0000 7f8938c2e640 1 -- start start 2023-11-28T17:43:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.938+0000 7f8938c2e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8920005180 0x7f89340ae4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.938+0000 7f8938c2e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f893415a960 0x7f89341efd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.938+0000 7f8938c2e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8934159030 0x7f89341f02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.938+0000 7f8938c2e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89340c8500 con 0x7f8920005180 2023-11-28T17:43:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.938+0000 7f8938c2e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89341ef8d0 con 0x7f8934159030 2023-11-28T17:43:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.938+0000 7f8938c2e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89341f3b40 con 0x7f893415a960 2023-11-28T17:43:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.938+0000 7f89337fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8920005180 0x7f89340ae4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.938+0000 7f89337fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8920005180 0x7f89340ae4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43284/0 (socket says 172.21.15.98:43284) 2023-11-28T17:43:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.938+0000 7f89337fe640 1 -- 172.21.15.98:0/1838343106 learned_addr learned my addr 172.21.15.98:0/1838343106 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.938+0000 7f89337fe640 1 -- 172.21.15.98:0/1838343106 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f893415a960 msgr2=0x7f89341efd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.938+0000 7f89337fe640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f893415a960 0x7f89341efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.939+0000 7f89337fe640 1 -- 172.21.15.98:0/1838343106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8934159030 msgr2=0x7f89341f02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.939+0000 7f89337fe640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8934159030 0x7f89341f02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.939+0000 7f89337fe640 1 -- 172.21.15.98:0/1838343106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89341f2fc0 con 0x7f8920005180 2023-11-28T17:43:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.939+0000 7f89337fe640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8920005180 0x7f89340ae4e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f892400e7a0 tx=0x7f892400ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:53.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.939+0000 7f8930ff9640 1 -- 172.21.15.98:0/1838343106 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892400eeb0 con 0x7f8920005180 2023-11-28T17:43:53.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.939+0000 7f8930ff9640 1 -- 172.21.15.98:0/1838343106 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892400c940 con 0x7f8920005180 2023-11-28T17:43:53.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.939+0000 7f8930ff9640 1 -- 172.21.15.98:0/1838343106 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892400cd10 con 0x7f8920005180 2023-11-28T17:43:53.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.939+0000 7f8938c2e640 1 -- 172.21.15.98:0/1838343106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89341f32a0 con 0x7f8920005180 2023-11-28T17:43:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.939+0000 7f8938c2e640 1 -- 172.21.15.98:0/1838343106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89341f3860 con 0x7f8920005180 2023-11-28T17:43:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.940+0000 7f8938c2e640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893414ffc0 0x7f89341503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.940+0000 7f8938c2e640 1 -- 172.21.15.98:0/1838343106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8934074310 con 0x7f893414ffc0 2023-11-28T17:43:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.941+0000 7f892bfff640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893414ffc0 0x7f89341503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.941+0000 7f892bfff640 1 -- 172.21.15.98:0/1838343106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f893406ce70 con 0x7f893414ffc0 2023-11-28T17:43:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.941+0000 7f8930ff9640 1 -- 172.21.15.98:0/1838343106 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f8924013470 con 0x7f8920005180 2023-11-28T17:43:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.941+0000 7f892bfff640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893414ffc0 0x7f89341503e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8934158fc0 tx=0x7f891c00c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.942+0000 7f8930ff9640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f891406b2d0 0x7f891406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.942+0000 7f8930ff9640 1 -- 172.21.15.98:0/1838343106 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8924089e90 con 0x7f8920005180 2023-11-28T17:43:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.942+0000 7f8933fff640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f891406b2d0 0x7f891406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.943+0000 7f8930ff9640 1 -- 172.21.15.98:0/1838343106 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c011790 con 0x7f893414ffc0 2023-11-28T17:43:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.943+0000 7f8930ff9640 1 -- 172.21.15.98:0/1838343106 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f891c0118f0 con 0x7f893414ffc0 2023-11-28T17:43:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.943+0000 7f8930ff9640 1 -- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893414ffc0 msgr2=0x7f89341503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.943+0000 7f8930ff9640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893414ffc0 0x7f89341503e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8934158fc0 tx=0x7f891c00c6e0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.943+0000 7f8930ff9640 1 -- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893414ffc0 msgr2=0x7f89341503e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.943+0000 7f8930ff9640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f893414ffc0 0x7f89341503e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.943+0000 7f8933fff640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f891406b2d0 0x7f891406d790 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f892c031000 tx=0x7f891c0036f0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:53.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.950+0000 7f8938c2e640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8934150920 0x7f8934157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:53.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.950+0000 7f8938c2e640 1 -- 172.21.15.98:0/1838343106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89341efa40 con 0x7f8934150920 2023-11-28T17:43:53.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.950+0000 7f892bfff640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8934150920 0x7f8934157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:53.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.950+0000 7f892bfff640 1 -- 172.21.15.98:0/1838343106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f893406cb90 con 0x7f8934150920 2023-11-28T17:43:53.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.951+0000 7f892bfff640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8934150920 0x7f8934157b00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8934158fc0 tx=0x7f891c0037d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:53.952 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.951+0000 7f8930ff9640 1 -- 172.21.15.98:0/1838343106 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c01c270 con 0x7f8934150920 2023-11-28T17:43:53.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.951+0000 7f8930ff9640 1 -- 172.21.15.98:0/1838343106 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4808 (secure 0 0 0) 0x7f891c012420 con 0x7f8934150920 2023-11-28T17:43:53.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.951+0000 7f8930ff9640 1 -- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8934150920 msgr2=0x7f8934157b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:53.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.951+0000 7f8930ff9640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8934150920 0x7f8934157b00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8934158fc0 tx=0x7f891c0037d0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.951+0000 7f8930ff9640 1 -- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8934150920 msgr2=0x7f8934157b00 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:53.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.951+0000 7f8930ff9640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8934150920 0x7f8934157b00 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8934158fc0 tx=0x7f891c0037d0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.954 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:43:53.954 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:43:53.954 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:43:53.954 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:43:53.954 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2023-11-28T17:43:53.954 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:43:53.954 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:43:53.954 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:43:53.954 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:43:53.954 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:53.955 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 16, 2023-11-28T17:43:53.955 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:53.955 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:43:53.955 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:43:53.955 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:53.955 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:53.955 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:53.955 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:53.955 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:53.955 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:53.955 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:53.955 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:53.955 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:53.956 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:53.956 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:53.956 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:43:53.956 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:43:53.956 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:53.956 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:53.956 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:53.956 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:53.956 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:53.956 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:53.956 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:53.956 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:53.956 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:53.957 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:53.957 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:53.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:53.957 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:43:53.957 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:43:53.957 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:43:53.957 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:43:53.957 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:43:53.957 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:43:53.957 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:53.957 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:53.957 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:43:53.958 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:43:53.958 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:43:53.958 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:43:53.958 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:43:53.958 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:43:53.958 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:43:53.958 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:53.958 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:43:53.958 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:53.958 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:53.958 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:53.958 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:53.958 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:43:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:43:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:43:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:43:53.959 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:53.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:53.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:53.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:53.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:43:53.960 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:53.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:53.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:53.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:53.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:43:53.961 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:53.961 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:53.961 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:53.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:53.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:43:53.961 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:43:53.961 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:53.961 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:53.961 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:53.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:53.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:53.962 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:43:53.962 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:43:53.962 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:53.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:53.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:53.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:53.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:43:53.962 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:53.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:53.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:53.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:53.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:43:53.963 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:53.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:53.963 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:53.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:53.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:43:53.963 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:43:53.963 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:53.963 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:53.963 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:53.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:53.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:53.963 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:43:53.964 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:43:53.964 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:53.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:53.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:53.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:53.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:43:53.964 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:53.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:53.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:53.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:53.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:43:53.965 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:53.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:53.965 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:53.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:53.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:43:53.965 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:43:53.965 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:53.965 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:53.965 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:53.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:53.965 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:53.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:53.966 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:43:53.966 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:43:53.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:53.966 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:53.966 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:53.966 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:43:53.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:53.966 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:53.966 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:43:53.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:53.966 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:53.966 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:53.967 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:43:53.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:53.967 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:53.967 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:43:53.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:53.967 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:53.967 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:53.967 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:43:53.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:53.967 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:53.967 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:43:53.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:53.968 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:53.968 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:53.968 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:43:53.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:53.968 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:53.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:53.969 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:43:53.969 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:43:53.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.952+0000 7f8938c2e640 1 -- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f891406b2d0 msgr2=0x7f891406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:53.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.952+0000 7f8938c2e640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f891406b2d0 0x7f891406d790 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f892c031000 tx=0x7f891c0036f0 comp rx=0 tx=0).stop 2023-11-28T17:43:53.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.952+0000 7f8938c2e640 1 -- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8920005180 msgr2=0x7f89340ae4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:53.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.952+0000 7f8938c2e640 1 --2- 172.21.15.98:0/1838343106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8920005180 0x7f89340ae4e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f892400e7a0 tx=0x7f892400ec60 comp rx=0 tx=0).stop 2023-11-28T17:43:53.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.952+0000 7f8933fff640 1 -- 172.21.15.98:0/1838343106 reap_dead start 2023-11-28T17:43:53.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.952+0000 7f8933fff640 1 -- 172.21.15.98:0/1838343106 reap_dead start 2023-11-28T17:43:53.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.952+0000 7f8938c2e640 1 -- 172.21.15.98:0/1838343106 shutdown_connections 2023-11-28T17:43:53.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.952+0000 7f8938c2e640 1 -- 172.21.15.98:0/1838343106 >> 172.21.15.98:0/1838343106 conn(0x7f89340cafe0 msgr2=0x7f89340cb450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:53.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.952+0000 7f8938c2e640 1 -- 172.21.15.98:0/1838343106 shutdown_connections 2023-11-28T17:43:53.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:53.952+0000 7f8938c2e640 1 -- 172.21.15.98:0/1838343106 wait complete. 2023-11-28T17:43:53.973 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:43:53.973 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:43:53.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:43:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.179+0000 7fd6d1e31640 1 Processor -- start 2023-11-28T17:43:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.179+0000 7fd6d1e31640 1 -- start start 2023-11-28T17:43:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.181+0000 7fd6d1e31640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd6cc153fb0 0x7fd6cc0c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.181+0000 7fd6d1e31640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd6cc0c76f0 0x7fd6cc0c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.181+0000 7fd6d1e31640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc15db80 con 0x7fd6cc0b5270 2023-11-28T17:43:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.181+0000 7fd6d1e31640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6cc0cb730 con 0x7fd6cc153fb0 2023-11-28T17:43:54.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.181+0000 7fd6d1e31640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6cc15e0c0 con 0x7fd6cc0c76f0 2023-11-28T17:43:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.181+0000 7fd6cb7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd6cc153fb0 0x7fd6cc0c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.181+0000 7fd6cb7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd6cc153fb0 0x7fd6cc0c6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43322/0 (socket says 172.21.15.98:43322) 2023-11-28T17:43:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.181+0000 7fd6cb7fe640 1 -- 172.21.15.98:0/737613188 learned_addr learned my addr 172.21.15.98:0/737613188 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:54.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.181+0000 7fd6caffd640 1 --1- 172.21.15.98:0/737613188 >> v1:172.21.15.98:6789/0 conn(0x7fd6cc0b5270 0x7fd6cc0ca300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40580/0 (socket says 172.21.15.98:40580) 2023-11-28T17:43:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.182+0000 7fd6cbfff640 1 --2- 172.21.15.98:0/737613188 >> v2:172.21.15.113:3300/0 conn(0x7fd6cc0c76f0 0x7fd6cc0c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.182+0000 7fd6cb7fe640 1 -- 172.21.15.98:0/737613188 >> v2:172.21.15.113:3300/0 conn(0x7fd6cc0c76f0 msgr2=0x7fd6cc0c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.182+0000 7fd6cb7fe640 1 --2- 172.21.15.98:0/737613188 >> v2:172.21.15.113:3300/0 conn(0x7fd6cc0c76f0 0x7fd6cc0c7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.182+0000 7fd6cb7fe640 1 -- 172.21.15.98:0/737613188 >> v1:172.21.15.98:6789/0 conn(0x7fd6cc0b5270 legacy=0x7fd6cc0ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.182+0000 7fd6cb7fe640 1 -- 172.21.15.98:0/737613188 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc0d00a0 con 0x7fd6cc153fb0 2023-11-28T17:43:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.182+0000 7fd6ca7fc640 1 -- 172.21.15.98:0/737613188 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038993390 0 0) 0x7fd6cc15db80 con 0x7fd6cc0b5270 2023-11-28T17:43:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.183+0000 7fd6cb7fe640 1 --2- 172.21.15.98:0/737613188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6cc153fb0 0x7fd6cc0c6ad0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc06c730 tx=0x7fd6bc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb242b9a1af4997 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.183+0000 7fd6ca7fc640 1 -- 172.21.15.98:0/737613188 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc02ca80 con 0x7fd6cc153fb0 2023-11-28T17:43:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.183+0000 7fd6ca7fc640 1 -- 172.21.15.98:0/737613188 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc02cbe0 con 0x7fd6cc153fb0 2023-11-28T17:43:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.183+0000 7fd6ca7fc640 1 -- 172.21.15.98:0/737613188 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc0343f0 con 0x7fd6cc153fb0 2023-11-28T17:43:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.184+0000 7fd6d1e31640 1 -- 172.21.15.98:0/737613188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6cc153fb0 msgr2=0x7fd6cc0c6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.184+0000 7fd6d1e31640 1 --2- 172.21.15.98:0/737613188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6cc153fb0 0x7fd6cc0c6ad0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc06c730 tx=0x7fd6bc02bee0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.184+0000 7fd6d1e31640 1 -- 172.21.15.98:0/737613188 shutdown_connections 2023-11-28T17:43:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.184+0000 7fd6d1e31640 1 --2- 172.21.15.98:0/737613188 >> v2:172.21.15.113:3300/0 conn(0x7fd6cc0c76f0 0x7fd6cc0c7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.184+0000 7fd6d1e31640 1 --2- 172.21.15.98:0/737613188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6cc153fb0 0x7fd6cc0c6ad0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.184+0000 7fd6d1e31640 1 -- 172.21.15.98:0/737613188 >> 172.21.15.98:0/737613188 conn(0x7fd6cc0cf4a0 msgr2=0x7fd6cc0cf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.184+0000 7fd6d1e31640 1 -- 172.21.15.98:0/737613188 shutdown_connections 2023-11-28T17:43:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.184+0000 7fd6d1e31640 1 -- 172.21.15.98:0/737613188 wait complete. 2023-11-28T17:43:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.186+0000 7fd6d1e31640 1 Processor -- start 2023-11-28T17:43:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.186+0000 7fd6d1e31640 1 -- start start 2023-11-28T17:43:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.187+0000 7fd6d1e31640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc0c76f0 0x7fd6cc1efd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.187+0000 7fd6d1e31640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6cc153fb0 0x7fd6cc14f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.187+0000 7fd6d1e31640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6cc14cdd0 0x7fd6cc14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.187+0000 7fd6d1e31640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc1f02a0 con 0x7fd6cc153fb0 2023-11-28T17:43:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.187+0000 7fd6d1e31640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc15c170 con 0x7fd6cc0c76f0 2023-11-28T17:43:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.187+0000 7fd6d1e31640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc15c450 con 0x7fd6cc14cdd0 2023-11-28T17:43:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.187+0000 7fd6caffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6cc153fb0 0x7fd6cc14f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.187+0000 7fd6caffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6cc153fb0 0x7fd6cc14f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43332/0 (socket says 172.21.15.98:43332) 2023-11-28T17:43:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.187+0000 7fd6caffd640 1 -- 172.21.15.98:0/134937076 learned_addr learned my addr 172.21.15.98:0/134937076 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.188+0000 7fd6caffd640 1 -- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6cc14cdd0 msgr2=0x7fd6cc14a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.188+0000 7fd6cbfff640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6cc14cdd0 0x7fd6cc14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.188+0000 7fd6caffd640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd6cc14cdd0 0x7fd6cc14a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.188+0000 7fd6cb7fe640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc0c76f0 0x7fd6cc1efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.188+0000 7fd6caffd640 1 -- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc0c76f0 msgr2=0x7fd6cc1efd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.188+0000 7fd6caffd640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc0c76f0 0x7fd6cc1efd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.188+0000 7fd6caffd640 1 -- 172.21.15.98:0/134937076 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc009b30 con 0x7fd6cc153fb0 2023-11-28T17:43:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.188+0000 7fd6caffd640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6cc153fb0 0x7fd6cc14f200 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400c410 tx=0x7fd6b400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.188+0000 7fd6c3fff640 1 -- 172.21.15.98:0/134937076 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b40074e0 con 0x7fd6cc153fb0 2023-11-28T17:43:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.188+0000 7fd6c3fff640 1 -- 172.21.15.98:0/134937076 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b4007640 con 0x7fd6cc153fb0 2023-11-28T17:43:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.188+0000 7fd6d1e31640 1 -- 172.21.15.98:0/134937076 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6cc15c730 con 0x7fd6cc153fb0 2023-11-28T17:43:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.188+0000 7fd6c3fff640 1 -- 172.21.15.98:0/134937076 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4004a20 con 0x7fd6cc153fb0 2023-11-28T17:43:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.188+0000 7fd6d1e31640 1 -- 172.21.15.98:0/134937076 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6cc15ccf0 con 0x7fd6cc153fb0 2023-11-28T17:43:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.190+0000 7fd6d1e31640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc0b5270 0x7fd6cc14ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.190+0000 7fd6d1e31640 1 -- 172.21.15.98:0/134937076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc074110 con 0x7fd6cc0b5270 2023-11-28T17:43:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.190+0000 7fd6cb7fe640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc0b5270 0x7fd6cc14ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.191+0000 7fd6c3fff640 1 -- 172.21.15.98:0/134937076 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fd6b4004b80 con 0x7fd6cc153fb0 2023-11-28T17:43:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.191+0000 7fd6cb7fe640 1 -- 172.21.15.98:0/134937076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6cc14d690 con 0x7fd6cc0b5270 2023-11-28T17:43:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.191+0000 7fd6cb7fe640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc0b5270 0x7fd6cc14ac10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0c2fa0 tx=0x7fd6bc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.191+0000 7fd6c3fff640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd6ac06b2d0 0x7fd6ac06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.192+0000 7fd6c3fff640 1 -- 172.21.15.98:0/134937076 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd6b40897b0 con 0x7fd6cc153fb0 2023-11-28T17:43:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.192+0000 7fd6cbfff640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd6ac06b2d0 0x7fd6ac06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.192+0000 7fd6c3fff640 1 -- 172.21.15.98:0/134937076 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc009510 con 0x7fd6cc0b5270 2023-11-28T17:43:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.192+0000 7fd6cbfff640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd6ac06b2d0 0x7fd6ac06d790 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4006fd0 tx=0x7fd6c4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.192+0000 7fd6c3fff640 1 -- 172.21.15.98:0/134937076 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fd6bc040470 con 0x7fd6cc0b5270 2023-11-28T17:43:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.192+0000 7fd6c3fff640 1 -- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc0b5270 msgr2=0x7fd6cc14ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.192+0000 7fd6c3fff640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc0b5270 0x7fd6cc14ac10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0c2fa0 tx=0x7fd6bc004a40 comp rx=0 tx=0).stop 2023-11-28T17:43:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.192+0000 7fd6c3fff640 1 -- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc0b5270 msgr2=0x7fd6cc14ac10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.192+0000 7fd6c3fff640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc0b5270 0x7fd6cc14ac10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.199+0000 7fd6d1e31640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc15ce80 0x7fd6cc1f2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.199+0000 7fd6d1e31640 1 -- 172.21.15.98:0/134937076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc14a4d0 con 0x7fd6cc15ce80 2023-11-28T17:43:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.200+0000 7fd6cb7fe640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc15ce80 0x7fd6cc1f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.201+0000 7fd6cb7fe640 1 -- 172.21.15.98:0/134937076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd6cc0c8060 con 0x7fd6cc15ce80 2023-11-28T17:43:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.201+0000 7fd6cb7fe640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc15ce80 0x7fd6cc1f2850 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc02bf20 tx=0x7fd6bc009000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.201+0000 7fd6c3fff640 1 -- 172.21.15.98:0/134937076 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc054630 con 0x7fd6cc15ce80 2023-11-28T17:43:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.202+0000 7fd6c3fff640 1 -- 172.21.15.98:0/134937076 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4466 (secure 0 0 0) 0x7fd6bc054790 con 0x7fd6cc15ce80 2023-11-28T17:43:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.202+0000 7fd6c3fff640 1 -- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc15ce80 msgr2=0x7fd6cc1f2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.202+0000 7fd6c3fff640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc15ce80 0x7fd6cc1f2850 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc02bf20 tx=0x7fd6bc009000 comp rx=0 tx=0).stop 2023-11-28T17:43:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.202+0000 7fd6c3fff640 1 -- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc15ce80 msgr2=0x7fd6cc1f2850 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.202+0000 7fd6c3fff640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6cc15ce80 0x7fd6cc1f2850 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc02bf20 tx=0x7fd6bc009000 comp rx=0 tx=0).stop 2023-11-28T17:43:54.204 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:43:54.205 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:43:54.205 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:43:54.205 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:43:54.205 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2023-11-28T17:43:54.205 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:43:54.205 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:43:54.205 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:43:54.205 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:43:54.205 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:54.206 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 16, 2023-11-28T17:43:54.206 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:54.206 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:43:54.206 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:43:54.206 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:54.206 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:54.206 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:54.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:54.206 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:54.206 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:54.207 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:54.207 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:54.207 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:54.207 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:54.207 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:54.207 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:43:54.207 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:43:54.207 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:54.207 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:54.207 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:54.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:54.208 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:54.208 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:54.208 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:54.208 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:54.208 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:54.208 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:54.208 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:54.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.208 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:43:54.208 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:43:54.209 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:43:54.209 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:43:54.209 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:43:54.209 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:43:54.209 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:54.209 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:54.209 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:43:54.209 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:43:54.209 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:43:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:43:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:43:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:43:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:43:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:43:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:43:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:43:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:43:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:43:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:43:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:43:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:43:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:43:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:43:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:43:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:43:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:43:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:43:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:43:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:43:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:43:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:43:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:43:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:43:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:43:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:43:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:43:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:43:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:43:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:43:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:43:54.219 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:43:54.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.203+0000 7fd6d1e31640 1 -- 172.21.15.98:0/134937076 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd6ac06b2d0 msgr2=0x7fd6ac06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.203+0000 7fd6d1e31640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd6ac06b2d0 0x7fd6ac06d790 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4006fd0 tx=0x7fd6c4008040 comp rx=0 tx=0).stop 2023-11-28T17:43:54.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.203+0000 7fd6d1e31640 1 -- 172.21.15.98:0/134937076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6cc153fb0 msgr2=0x7fd6cc14f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.203+0000 7fd6d1e31640 1 --2- 172.21.15.98:0/134937076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6cc153fb0 0x7fd6cc14f200 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400c410 tx=0x7fd6b400c8d0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.203+0000 7fd6cbfff640 1 -- 172.21.15.98:0/134937076 reap_dead start 2023-11-28T17:43:54.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.203+0000 7fd6cbfff640 1 -- 172.21.15.98:0/134937076 reap_dead start 2023-11-28T17:43:54.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.203+0000 7fd6d1e31640 1 -- 172.21.15.98:0/134937076 shutdown_connections 2023-11-28T17:43:54.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.203+0000 7fd6d1e31640 1 -- 172.21.15.98:0/134937076 >> 172.21.15.98:0/134937076 conn(0x7fd6cc0cf4a0 msgr2=0x7fd6cc0c8270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:54.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.203+0000 7fd6d1e31640 1 -- 172.21.15.98:0/134937076 shutdown_connections 2023-11-28T17:43:54.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.203+0000 7fd6d1e31640 1 -- 172.21.15.98:0/134937076 wait complete. 2023-11-28T17:43:54.224 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:43:54.224 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:43:54.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:43:54.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.441+0000 7f99722e0640 1 Processor -- start 2023-11-28T17:43:54.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.441+0000 7f99722e0640 1 -- start start 2023-11-28T17:43:54.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.443+0000 7f99722e0640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f996c0c5a60 0x7f996c0c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.443+0000 7f99722e0640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c1584d0 0x7f996c06dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.443+0000 7f99722e0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f996c0c8150 con 0x7f996c151be0 2023-11-28T17:43:54.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.443+0000 7f99722e0640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f996c0c8850 con 0x7f996c0c5a60 2023-11-28T17:43:54.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.443+0000 7f99722e0640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0c5f50 con 0x7f996c1584d0 2023-11-28T17:43:54.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.443+0000 7f996bfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f996c151be0 0x7f996c0c4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40594/0 (socket says 172.21.15.98:40594) 2023-11-28T17:43:54.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.443+0000 7f996bfff640 1 -- 172.21.15.98:0/1414628693 learned_addr learned my addr 172.21.15.98:0/1414628693 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:43:54.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f997087d640 1 --2- 172.21.15.98:0/1414628693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c1584d0 0x7f996c06dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f9963fff640 1 --2- 172.21.15.98:0/1414628693 >> v2:172.21.15.113:3300/0 conn(0x7f996c0c5a60 0x7f996c0c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 -- 172.21.15.98:0/1414628693 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605815257 0 0) 0x7f996c0c8150 con 0x7f996c151be0 2023-11-28T17:43:54.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 -- 172.21.15.98:0/1414628693 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9958003520 con 0x7f996c151be0 2023-11-28T17:43:54.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 -- 172.21.15.98:0/1414628693 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 136911302 0 0) 0x7f9958003520 con 0x7f996c151be0 2023-11-28T17:43:54.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 -- 172.21.15.98:0/1414628693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c1584d0 msgr2=0x7f996c06dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:54.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 --2- 172.21.15.98:0/1414628693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c1584d0 0x7f996c06dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 -- 172.21.15.98:0/1414628693 >> v2:172.21.15.113:3300/0 conn(0x7f996c0c5a60 msgr2=0x7f996c0c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:54.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 --2- 172.21.15.98:0/1414628693 >> v2:172.21.15.113:3300/0 conn(0x7f996c0c5a60 0x7f996c0c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 -- 172.21.15.98:0/1414628693 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c0c7b90 con 0x7f996c151be0 2023-11-28T17:43:54.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 -- 172.21.15.98:0/1414628693 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f995c001670 con 0x7f996c151be0 2023-11-28T17:43:54.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 -- 172.21.15.98:0/1414628693 >> v1:172.21.15.98:6789/0 conn(0x7f996c151be0 legacy=0x7f996c0c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 --2- 172.21.15.98:0/1414628693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9958003970 0x7f9958003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 --2- 172.21.15.98:0/1414628693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c1584d0 0x7f99580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 --2- 172.21.15.98:0/1414628693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9958004d00 0x7f9958005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 -- 172.21.15.98:0/1414628693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0c8850 con 0x7f9958004d00 2023-11-28T17:43:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 -- 172.21.15.98:0/1414628693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0c5f50 con 0x7f9958003970 2023-11-28T17:43:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.444+0000 7f996b7fe640 1 -- 172.21.15.98:0/1414628693 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f995800b980 con 0x7f996c1584d0 2023-11-28T17:43:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.445+0000 7f997087d640 1 --2- 172.21.15.98:0/1414628693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9958004d00 0x7f9958005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.445+0000 7f9963fff640 1 --2- 172.21.15.98:0/1414628693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c1584d0 0x7f99580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.445+0000 7f996bfff640 1 --2- 172.21.15.98:0/1414628693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9958003970 0x7f9958003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.445+0000 7f997087d640 1 -- 172.21.15.98:0/1414628693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c1584d0 msgr2=0x7f99580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.445+0000 7f997087d640 1 --2- 172.21.15.98:0/1414628693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c1584d0 0x7f99580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.445+0000 7f997087d640 1 -- 172.21.15.98:0/1414628693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9958003970 msgr2=0x7f9958003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:43:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.445+0000 7f997087d640 1 --2- 172.21.15.98:0/1414628693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9958003970 0x7f9958003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.445+0000 7f997087d640 1 -- 172.21.15.98:0/1414628693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9958003520 con 0x7f9958004d00 2023-11-28T17:43:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.446+0000 7f997087d640 1 --2- 172.21.15.98:0/1414628693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9958004d00 0x7f9958005180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9964009fc0 tx=0x7f99580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ca5384ab3cae9e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.446+0000 7f996b7fe640 1 -- 172.21.15.98:0/1414628693 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996402cc20 con 0x7f9958004d00 2023-11-28T17:43:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.446+0000 7f996b7fe640 1 -- 172.21.15.98:0/1414628693 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996400f040 con 0x7f9958004d00 2023-11-28T17:43:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.446+0000 7f996b7fe640 1 -- 172.21.15.98:0/1414628693 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964034440 con 0x7f9958004d00 2023-11-28T17:43:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.447+0000 7f99722e0640 1 -- 172.21.15.98:0/1414628693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9958004d00 msgr2=0x7f9958005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.447+0000 7f99722e0640 1 --2- 172.21.15.98:0/1414628693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9958004d00 0x7f9958005180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9964009fc0 tx=0x7f99580057f0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.447+0000 7f997087d640 1 -- 172.21.15.98:0/1414628693 reap_dead start 2023-11-28T17:43:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.447+0000 7f99722e0640 1 -- 172.21.15.98:0/1414628693 shutdown_connections 2023-11-28T17:43:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.447+0000 7f99722e0640 1 -- 172.21.15.98:0/1414628693 >> 172.21.15.98:0/1414628693 conn(0x7f996c06c0f0 msgr2=0x7f996c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.447+0000 7f99722e0640 1 -- 172.21.15.98:0/1414628693 shutdown_connections 2023-11-28T17:43:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.447+0000 7f99722e0640 1 -- 172.21.15.98:0/1414628693 wait complete. 2023-11-28T17:43:54.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.449+0000 7f99722e0640 1 Processor -- start 2023-11-28T17:43:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.449+0000 7f99722e0640 1 -- start start 2023-11-28T17:43:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.450+0000 7f99722e0640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9958004d00 0x7f996c153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.450+0000 7f99722e0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f996c152390 0x7f996c14f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.450+0000 7f99722e0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f996c0beb10 0x7f996c0be1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.450+0000 7f99722e0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0bddc0 con 0x7f996c152390 2023-11-28T17:43:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.450+0000 7f99722e0640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c153750 con 0x7f996c0beb10 2023-11-28T17:43:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.450+0000 7f99722e0640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0bd4b0 con 0x7f9958004d00 2023-11-28T17:43:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.450+0000 7f9963fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f996c152390 0x7f996c14f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.450+0000 7f9963fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f996c152390 0x7f996c14f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43362/0 (socket says 172.21.15.98:43362) 2023-11-28T17:43:54.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.450+0000 7f9963fff640 1 -- 172.21.15.98:0/3827093392 learned_addr learned my addr 172.21.15.98:0/3827093392 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:43:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.451+0000 7f996bfff640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9958004d00 0x7f996c153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.451+0000 7f9963fff640 1 -- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9958004d00 msgr2=0x7f996c153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.451+0000 7f9963fff640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9958004d00 0x7f996c153210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.451+0000 7f9963fff640 1 -- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f996c0beb10 msgr2=0x7f996c0be1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.451+0000 7f9963fff640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f996c0beb10 0x7f996c0be1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.451+0000 7f9963fff640 1 -- 172.21.15.98:0/3827093392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c1f3040 con 0x7f996c152390 2023-11-28T17:43:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.451+0000 7f9963fff640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f996c152390 0x7f996c14f9f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9954011700 tx=0x7f9954011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.451+0000 7f99697fa640 1 -- 172.21.15.98:0/3827093392 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995400f940 con 0x7f996c152390 2023-11-28T17:43:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.451+0000 7f99697fa640 1 -- 172.21.15.98:0/3827093392 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995400faa0 con 0x7f996c152390 2023-11-28T17:43:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.451+0000 7f99722e0640 1 -- 172.21.15.98:0/3827093392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f996c1f3320 con 0x7f996c152390 2023-11-28T17:43:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.451+0000 7f99697fa640 1 -- 172.21.15.98:0/3827093392 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995400fe10 con 0x7f996c152390 2023-11-28T17:43:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.451+0000 7f99722e0640 1 -- 172.21.15.98:0/3827093392 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996c0bd9e0 con 0x7f996c152390 2023-11-28T17:43:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.452+0000 7f99722e0640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c14e920 0x7f996c14ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.452+0000 7f99722e0640 1 -- 172.21.15.98:0/3827093392 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0744d0 con 0x7f996c14e920 2023-11-28T17:43:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.453+0000 7f996bfff640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c14e920 0x7f996c14ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.454+0000 7f99697fa640 1 -- 172.21.15.98:0/3827093392 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f9954015460 con 0x7f996c152390 2023-11-28T17:43:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.454+0000 7f996bfff640 1 -- 172.21.15.98:0/3827093392 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f996c158960 con 0x7f996c14e920 2023-11-28T17:43:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.454+0000 7f996bfff640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c14e920 0x7f996c14ed00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f995c00f6f0 tx=0x7f995c00fbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.454+0000 7f99697fa640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f994c06b2a0 0x7f994c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.455+0000 7f99697fa640 1 -- 172.21.15.98:0/3827093392 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f995408a0a0 con 0x7f996c152390 2023-11-28T17:43:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.455+0000 7f997087d640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f994c06b2a0 0x7f994c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.455+0000 7f99697fa640 1 -- 172.21.15.98:0/3827093392 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c00fee0 con 0x7f996c14e920 2023-11-28T17:43:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.455+0000 7f99697fa640 1 -- 172.21.15.98:0/3827093392 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f995c00d770 con 0x7f996c14e920 2023-11-28T17:43:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.455+0000 7f99697fa640 1 -- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c14e920 msgr2=0x7f996c14ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.455+0000 7f99697fa640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c14e920 0x7f996c14ed00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f995c00f6f0 tx=0x7f995c00fbb0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.455+0000 7f99697fa640 1 -- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c14e920 msgr2=0x7f996c14ed00 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.455+0000 7f99697fa640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c14e920 0x7f996c14ed00 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f995c00f6f0 tx=0x7f995c00fbb0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.456+0000 7f997087d640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f994c06b2a0 0x7f994c06d760 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9964036000 tx=0x7f996402e070 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:54.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.462+0000 7f99722e0640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c1eefe0 0x7f996c1ef3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:43:54.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.462+0000 7f99722e0640 1 -- 172.21.15.98:0/3827093392 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996c0c84e0 con 0x7f996c1eefe0 2023-11-28T17:43:54.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.463+0000 7f996bfff640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c1eefe0 0x7f996c1ef3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:43:54.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.464+0000 7f996bfff640 1 -- 172.21.15.98:0/3827093392 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f996c0bdf00 con 0x7f996c1eefe0 2023-11-28T17:43:54.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.465+0000 7f996bfff640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c1eefe0 0x7f996c1ef3f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f995c00f6f0 tx=0x7f995c001930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:43:54.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.465+0000 7f99697fa640 1 -- 172.21.15.98:0/3827093392 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c014840 con 0x7f996c1eefe0 2023-11-28T17:43:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.466+0000 7f99697fa640 1 -- 172.21.15.98:0/3827093392 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4976 (secure 0 0 0) 0x7f995c0149a0 con 0x7f996c1eefe0 2023-11-28T17:43:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.466+0000 7f99697fa640 1 -- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c1eefe0 msgr2=0x7f996c1ef3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.466+0000 7f99697fa640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c1eefe0 0x7f996c1ef3f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f995c00f6f0 tx=0x7f995c001930 comp rx=0 tx=0).stop 2023-11-28T17:43:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.466+0000 7f99697fa640 1 -- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c1eefe0 msgr2=0x7f996c1ef3f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:43:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.466+0000 7f99697fa640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f996c1eefe0 0x7f996c1ef3f0 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f995c00f6f0 tx=0x7f995c001930 comp rx=0 tx=0).stop 2023-11-28T17:43:54.468 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:43:54.469 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:43:54.469 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:43:54.469 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:43:54.469 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2023-11-28T17:43:54.469 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:43:54.469 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:43:54.469 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:43:54.469 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:43:54.470 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:54.470 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 16, 2023-11-28T17:43:54.470 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:54.470 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:43:54.470 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:43:54.470 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:54.470 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:54.470 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:54.471 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:54.471 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:54.471 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:54.471 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:54.471 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:54.471 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:54.471 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:54.471 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:54.471 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:43:54.471 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:43:54.472 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:54.472 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:54.472 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:54.472 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:54.472 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:54.472 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:54.472 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:54.472 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:54.472 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:54.472 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:54.472 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:54.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.473 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:43:54.473 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:43:54.473 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:43:54.473 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:43:54.473 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:43:54.473 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:43:54.473 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:54.473 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:43:54.473 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:43:54.473 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:43:54.473 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:43:54.474 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:43:54.474 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:43:54.474 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:43:54.474 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:43:54.474 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:43:54.474 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:43:54.474 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:43:54.474 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:43:54.474 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:43:54.474 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:43:54.474 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:43:54.475 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:43:54.475 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:43:54.475 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:43:54.475 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:43:54.475 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:43:54.475 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:54.475 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:43:54.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.475 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:43:54.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.475 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:43:54.476 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:43:54.476 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:54.476 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:54.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:54.476 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:43:54.476 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:54.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:54.476 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:43:54.477 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:54.477 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:54.477 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:54.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.477 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:43:54.477 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:43:54.477 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:54.477 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:54.477 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:54.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.478 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:43:54.478 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:43:54.478 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:54.478 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:54.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:54.478 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:43:54.478 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:54.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:54.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:43:54.479 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:54.479 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:54.480 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:54.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.480 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:43:54.480 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:43:54.480 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:54.481 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:54.481 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:54.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.481 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:43:54.481 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:43:54.481 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:43:54.481 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:43:54.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:43:54.482 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:43:54.482 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:54.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:43:54.482 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:43:54.482 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:43:54.482 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:54.483 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:54.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.483 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:43:54.483 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:43:54.483 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:43:54.483 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:43:54.483 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:43:54.483 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:54.483 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:54.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.484 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:43:54.484 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:43:54.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.484 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:54.484 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:54.484 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:43:54.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:54.484 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:54.484 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:43:54.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.485 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:54.485 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:54.485 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:43:54.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:54.485 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:54.485 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:43:54.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.485 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:54.485 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:54.486 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 6 2023-11-28T17:43:54.486 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:54.486 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:54.486 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:43:54.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.486 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:54.486 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:54.486 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:43:54.487 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:54.487 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:43:54.487 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:43:54.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:43:54.487 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:43:54.487 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:43:54.487 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:43:54.488 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:43:54.488 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:43:54.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:43:54.488 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:43:54.488 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:43:54.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.467+0000 7f99722e0640 1 -- 172.21.15.98:0/3827093392 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f994c06b2a0 msgr2=0x7f994c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.467+0000 7f99722e0640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f994c06b2a0 0x7f994c06d760 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9964036000 tx=0x7f996402e070 comp rx=0 tx=0).stop 2023-11-28T17:43:54.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.467+0000 7f99722e0640 1 -- 172.21.15.98:0/3827093392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f996c152390 msgr2=0x7f996c14f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:43:54.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.467+0000 7f99722e0640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f996c152390 0x7f996c14f9f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9954011700 tx=0x7f9954011bc0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.467+0000 7f997087d640 1 -- 172.21.15.98:0/3827093392 reap_dead start 2023-11-28T17:43:54.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.467+0000 7f99722e0640 1 -- 172.21.15.98:0/3827093392 shutdown_connections 2023-11-28T17:43:54.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.467+0000 7f99722e0640 1 --2- 172.21.15.98:0/3827093392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f996c152390 0x7f996c14f9f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:43:54.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.467+0000 7f99722e0640 1 -- 172.21.15.98:0/3827093392 >> 172.21.15.98:0/3827093392 conn(0x7f996c06c0f0 msgr2=0x7f996c0c25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:43:54.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.467+0000 7f99722e0640 1 -- 172.21.15.98:0/3827093392 shutdown_connections 2023-11-28T17:43:54.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:43:54.467+0000 7f99722e0640 1 -- 172.21.15.98:0/3827093392 wait complete. 2023-11-28T17:43:54.491 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:43:54.491 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-11-28T17:43:54.491 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2023-11-28T17:43:54.492 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-11-28T17:43:54.492 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-11-28T17:43:54.492 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-28T17:43:54.492 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:43:54.518 INFO:tasks.ceph.mon.a:Stopped 2023-11-28T17:43:54.518 INFO:tasks.mon_thrash:Sending STOP to mon b 2023-11-28T17:43:54.518 INFO:tasks.ceph.mon.b:Sent signal 19 2023-11-28T17:43:54.519 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-11-28T17:43:55.428 INFO:tasks.workunit.client.0.smithi098.stdout:checking for mkdir... yes 2023-11-28T17:43:55.776 INFO:tasks.workunit.client.0.smithi098.stdout:checking for strerror... yes 2023-11-28T17:43:56.152 INFO:tasks.workunit.client.0.smithi098.stdout:checking for open64... yes 2023-11-28T17:43:58.170 INFO:tasks.workunit.client.0.smithi098.stdout:checking for stat64... yes 2023-11-28T17:43:58.528 INFO:tasks.workunit.client.0.smithi098.stdout:checking for fseeko64... yes 2023-11-28T17:43:59.289 INFO:tasks.workunit.client.0.smithi098.stdout:checking for lrand48_r... yes 2023-11-28T17:44:00.390 INFO:tasks.workunit.client.0.smithi098.stdout:checking for srand48_r... yes 2023-11-28T17:44:00.476 INFO:tasks.workunit.client.0.smithi098.stdout:configure: creating ./config.status 2023-11-28T17:44:02.069 INFO:tasks.workunit.client.0.smithi098.stdout:config.status: creating Makefile 2023-11-28T17:44:02.161 INFO:tasks.workunit.client.0.smithi098.stdout:config.status: creating config.h 2023-11-28T17:44:02.309 INFO:tasks.workunit.client.0.smithi098.stdout:config.status: executing depfiles commands 2023-11-28T17:44:03.276 INFO:tasks.workunit.client.0.smithi098.stdout:config.status: executing default commands 2023-11-28T17:44:08.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:44:08.362 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:44:09.520 INFO:tasks.mon_thrash:Sending CONT to mon b 2023-11-28T17:44:09.520 INFO:tasks.ceph.mon.b:Sent signal 18 2023-11-28T17:44:09.520 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-11-28T17:44:09.521 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:44:09.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.734+0000 7f4f3f701640 1 Processor -- start 2023-11-28T17:44:09.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.735+0000 7f4f3f701640 1 -- start start 2023-11-28T17:44:09.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.736+0000 7f4f3f701640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4f380cafb0 0x7f4f3815b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:09.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.736+0000 7f4f3f701640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 0x7f4f3806e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:09.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.736+0000 7f4f3f701640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f380cb8d0 con 0x7f4f38151c40 2023-11-28T17:44:09.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.736+0000 7f4f3f701640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f380c7510 con 0x7f4f380cafb0 2023-11-28T17:44:09.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.736+0000 7f4f3f701640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3815be30 con 0x7f4f3806dcf0 2023-11-28T17:44:09.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.736+0000 7f4f3d49d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4f38151c40 legacy=0x7f4f380c4730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:44:09.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.736+0000 7f4f3cc9c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4f380cafb0 msgr2=0x7f4f3815b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:44:09.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.736+0000 7f4f3cc9c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4f380cafb0 0x7f4f3815b8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:44:09.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.737+0000 7f4f3dc9e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 0x7f4f3806e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:09.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.737+0000 7f4f3dc9e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 0x7f4f3806e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:46010/0 (socket says 172.21.15.98:46010) 2023-11-28T17:44:09.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.737+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/3380751234 learned_addr learned my addr 172.21.15.98:0/3380751234 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:44:09.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.737+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 msgr2=0x7f4f3806e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:44:09.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.737+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 msgr2=0x7f4f3806e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:44:09.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.737+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 0x7f4f3806e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:44:09.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.737+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 0x7f4f3806e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:44:09.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.937+0000 7f4f3d49d640 1 -- 172.21.15.98:0/3380751234 >> v1:172.21.15.98:6789/0 conn(0x7f4f38151c40 legacy=0x7f4f380c4730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:44:09.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.937+0000 7f4f3cc9c640 1 -- 172.21.15.98:0/3380751234 >> v2:172.21.15.98:3300/0 conn(0x7f4f380cafb0 msgr2=0x7f4f3815b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:44:09.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.937+0000 7f4f3cc9c640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.98:3300/0 conn(0x7f4f380cafb0 0x7f4f3815b8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:44:09.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.938+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 0x7f4f3806e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:09.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.939+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 msgr2=0x7f4f3806e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:44:09.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.939+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 msgr2=0x7f4f3806e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:44:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.939+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 0x7f4f3806e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:44:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:09.939+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 0x7f4f3806e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:44:10.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:10.338+0000 7f4f3cc9c640 1 -- 172.21.15.98:0/3380751234 >> v2:172.21.15.98:3300/0 conn(0x7f4f380cafb0 msgr2=0x7f4f3815b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:44:10.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:10.338+0000 7f4f3d49d640 1 -- 172.21.15.98:0/3380751234 >> v1:172.21.15.98:6789/0 conn(0x7f4f38151c40 legacy=0x7f4f380c4730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:44:10.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:10.338+0000 7f4f3cc9c640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.98:3300/0 conn(0x7f4f380cafb0 0x7f4f3815b8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:44:10.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:10.340+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 0x7f4f3806e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:10.341+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 msgr2=0x7f4f3806e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:44:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:10.341+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 msgr2=0x7f4f3806e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:44:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:10.341+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 0x7f4f3806e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:44:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:10.341+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 0x7f4f3806e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:44:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:11.140+0000 7f4f3cc9c640 1 -- 172.21.15.98:0/3380751234 >> v2:172.21.15.98:3300/0 conn(0x7f4f380cafb0 msgr2=0x7f4f3815b8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:44:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:11.140+0000 7f4f3d49d640 1 -- 172.21.15.98:0/3380751234 >> v1:172.21.15.98:6789/0 conn(0x7f4f38151c40 legacy=0x7f4f380c4730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:44:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:11.140+0000 7f4f3cc9c640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.98:3300/0 conn(0x7f4f380cafb0 0x7f4f3815b8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:44:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:11.143+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 0x7f4f3806e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:11.143+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 msgr2=0x7f4f3806e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:44:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:11.143+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 msgr2=0x7f4f3806e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:44:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:11.143+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 0x7f4f3806e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:44:11.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:11.144+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 0x7f4f3806e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:44:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.736+0000 7f4f2effd640 1 -- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 msgr2=0x7f4f3806e180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:44:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.736+0000 7f4f2effd640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f3806dcf0 0x7f4f3806e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.736+0000 7f4f2effd640 1 -- 172.21.15.98:0/3380751234 >> v2:172.21.15.98:3300/0 conn(0x7f4f380cafb0 msgr2=0x7f4f3815b8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:44:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.736+0000 7f4f2effd640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.98:3300/0 conn(0x7f4f380cafb0 0x7f4f3815b8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.736+0000 7f4f2effd640 1 -- 172.21.15.98:0/3380751234 >> v1:172.21.15.98:6789/0 conn(0x7f4f38151c40 legacy=0x7f4f380c4730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:44:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.737+0000 7f4f2effd640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 0x7f4f1c005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.737+0000 7f4f2effd640 1 -- 172.21.15.98:0/3380751234 --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3806e6c0 con 0x7f4f1c006170 2023-11-28T17:44:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.737+0000 7f4f2effd640 1 -- 172.21.15.98:0/3380751234 --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f380c7510 con 0x7f4f1c0036a0 2023-11-28T17:44:12.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.738+0000 7f4f3d49d640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 0x7f4f1c005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:12.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.738+0000 7f4f3d49d640 1 -- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 msgr2=0x7f4f1c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:44:12.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.738+0000 7f4f3d49d640 1 -- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 msgr2=0x7f4f1c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:44:12.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.738+0000 7f4f3d49d640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 0x7f4f1c005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:44:12.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.738+0000 7f4f3d49d640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 0x7f4f1c005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:44:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.939+0000 7f4f3d49d640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 0x7f4f1c005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.940+0000 7f4f3d49d640 1 -- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 msgr2=0x7f4f1c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:44:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.940+0000 7f4f3d49d640 1 -- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 msgr2=0x7f4f1c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:44:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.940+0000 7f4f3d49d640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 0x7f4f1c005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:44:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:12.940+0000 7f4f3d49d640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 0x7f4f1c005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:44:13.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:13.341+0000 7f4f3d49d640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 0x7f4f1c005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:13.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:13.342+0000 7f4f3d49d640 1 -- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 msgr2=0x7f4f1c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:44:13.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:13.342+0000 7f4f3d49d640 1 -- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 msgr2=0x7f4f1c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:44:13.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:13.342+0000 7f4f3d49d640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 0x7f4f1c005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:44:13.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:13.342+0000 7f4f3d49d640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 0x7f4f1c005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:44:14.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.144+0000 7f4f3d49d640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 0x7f4f1c005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:14.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.144+0000 7f4f3d49d640 1 -- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 msgr2=0x7f4f1c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:44:14.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.144+0000 7f4f3d49d640 1 -- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 msgr2=0x7f4f1c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:44:14.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.144+0000 7f4f3d49d640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 0x7f4f1c005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:44:14.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.144+0000 7f4f3d49d640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 0x7f4f1c005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:44:14.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.554+0000 7f4f2ffff640 1 -- 172.21.15.98:0/3380751234 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096435334 0 0) 0x7f4f3806e6c0 con 0x7f4f1c006170 2023-11-28T17:44:14.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.554+0000 7f4f2ffff640 1 -- 172.21.15.98:0/3380751234 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f24003520 con 0x7f4f1c006170 2023-11-28T17:44:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.567+0000 7f4f2ffff640 1 -- 172.21.15.98:0/3380751234 <== mon.1 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4265493671 0 0) 0x7f4f24003520 con 0x7f4f1c006170 2023-11-28T17:44:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.567+0000 7f4f2ffff640 1 -- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 msgr2=0x7f4f1c005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:44:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.567+0000 7f4f2ffff640 1 --2- 172.21.15.98:0/3380751234 >> v2:172.21.15.113:3300/0 conn(0x7f4f1c0036a0 0x7f4f1c005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.567+0000 7f4f2ffff640 1 -- 172.21.15.98:0/3380751234 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f380cb8d0 con 0x7f4f1c006170 2023-11-28T17:44:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.567+0000 7f4f2ffff640 1 -- 172.21.15.98:0/3380751234 <== mon.1 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f4f280041f0 con 0x7f4f1c006170 2023-11-28T17:44:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.567+0000 7f4f2ffff640 1 -- 172.21.15.98:0/3380751234 >> v1:172.21.15.113:6789/0 conn(0x7f4f1c006170 legacy=0x7f4f1c008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.567+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/3380751234 reap_dead start 2023-11-28T17:44:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.567+0000 7f4f2ffff640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f24003970 0x7f4f24003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.567+0000 7f4f2ffff640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f240047c0 0x7f4f24004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.567+0000 7f4f2ffff640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f240051d0 0x7f4f2400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.567+0000 7f4f2ffff640 1 -- 172.21.15.98:0/3380751234 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f380c7510 con 0x7f4f240047c0 2023-11-28T17:44:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.567+0000 7f4f2ffff640 1 -- 172.21.15.98:0/3380751234 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2400c100 con 0x7f4f24003970 2023-11-28T17:44:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.567+0000 7f4f2ffff640 1 -- 172.21.15.98:0/3380751234 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2400c2c0 con 0x7f4f240051d0 2023-11-28T17:44:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.567+0000 7f4f3cc9c640 1 -- 172.21.15.98:0/3380751234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f240047c0 msgr2=0x7f4f24004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:44:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.567+0000 7f4f3cc9c640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f240047c0 0x7f4f24004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:44:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.568+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f240051d0 0x7f4f2400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.568+0000 7f4f3d49d640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f24003970 0x7f4f24003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.569+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f24003970 msgr2=0x7f4f24003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:44:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.569+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f24003970 0x7f4f24003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.569+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/3380751234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f240047c0 msgr2=0x7f4f24004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:44:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.569+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f240047c0 0x7f4f24004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.569+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/3380751234 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f24003520 con 0x7f4f240051d0 2023-11-28T17:44:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.571+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f240051d0 0x7f4f2400b980 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4f3806f3e0 tx=0x7f4f240057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c232722bf30a80b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:14.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.583+0000 7f4f2ffff640 1 -- 172.21.15.98:0/3380751234 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f30035410 con 0x7f4f240051d0 2023-11-28T17:44:14.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.583+0000 7f4f2ffff640 1 -- 172.21.15.98:0/3380751234 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3002f030 con 0x7f4f240051d0 2023-11-28T17:44:14.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.583+0000 7f4f2ffff640 1 -- 172.21.15.98:0/3380751234 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f300357d0 con 0x7f4f240051d0 2023-11-28T17:44:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.584+0000 7f4f3f701640 1 -- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f240051d0 msgr2=0x7f4f2400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.584+0000 7f4f3f701640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f240051d0 0x7f4f2400b980 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4f3806f3e0 tx=0x7f4f240057f0 comp rx=0 tx=0).stop 2023-11-28T17:44:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.584+0000 7f4f3f701640 1 -- 172.21.15.98:0/3380751234 shutdown_connections 2023-11-28T17:44:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.584+0000 7f4f3f701640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f240051d0 0x7f4f2400b980 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.584+0000 7f4f3f701640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f240047c0 0x7f4f24004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.584+0000 7f4f3f701640 1 --2- 172.21.15.98:0/3380751234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f24003970 0x7f4f24003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.584+0000 7f4f3f701640 1 -- 172.21.15.98:0/3380751234 >> 172.21.15.98:0/3380751234 conn(0x7f4f3806c0f0 msgr2=0x7f4f3806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:14.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.584+0000 7f4f3f701640 1 -- 172.21.15.98:0/3380751234 shutdown_connections 2023-11-28T17:44:14.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.584+0000 7f4f3f701640 1 -- 172.21.15.98:0/3380751234 wait complete. 2023-11-28T17:44:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.589+0000 7f4f3f701640 1 Processor -- start 2023-11-28T17:44:14.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.589+0000 7f4f3f701640 1 -- start start 2023-11-28T17:44:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.591+0000 7f4f3f701640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f24003970 0x7f4f381f40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.591+0000 7f4f3f701640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f240047c0 0x7f4f38158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.591+0000 7f4f3f701640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f240051d0 0x7f4f38149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.591+0000 7f4f3f701640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f380c32c0 con 0x7f4f24003970 2023-11-28T17:44:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.591+0000 7f4f3f701640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f381f45e0 con 0x7f4f240047c0 2023-11-28T17:44:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.591+0000 7f4f3f701640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f38158ff0 con 0x7f4f240051d0 2023-11-28T17:44:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.592+0000 7f4f3d49d640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f24003970 msgr2=0x7f4f381f40a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:44:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.592+0000 7f4f3d49d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f24003970 0x7f4f381f40a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:44:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.592+0000 7f4f3dc9e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f240051d0 0x7f4f38149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.592+0000 7f4f3cc9c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f240047c0 0x7f4f38158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:14.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.592+0000 7f4f3cc9c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f240047c0 0x7f4f38158ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:58756/0 (socket says 172.21.15.98:58756) 2023-11-28T17:44:14.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.592+0000 7f4f3cc9c640 1 -- 172.21.15.98:0/345766788 learned_addr learned my addr 172.21.15.98:0/345766788 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:44:14.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.592+0000 7f4f3dc9e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f240051d0 0x7f4f38149c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:45866/0 (socket says 172.21.15.98:45866) 2023-11-28T17:44:14.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.593+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/345766788 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f240047c0 msgr2=0x7f4f38158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:14.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.593+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/345766788 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f240047c0 0x7f4f38158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:14.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.593+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/345766788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f24003970 msgr2=0x7f4f381f40a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:44:14.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.593+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/345766788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f24003970 0x7f4f381f40a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:14.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.593+0000 7f4f3dc9e640 1 -- 172.21.15.98:0/345766788 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3815ba10 con 0x7f4f240051d0 2023-11-28T17:44:14.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.593+0000 7f4f3cc9c640 1 --2- 172.21.15.98:0/345766788 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f240047c0 0x7f4f38158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:44:14.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.594+0000 7f4f3dc9e640 1 --2- 172.21.15.98:0/345766788 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f240051d0 0x7f4f38149c20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4f30034000 tx=0x7f4f300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:14.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.594+0000 7f4f2dffb640 1 -- 172.21.15.98:0/345766788 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f30035e50 con 0x7f4f240051d0 2023-11-28T17:44:14.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.594+0000 7f4f2dffb640 1 -- 172.21.15.98:0/345766788 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3000c040 con 0x7f4f240051d0 2023-11-28T17:44:14.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.594+0000 7f4f2dffb640 1 -- 172.21.15.98:0/345766788 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3003f6b0 con 0x7f4f240051d0 2023-11-28T17:44:14.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.594+0000 7f4f3f701640 1 -- 172.21.15.98:0/345766788 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f380c3510 con 0x7f4f240051d0 2023-11-28T17:44:14.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.595+0000 7f4f3f701640 1 -- 172.21.15.98:0/345766788 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f380c3ad0 con 0x7f4f240051d0 2023-11-28T17:44:14.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.596+0000 7f4f3f701640 1 -- 172.21.15.98:0/345766788 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f38074ef0 con 0x7f4f240051d0 2023-11-28T17:44:14.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.598+0000 7f4f2dffb640 1 -- 172.21.15.98:0/345766788 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4f3003c040 con 0x7f4f240051d0 2023-11-28T17:44:14.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.599+0000 7f4f2dffb640 1 --2- 172.21.15.98:0/345766788 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f1c06b8e0 0x7f4f1c06dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:14.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.599+0000 7f4f2dffb640 1 -- 172.21.15.98:0/345766788 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4f300ab3e0 con 0x7f4f240051d0 2023-11-28T17:44:14.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.599+0000 7f4f3d49d640 1 --2- 172.21.15.98:0/345766788 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f1c06b8e0 0x7f4f1c06dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:14.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.600+0000 7f4f3d49d640 1 --2- 172.21.15.98:0/345766788 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f1c06b8e0 0x7f4f1c06dda0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4f34003d00 tx=0x7f4f34005040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:14.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.615+0000 7f4f2dffb640 1 -- 172.21.15.98:0/345766788 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f30076270 con 0x7f4f240051d0 2023-11-28T17:44:14.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.883+0000 7f4f3f701640 1 -- 172.21.15.98:0/345766788 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4f380cb6b0 con 0x7f4f240051d0 2023-11-28T17:44:14.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.884+0000 7f4f2dffb640 1 -- 172.21.15.98:0/345766788 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f4f3000f040 con 0x7f4f240051d0 2023-11-28T17:44:14.886 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":28,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:44:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.887+0000 7f4f3f701640 1 -- 172.21.15.98:0/345766788 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f1c06b8e0 msgr2=0x7f4f1c06dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.887+0000 7f4f3f701640 1 --2- 172.21.15.98:0/345766788 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f1c06b8e0 0x7f4f1c06dda0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4f34003d00 tx=0x7f4f34005040 comp rx=0 tx=0).stop 2023-11-28T17:44:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.887+0000 7f4f3f701640 1 -- 172.21.15.98:0/345766788 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f240051d0 msgr2=0x7f4f38149c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.887+0000 7f4f3f701640 1 --2- 172.21.15.98:0/345766788 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f240051d0 0x7f4f38149c20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4f30034000 tx=0x7f4f300046c0 comp rx=0 tx=0).stop 2023-11-28T17:44:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.887+0000 7f4f3f701640 1 -- 172.21.15.98:0/345766788 shutdown_connections 2023-11-28T17:44:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.887+0000 7f4f3f701640 1 --2- 172.21.15.98:0/345766788 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f240051d0 0x7f4f38149c20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.887+0000 7f4f3f701640 1 --2- 172.21.15.98:0/345766788 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f1c06b8e0 0x7f4f1c06dda0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.887+0000 7f4f3f701640 1 --2- 172.21.15.98:0/345766788 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f240047c0 0x7f4f38158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.887+0000 7f4f3f701640 1 --2- 172.21.15.98:0/345766788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f24003970 0x7f4f381f40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:14.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.887+0000 7f4f3f701640 1 -- 172.21.15.98:0/345766788 >> 172.21.15.98:0/345766788 conn(0x7f4f3806c0f0 msgr2=0x7f4f3814ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:14.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.887+0000 7f4f3f701640 1 -- 172.21.15.98:0/345766788 shutdown_connections 2023-11-28T17:44:14.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:14.887+0000 7f4f3f701640 1 -- 172.21.15.98:0/345766788 wait complete. 2023-11-28T17:44:14.907 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-11-28T17:44:14.908 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-11-28T17:44:14.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:44:15.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.110+0000 7fefd730a640 1 Processor -- start 2023-11-28T17:44:15.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.111+0000 7fefd730a640 1 -- start start 2023-11-28T17:44:15.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.112+0000 7fefd730a640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefd006a740 0x7fefd00c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.112+0000 7fefd730a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fefd006ab20 0x7fefd00c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.112+0000 7fefd730a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd00c81a0 con 0x7fefd00c6c30 2023-11-28T17:44:15.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.112+0000 7fefd730a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd006e8f0 con 0x7fefd006ab20 2023-11-28T17:44:15.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.112+0000 7fefd730a640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd006dce0 con 0x7fefd006a740 2023-11-28T17:44:15.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.112+0000 7fefd58a7640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fefd00c6c30 legacy=0x7fefd0153a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:44:15.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.112+0000 7fefd48a5640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7fefd006ab20 msgr2=0x7fefd00c7b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:44:15.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.113+0000 7fefd48a5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fefd006ab20 0x7fefd00c7b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:44:15.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.113+0000 7fefd50a6640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefd006a740 0x7fefd00c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:15.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.113+0000 7fefd50a6640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefd006a740 0x7fefd00c75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:45882/0 (socket says 172.21.15.98:45882) 2023-11-28T17:44:15.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.113+0000 7fefd50a6640 1 -- 172.21.15.98:0/2033553295 learned_addr learned my addr 172.21.15.98:0/2033553295 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:44:15.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.114+0000 7fefd50a6640 1 -- 172.21.15.98:0/2033553295 >> v2:172.21.15.98:3300/0 conn(0x7fefd006ab20 msgr2=0x7fefd00c7b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:44:15.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.114+0000 7fefd50a6640 1 --2- 172.21.15.98:0/2033553295 >> v2:172.21.15.98:3300/0 conn(0x7fefd006ab20 0x7fefd00c7b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:15.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.114+0000 7fefd50a6640 1 -- 172.21.15.98:0/2033553295 >> v1:172.21.15.98:6789/0 conn(0x7fefd00c6c30 legacy=0x7fefd0153a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:44:15.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.114+0000 7fefd50a6640 1 -- 172.21.15.98:0/2033553295 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd006dfc0 con 0x7fefd006a740 2023-11-28T17:44:15.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.115+0000 7fefd50a6640 1 --2- 172.21.15.98:0/2033553295 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefd006a740 0x7fefd00c75f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009e50 tx=0x7fefc002bf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3da171496da41e43 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:15.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.116+0000 7fefcffff640 1 -- 172.21.15.98:0/2033553295 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc002caa0 con 0x7fefd006a740 2023-11-28T17:44:15.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.116+0000 7fefcffff640 1 -- 172.21.15.98:0/2033553295 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc002cc00 con 0x7fefd006a740 2023-11-28T17:44:15.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.116+0000 7fefcffff640 1 -- 172.21.15.98:0/2033553295 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc00343f0 con 0x7fefd006a740 2023-11-28T17:44:15.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.116+0000 7fefd730a640 1 -- 172.21.15.98:0/2033553295 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefd006a740 msgr2=0x7fefd00c75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:15.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.116+0000 7fefd730a640 1 --2- 172.21.15.98:0/2033553295 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefd006a740 0x7fefd00c75f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009e50 tx=0x7fefc002bf00 comp rx=0 tx=0).stop 2023-11-28T17:44:15.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.117+0000 7fefd730a640 1 -- 172.21.15.98:0/2033553295 shutdown_connections 2023-11-28T17:44:15.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.117+0000 7fefd730a640 1 --2- 172.21.15.98:0/2033553295 >> v2:172.21.15.98:3300/0 conn(0x7fefd006ab20 0x7fefd00c7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:15.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.117+0000 7fefd730a640 1 --2- 172.21.15.98:0/2033553295 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefd006a740 0x7fefd00c75f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:15.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.117+0000 7fefd730a640 1 -- 172.21.15.98:0/2033553295 >> 172.21.15.98:0/2033553295 conn(0x7fefd006ef10 msgr2=0x7fefd006f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:15.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.117+0000 7fefd730a640 1 -- 172.21.15.98:0/2033553295 shutdown_connections 2023-11-28T17:44:15.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.117+0000 7fefd730a640 1 -- 172.21.15.98:0/2033553295 wait complete. 2023-11-28T17:44:15.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.118+0000 7fefd730a640 1 Processor -- start 2023-11-28T17:44:15.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.118+0000 7fefd730a640 1 -- start start 2023-11-28T17:44:15.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.119+0000 7fefd730a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd006a740 0x7fefd01467f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.119+0000 7fefd730a640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefd006ab20 0x7fefd0157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.119+0000 7fefd730a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd00c6c30 0x7fefd0157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.119+0000 7fefd730a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd0148f20 con 0x7fefd00c6c30 2023-11-28T17:44:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.119+0000 7fefd730a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd0150180 con 0x7fefd006a740 2023-11-28T17:44:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.119+0000 7fefd730a640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd01492b0 con 0x7fefd006ab20 2023-11-28T17:44:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.120+0000 7fefd58a7640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd00c6c30 msgr2=0x7fefd0157fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:44:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.120+0000 7fefd58a7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd00c6c30 0x7fefd0157fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:44:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.120+0000 7fefd48a5640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefd006ab20 0x7fefd0157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.120+0000 7fefd50a6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd006a740 0x7fefd01467f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.120+0000 7fefd50a6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd006a740 0x7fefd01467f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:58766/0 (socket says 172.21.15.98:58766) 2023-11-28T17:44:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.120+0000 7fefd48a5640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefd006ab20 0x7fefd0157a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:45896/0 (socket says 172.21.15.98:45896) 2023-11-28T17:44:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.120+0000 7fefd50a6640 1 -- 172.21.15.98:0/2915397834 learned_addr learned my addr 172.21.15.98:0/2915397834 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:44:15.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.121+0000 7fefd50a6640 1 -- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefd006ab20 msgr2=0x7fefd0157a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:15.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.121+0000 7fefd50a6640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefd006ab20 0x7fefd0157a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:15.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.121+0000 7fefd50a6640 1 -- 172.21.15.98:0/2915397834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd00c6c30 msgr2=0x7fefd0157fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:44:15.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.121+0000 7fefd50a6640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd00c6c30 0x7fefd0157fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:15.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.121+0000 7fefd50a6640 1 -- 172.21.15.98:0/2915397834 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc0009b30 con 0x7fefd006a740 2023-11-28T17:44:15.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.121+0000 7fefd48a5640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fefd006ab20 0x7fefd0157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:44:15.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.122+0000 7fefd50a6640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd006a740 0x7fefd01467f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009e50 tx=0x7fefc0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:15.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.122+0000 7fefcdffb640 1 -- 172.21.15.98:0/2915397834 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc003e450 con 0x7fefd006a740 2023-11-28T17:44:15.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.122+0000 7fefcdffb640 1 -- 172.21.15.98:0/2915397834 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc003e5b0 con 0x7fefd006a740 2023-11-28T17:44:15.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.122+0000 7fefd730a640 1 -- 172.21.15.98:0/2915397834 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd0149590 con 0x7fefd006a740 2023-11-28T17:44:15.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.122+0000 7fefcdffb640 1 -- 172.21.15.98:0/2915397834 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc003e960 con 0x7fefd006a740 2023-11-28T17:44:15.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.122+0000 7fefd730a640 1 -- 172.21.15.98:0/2915397834 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd0150700 con 0x7fefd006a740 2023-11-28T17:44:15.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.123+0000 7fefd730a640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd0147a80 0x7fefd01508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.123+0000 7fefd730a640 1 -- 172.21.15.98:0/2915397834 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd0074460 con 0x7fefd0147a80 2023-11-28T17:44:15.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.124+0000 7fefd48a5640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd0147a80 0x7fefd01508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:15.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.125+0000 7fefd48a5640 1 -- 172.21.15.98:0/2915397834 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefd0075bb0 con 0x7fefd0147a80 2023-11-28T17:44:15.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.125+0000 7fefcdffb640 1 -- 172.21.15.98:0/2915397834 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fefc003eac0 con 0x7fefd006a740 2023-11-28T17:44:15.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.126+0000 7fefd48a5640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd0147a80 0x7fefd01508c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fefd00c6890 tx=0x7fefc4005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:15.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.126+0000 7fefcdffb640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fefbc06b280 0x7fefbc06d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.126+0000 7fefcdffb640 1 -- 172.21.15.98:0/2915397834 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fefc00ac7f0 con 0x7fefd006a740 2023-11-28T17:44:15.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.127+0000 7fefd58a7640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fefbc06b280 0x7fefbc06d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:15.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.127+0000 7fefcdffb640 1 -- 172.21.15.98:0/2915397834 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc400e9b0 con 0x7fefd0147a80 2023-11-28T17:44:15.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.127+0000 7fefcdffb640 1 -- 172.21.15.98:0/2915397834 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fefc400eb10 con 0x7fefd0147a80 2023-11-28T17:44:15.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.127+0000 7fefcdffb640 1 -- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd0147a80 msgr2=0x7fefd01508c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:15.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.127+0000 7fefcdffb640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd0147a80 0x7fefd01508c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fefd00c6890 tx=0x7fefc4005780 comp rx=0 tx=0).stop 2023-11-28T17:44:15.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.127+0000 7fefcdffb640 1 -- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd0147a80 msgr2=0x7fefd01508c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:44:15.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.127+0000 7fefcdffb640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd0147a80 0x7fefd01508c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:15.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.127+0000 7fefd58a7640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fefbc06b280 0x7fefbc06d740 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fefd00c70e0 tx=0x7fefc8006c90 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:15.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.134+0000 7fefd730a640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd01521d0 0x7fefd01419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.134+0000 7fefd730a640 1 -- 172.21.15.98:0/2915397834 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd01502c0 con 0x7fefd01521d0 2023-11-28T17:44:15.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.134+0000 7fefd48a5640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd01521d0 0x7fefd01419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:15.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.135+0000 7fefd48a5640 1 -- 172.21.15.98:0/2915397834 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fefd006e580 con 0x7fefd01521d0 2023-11-28T17:44:15.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.136+0000 7fefd48a5640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd01521d0 0x7fefd01419a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fefc4018000 tx=0x7fefc40059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:15.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.136+0000 7fefcdffb640 1 -- 172.21.15.98:0/2915397834 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc40128c0 con 0x7fefd01521d0 2023-11-28T17:44:15.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.137+0000 7fefcdffb640 1 -- 172.21.15.98:0/2915397834 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4456 (secure 0 0 0) 0x7fefc4012a20 con 0x7fefd01521d0 2023-11-28T17:44:15.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.137+0000 7fefcdffb640 1 -- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd01521d0 msgr2=0x7fefd01419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:15.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.137+0000 7fefcdffb640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd01521d0 0x7fefd01419a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fefc4018000 tx=0x7fefc40059d0 comp rx=0 tx=0).stop 2023-11-28T17:44:15.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.137+0000 7fefcdffb640 1 -- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd01521d0 msgr2=0x7fefd01419a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:44:15.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.137+0000 7fefcdffb640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd01521d0 0x7fefd01419a0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fefc4018000 tx=0x7fefc40059d0 comp rx=0 tx=0).stop 2023-11-28T17:44:15.139 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:44:15.139 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:44:15.139 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:44:15.139 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:44:15.139 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 28, 2023-11-28T17:44:15.139 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:44:15.140 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:44:15.140 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:44:15.140 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:15.140 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:44:15.140 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:44:15.140 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:44:15.140 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:44:15.140 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:44:15.140 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:44:15.140 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:44:15.140 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:44:15.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:44:15.140 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:44:15.141 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:44:15.141 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:44:15.141 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:44:15.141 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:44:15.141 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:15.141 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:44:15.141 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:44:15.141 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:44:15.141 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:44:15.141 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:44:15.141 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:44:15.141 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:44:15.141 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:44:15.142 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:44:15.142 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:44:15.142 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:44:15.142 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:44:15.142 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:15.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.142 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:44:15.142 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:44:15.142 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:44:15.142 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:44:15.142 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:44:15.142 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:44:15.142 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:44:15.143 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:44:15.143 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:44:15.143 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:44:15.143 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:44:15.143 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:44:15.143 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:44:15.143 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:44:15.143 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:44:15.143 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:44:15.143 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:44:15.143 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:44:15.143 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:44:15.143 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:44:15.144 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:44:15.144 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:44:15.144 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:44:15.144 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:44:15.144 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:44:15.144 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:44:15.144 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:44:15.144 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:15.144 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:44:15.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.144 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:44:15.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.144 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:44:15.144 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:44:15.145 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:44:15.145 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:44:15.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:44:15.145 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:44:15.145 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:15.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:44:15.145 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:44:15.145 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:15.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:15.145 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:15.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:44:15.146 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:44:15.146 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:44:15.146 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:44:15.146 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:44:15.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.146 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:44:15.146 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:44:15.146 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:44:15.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:44:15.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:44:15.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:44:15.147 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:15.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:44:15.147 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:44:15.147 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:15.147 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:15.147 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:15.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.147 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:44:15.147 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:44:15.147 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:44:15.147 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:44:15.147 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:44:15.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.148 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:44:15.148 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:44:15.148 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:44:15.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:44:15.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:44:15.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:44:15.148 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:15.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:44:15.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:44:15.148 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:15.149 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:15.149 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:15.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.149 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:44:15.149 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:44:15.149 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:44:15.149 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:44:15.149 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:44:15.149 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:15.149 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:15.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.149 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:44:15.149 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:44:15.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.150 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:15.150 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:15.150 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:44:15.150 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:15.150 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:15.150 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:44:15.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.150 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:15.150 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:15.150 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:44:15.150 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:15.150 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:15.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.151 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:44:15.151 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:44:15.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.137+0000 7fefd730a640 1 -- 172.21.15.98:0/2915397834 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fefbc06b280 msgr2=0x7fefbc06d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:15.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.137+0000 7fefd730a640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fefbc06b280 0x7fefbc06d740 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fefd00c70e0 tx=0x7fefc8006c90 comp rx=0 tx=0).stop 2023-11-28T17:44:15.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.137+0000 7fefd730a640 1 -- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd006a740 msgr2=0x7fefd01467f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:15.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.137+0000 7fefd730a640 1 --2- 172.21.15.98:0/2915397834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fefd006a740 0x7fefd01467f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009e50 tx=0x7fefc0004a40 comp rx=0 tx=0).stop 2023-11-28T17:44:15.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.137+0000 7fefd58a7640 1 -- 172.21.15.98:0/2915397834 reap_dead start 2023-11-28T17:44:15.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.137+0000 7fefd58a7640 1 -- 172.21.15.98:0/2915397834 reap_dead start 2023-11-28T17:44:15.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.137+0000 7fefd730a640 1 -- 172.21.15.98:0/2915397834 shutdown_connections 2023-11-28T17:44:15.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.137+0000 7fefd730a640 1 -- 172.21.15.98:0/2915397834 >> 172.21.15.98:0/2915397834 conn(0x7fefd006ef10 msgr2=0x7fefd014e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:15.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.137+0000 7fefd730a640 1 -- 172.21.15.98:0/2915397834 shutdown_connections 2023-11-28T17:44:15.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.137+0000 7fefd730a640 1 -- 172.21.15.98:0/2915397834 wait complete. 2023-11-28T17:44:15.156 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:44:15.156 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-11-28T17:44:15.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:44:15.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.367+0000 7fbfbe51d640 1 Processor -- start 2023-11-28T17:44:15.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.367+0000 7fbfbe51d640 1 -- start start 2023-11-28T17:44:15.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.369+0000 7fbfbe51d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbfb806a740 0x7fbfb80c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.369+0000 7fbfbe51d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb8154120 0x7fbfb80c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.369+0000 7fbfbe51d640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb80c81a0 con 0x7fbfb806ab20 2023-11-28T17:44:15.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.369+0000 7fbfbe51d640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb806e8f0 con 0x7fbfb806a740 2023-11-28T17:44:15.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.369+0000 7fbfbe51d640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb806dce0 con 0x7fbfb8154120 2023-11-28T17:44:15.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.369+0000 7fbfbcaba640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb8154120 0x7fbfb80c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:15.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.369+0000 7fbfbcaba640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb8154120 0x7fbfb80c7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:45900/0 (socket says 172.21.15.98:45900) 2023-11-28T17:44:15.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.369+0000 7fbfbcaba640 1 -- 172.21.15.98:0/1092071967 learned_addr learned my addr 172.21.15.98:0/1092071967 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:44:15.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.370+0000 7fbfb7fff640 1 --2- 172.21.15.98:0/1092071967 >> v2:172.21.15.113:3300/0 conn(0x7fbfb806a740 0x7fbfb80c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:15.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.370+0000 7fbfbcaba640 1 -- 172.21.15.98:0/1092071967 >> v2:172.21.15.113:3300/0 conn(0x7fbfb806a740 msgr2=0x7fbfb80c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:44:15.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.370+0000 7fbfbcaba640 1 --2- 172.21.15.98:0/1092071967 >> v2:172.21.15.113:3300/0 conn(0x7fbfb806a740 0x7fbfb80c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:15.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.370+0000 7fbfbcaba640 1 -- 172.21.15.98:0/1092071967 >> v1:172.21.15.113:6789/0 conn(0x7fbfb806ab20 legacy=0x7fbfb8153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:15.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.370+0000 7fbfbcaba640 1 -- 172.21.15.98:0/1092071967 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb806dfc0 con 0x7fbfb8154120 2023-11-28T17:44:15.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.371+0000 7fbfbcaba640 1 --2- 172.21.15.98:0/1092071967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb8154120 0x7fbfb80c7b30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009e50 tx=0x7fbfb002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc98ecb2e5576719 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:15.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.372+0000 7fbfb6ffd640 1 -- 172.21.15.98:0/1092071967 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb002cb00 con 0x7fbfb8154120 2023-11-28T17:44:15.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.372+0000 7fbfb6ffd640 1 -- 172.21.15.98:0/1092071967 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb002cc60 con 0x7fbfb8154120 2023-11-28T17:44:15.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.372+0000 7fbfb6ffd640 1 -- 172.21.15.98:0/1092071967 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00343f0 con 0x7fbfb8154120 2023-11-28T17:44:15.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.372+0000 7fbfbe51d640 1 -- 172.21.15.98:0/1092071967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb8154120 msgr2=0x7fbfb80c7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:15.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.372+0000 7fbfbe51d640 1 --2- 172.21.15.98:0/1092071967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb8154120 0x7fbfb80c7b30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009e50 tx=0x7fbfb002bf60 comp rx=0 tx=0).stop 2023-11-28T17:44:15.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.373+0000 7fbfbe51d640 1 -- 172.21.15.98:0/1092071967 shutdown_connections 2023-11-28T17:44:15.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.373+0000 7fbfbe51d640 1 --2- 172.21.15.98:0/1092071967 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb8154120 0x7fbfb80c7b30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:15.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.373+0000 7fbfbe51d640 1 --2- 172.21.15.98:0/1092071967 >> v2:172.21.15.113:3300/0 conn(0x7fbfb806a740 0x7fbfb80c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:15.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.373+0000 7fbfbe51d640 1 -- 172.21.15.98:0/1092071967 >> 172.21.15.98:0/1092071967 conn(0x7fbfb806ef10 msgr2=0x7fbfb806f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:15.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.373+0000 7fbfbe51d640 1 -- 172.21.15.98:0/1092071967 shutdown_connections 2023-11-28T17:44:15.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.373+0000 7fbfbe51d640 1 -- 172.21.15.98:0/1092071967 wait complete. 2023-11-28T17:44:15.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.374+0000 7fbfbe51d640 1 Processor -- start 2023-11-28T17:44:15.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.375+0000 7fbfbe51d640 1 -- start start 2023-11-28T17:44:15.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.375+0000 7fbfbe51d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfb806a740 0x7fbfb81f0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.375+0000 7fbfbe51d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfb806ab20 0x7fbfb81f3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.375+0000 7fbfbe51d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb8154120 0x7fbfb8145240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.375+0000 7fbfbe51d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb81f0a80 con 0x7fbfb806a740 2023-11-28T17:44:15.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.375+0000 7fbfbe51d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb8158a50 con 0x7fbfb806ab20 2023-11-28T17:44:15.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.375+0000 7fbfbe51d640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb8158d30 con 0x7fbfb8154120 2023-11-28T17:44:15.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.376+0000 7fbfb7fff640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfb806a740 msgr2=0x7fbfb81f0540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:44:15.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.376+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfb806a740 0x7fbfb81f0540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:44:15.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.376+0000 7fbfbcaba640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb8154120 0x7fbfb8145240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:15.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.376+0000 7fbfb77fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfb806ab20 0x7fbfb81f3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:15.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.376+0000 7fbfb77fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfb806ab20 0x7fbfb81f3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:58794/0 (socket says 172.21.15.98:58794) 2023-11-28T17:44:15.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.376+0000 7fbfbcaba640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb8154120 0x7fbfb8145240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:45916/0 (socket says 172.21.15.98:45916) 2023-11-28T17:44:15.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.376+0000 7fbfb77fe640 1 -- 172.21.15.98:0/3714273485 learned_addr learned my addr 172.21.15.98:0/3714273485 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:44:15.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.377+0000 7fbfb77fe640 1 -- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb8154120 msgr2=0x7fbfb8145240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:15.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.377+0000 7fbfb77fe640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb8154120 0x7fbfb8145240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:15.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.377+0000 7fbfb77fe640 1 -- 172.21.15.98:0/3714273485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfb806a740 msgr2=0x7fbfb81f0540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:44:15.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.377+0000 7fbfb77fe640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfb806a740 0x7fbfb81f0540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:15.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.377+0000 7fbfb77fe640 1 -- 172.21.15.98:0/3714273485 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb0009b30 con 0x7fbfb806ab20 2023-11-28T17:44:15.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.377+0000 7fbfb77fe640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfb806ab20 0x7fbfb81f3950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00c400 tx=0x7fbfac00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:15.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.378+0000 7fbfb4ff9640 1 -- 172.21.15.98:0/3714273485 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac0052d0 con 0x7fbfb806ab20 2023-11-28T17:44:15.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.378+0000 7fbfb4ff9640 1 -- 172.21.15.98:0/3714273485 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfac005430 con 0x7fbfb806ab20 2023-11-28T17:44:15.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.378+0000 7fbfbe51d640 1 -- 172.21.15.98:0/3714273485 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb8159010 con 0x7fbfb806ab20 2023-11-28T17:44:15.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.378+0000 7fbfb4ff9640 1 -- 172.21.15.98:0/3714273485 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac003830 con 0x7fbfb806ab20 2023-11-28T17:44:15.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.378+0000 7fbfbe51d640 1 -- 172.21.15.98:0/3714273485 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb81f3150 con 0x7fbfb806ab20 2023-11-28T17:44:15.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.379+0000 7fbfbe51d640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb81f3370 0x7fbfb80cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.379+0000 7fbfbe51d640 1 -- 172.21.15.98:0/3714273485 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb8074460 con 0x7fbfb81f3370 2023-11-28T17:44:15.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.379+0000 7fbfb7fff640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb81f3370 0x7fbfb80cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:15.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.380+0000 7fbfb7fff640 1 -- 172.21.15.98:0/3714273485 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbfb80c87e0 con 0x7fbfb81f3370 2023-11-28T17:44:15.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.381+0000 7fbfb7fff640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb81f3370 0x7fbfb80cc2a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800e770 tx=0x7fbfa800ec30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:15.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.381+0000 7fbfb4ff9640 1 -- 172.21.15.98:0/3714273485 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fbfac003990 con 0x7fbfb806ab20 2023-11-28T17:44:15.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.382+0000 7fbfb4ff9640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbfa406b250 0x7fbfa406d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.382+0000 7fbfb4ff9640 1 -- 172.21.15.98:0/3714273485 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbfac089180 con 0x7fbfb806ab20 2023-11-28T17:44:15.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.382+0000 7fbfb4ff9640 1 -- 172.21.15.98:0/3714273485 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa800ee80 con 0x7fbfb81f3370 2023-11-28T17:44:15.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.382+0000 7fbfb4ff9640 1 -- 172.21.15.98:0/3714273485 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fbfa8004020 con 0x7fbfb81f3370 2023-11-28T17:44:15.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.382+0000 7fbfb4ff9640 1 -- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb81f3370 msgr2=0x7fbfb80cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:15.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.382+0000 7fbfbcaba640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbfa406b250 0x7fbfa406d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:15.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.382+0000 7fbfb4ff9640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb81f3370 0x7fbfb80cc2a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800e770 tx=0x7fbfa800ec30 comp rx=0 tx=0).stop 2023-11-28T17:44:15.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.382+0000 7fbfb4ff9640 1 -- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb81f3370 msgr2=0x7fbfb80cc2a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:44:15.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.382+0000 7fbfb4ff9640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb81f3370 0x7fbfb80cc2a0 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800e770 tx=0x7fbfa800ec30 comp rx=0 tx=0).stop 2023-11-28T17:44:15.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.383+0000 7fbfbcaba640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbfa406b250 0x7fbfa406d710 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0035000 tx=0x7fbfb0030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:15.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.390+0000 7fbfbe51d640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb80cc7e0 0x7fbfb8143a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:15.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.390+0000 7fbfbe51d640 1 -- 172.21.15.98:0/3714273485 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb81437e0 con 0x7fbfb80cc7e0 2023-11-28T17:44:15.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.390+0000 7fbfb7fff640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb80cc7e0 0x7fbfb8143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:15.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.391+0000 7fbfb7fff640 1 -- 172.21.15.98:0/3714273485 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbfb806e580 con 0x7fbfb80cc7e0 2023-11-28T17:44:15.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.392+0000 7fbfb7fff640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb80cc7e0 0x7fbfb8143a30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800c5c0 tx=0x7fbfa8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:15.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.392+0000 7fbfb4ff9640 1 -- 172.21.15.98:0/3714273485 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8005da0 con 0x7fbfb80cc7e0 2023-11-28T17:44:15.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.393+0000 7fbfb4ff9640 1 -- 172.21.15.98:0/3714273485 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4964 (secure 0 0 0) 0x7fbfa8003890 con 0x7fbfb80cc7e0 2023-11-28T17:44:15.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.393+0000 7fbfb4ff9640 1 -- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb80cc7e0 msgr2=0x7fbfb8143a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:15.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.393+0000 7fbfb4ff9640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb80cc7e0 0x7fbfb8143a30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800c5c0 tx=0x7fbfa8004730 comp rx=0 tx=0).stop 2023-11-28T17:44:15.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.393+0000 7fbfb4ff9640 1 -- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb80cc7e0 msgr2=0x7fbfb8143a30 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:44:15.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.393+0000 7fbfb4ff9640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbfb80cc7e0 0x7fbfb8143a30 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800c5c0 tx=0x7fbfa8004730 comp rx=0 tx=0).stop 2023-11-28T17:44:15.395 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:44:15.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:44:15.396 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:44:15.396 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:44:15.396 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 28, 2023-11-28T17:44:15.396 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:44:15.396 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:44:15.396 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:44:15.396 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:15.397 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:44:15.397 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:44:15.397 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:44:15.397 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:44:15.397 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:44:15.397 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:44:15.397 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:44:15.398 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:44:15.398 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:44:15.398 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:44:15.398 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:44:15.398 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:44:15.398 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:44:15.398 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:44:15.399 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:15.399 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:44:15.399 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:44:15.399 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:44:15.399 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:44:15.399 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:44:15.399 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:44:15.399 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:44:15.400 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:44:15.400 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:44:15.400 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:44:15.400 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:44:15.400 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:44:15.400 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:15.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.401 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:44:15.401 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:44:15.401 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:44:15.401 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:44:15.401 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:44:15.401 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:44:15.401 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:44:15.401 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:44:15.402 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:44:15.402 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:44:15.402 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:44:15.402 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:44:15.402 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:44:15.402 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:44:15.402 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:44:15.402 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:44:15.403 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:44:15.403 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:44:15.403 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:44:15.403 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:44:15.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:44:15.403 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:44:15.404 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:44:15.404 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:44:15.404 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:44:15.404 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:44:15.404 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:44:15.404 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:15.405 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:44:15.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.405 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:44:15.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.405 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:44:15.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:44:15.405 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:44:15.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:44:15.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:44:15.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:44:15.406 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:15.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:44:15.407 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:44:15.407 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:15.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:15.407 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:15.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.407 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:44:15.407 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:44:15.408 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:44:15.408 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:44:15.408 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:44:15.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.408 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:44:15.408 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:44:15.409 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:44:15.409 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:44:15.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:44:15.409 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:44:15.409 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:15.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:44:15.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:44:15.410 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:15.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:15.410 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:15.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:44:15.411 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:44:15.411 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:44:15.411 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:44:15.411 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:44:15.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.411 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:44:15.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:44:15.412 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:44:15.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:44:15.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:44:15.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:44:15.412 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:15.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:44:15.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:44:15.413 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:15.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:15.413 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:15.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:44:15.413 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:44:15.413 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:44:15.414 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:44:15.414 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:44:15.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:15.414 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:15.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.414 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:44:15.414 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:44:15.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.415 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:15.415 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:15.415 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:44:15.415 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:15.415 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:15.415 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:44:15.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.415 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:15.416 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:15.416 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 6 2023-11-28T17:44:15.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:15.416 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:15.416 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:44:15.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.416 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:15.416 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:15.417 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 8 2023-11-28T17:44:15.417 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:15.417 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:15.417 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:44:15.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.417 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:15.417 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:15.418 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:44:15.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:15.418 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:15.418 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:44:15.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:15.418 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:15.418 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:15.418 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:44:15.419 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:15.419 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:15.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:15.419 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:44:15.419 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:44:15.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.393+0000 7fbfbe51d640 1 -- 172.21.15.98:0/3714273485 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbfa406b250 msgr2=0x7fbfa406d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:15.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.393+0000 7fbfbe51d640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbfa406b250 0x7fbfa406d710 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0035000 tx=0x7fbfb0030000 comp rx=0 tx=0).stop 2023-11-28T17:44:15.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.393+0000 7fbfbe51d640 1 -- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfb806ab20 msgr2=0x7fbfb81f3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:15.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.393+0000 7fbfbe51d640 1 --2- 172.21.15.98:0/3714273485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfb806ab20 0x7fbfb81f3950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00c400 tx=0x7fbfac00c8c0 comp rx=0 tx=0).stop 2023-11-28T17:44:15.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.393+0000 7fbfbcaba640 1 -- 172.21.15.98:0/3714273485 reap_dead start 2023-11-28T17:44:15.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.393+0000 7fbfbcaba640 1 -- 172.21.15.98:0/3714273485 reap_dead start 2023-11-28T17:44:15.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.393+0000 7fbfbe51d640 1 -- 172.21.15.98:0/3714273485 shutdown_connections 2023-11-28T17:44:15.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.393+0000 7fbfbe51d640 1 -- 172.21.15.98:0/3714273485 >> 172.21.15.98:0/3714273485 conn(0x7fbfb806ef10 msgr2=0x7fbfb814f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:15.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.393+0000 7fbfbe51d640 1 -- 172.21.15.98:0/3714273485 shutdown_connections 2023-11-28T17:44:15.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:15.393+0000 7fbfbe51d640 1 -- 172.21.15.98:0/3714273485 wait complete. 2023-11-28T17:44:15.422 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:44:15.422 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-11-28T17:44:17.633 INFO:tasks.workunit.client.0.smithi098.stderr:+ make 2023-11-28T17:44:19.114 INFO:tasks.workunit.client.0.smithi098.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2023-11-28T17:44:19.152 INFO:tasks.workunit.client.0.smithi098.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2023-11-28T17:44:19.157 INFO:tasks.workunit.client.0.smithi098.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2023-11-28T17:44:19.157 INFO:tasks.workunit.client.0.smithi098.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2023-11-28T17:44:19.157 INFO:tasks.workunit.client.0.smithi098.stderr: to install the `Automake' and `Perl' packages. Grab them from 2023-11-28T17:44:19.158 INFO:tasks.workunit.client.0.smithi098.stderr: any GNU archive site. 2023-11-28T17:44:19.177 INFO:tasks.workunit.client.0.smithi098.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2023-11-28T17:44:19.198 INFO:tasks.workunit.client.0.smithi098.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2023-11-28T17:44:19.202 INFO:tasks.workunit.client.0.smithi098.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2023-11-28T17:44:19.202 INFO:tasks.workunit.client.0.smithi098.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2023-11-28T17:44:19.202 INFO:tasks.workunit.client.0.smithi098.stderr: You might want to install the `Automake' and `Perl' packages. 2023-11-28T17:44:19.202 INFO:tasks.workunit.client.0.smithi098.stderr: Grab them from any GNU archive site. 2023-11-28T17:44:21.995 INFO:tasks.workunit.client.0.smithi098.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2023-11-28T17:44:22.701 INFO:tasks.workunit.client.0.smithi098.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2023-11-28T17:44:22.702 INFO:tasks.workunit.client.0.smithi098.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2023-11-28T17:44:22.702 INFO:tasks.workunit.client.0.smithi098.stderr:If you have problems, you may need to regenerate the build system entirely. 2023-11-28T17:44:22.702 INFO:tasks.workunit.client.0.smithi098.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2023-11-28T17:44:23.060 INFO:tasks.workunit.client.0.smithi098.stdout:/bin/sh ./config.status --recheck 2023-11-28T17:44:23.355 INFO:tasks.workunit.client.0.smithi098.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2023-11-28T17:44:24.044 INFO:tasks.workunit.client.0.smithi098.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2023-11-28T17:44:25.094 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether build environment is sane... yes 2023-11-28T17:44:25.130 INFO:tasks.workunit.client.0.smithi098.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2023-11-28T17:44:25.132 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gawk... gawk 2023-11-28T17:44:25.196 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether make sets $(MAKE)... yes 2023-11-28T17:44:25.359 INFO:tasks.workunit.client.0.smithi098.stdout:checking build system type... x86_64-unknown-linux-gnu 2023-11-28T17:44:25.360 INFO:tasks.workunit.client.0.smithi098.stdout:checking host system type... x86_64-unknown-linux-gnu 2023-11-28T17:44:25.415 INFO:tasks.workunit.client.0.smithi098.stdout:checking for style of include used by make... GNU 2023-11-28T17:44:25.446 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gcc... gcc 2023-11-28T17:44:25.939 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether the C compiler works... yes 2023-11-28T17:44:25.941 INFO:tasks.workunit.client.0.smithi098.stdout:checking for C compiler default output file name... a.out 2023-11-28T17:44:26.169 INFO:tasks.workunit.client.0.smithi098.stdout:checking for suffix of executables... 2023-11-28T17:44:26.401 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether we are cross compiling... no 2023-11-28T17:44:26.595 INFO:tasks.workunit.client.0.smithi098.stdout:checking for suffix of object files... o 2023-11-28T17:44:26.757 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether we are using the GNU C compiler... yes 2023-11-28T17:44:26.933 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether gcc accepts -g... yes 2023-11-28T17:44:27.133 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gcc option to accept ISO C89... none needed 2023-11-28T17:44:27.996 INFO:tasks.workunit.client.0.smithi098.stdout:checking dependency style of gcc... gcc3 2023-11-28T17:44:28.500 INFO:tasks.workunit.client.0.smithi098.stdout:checking how to run the C preprocessor... gcc -E 2023-11-28T17:44:28.829 INFO:tasks.workunit.client.0.smithi098.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2023-11-28T17:44:28.836 INFO:tasks.workunit.client.0.smithi098.stdout:checking for egrep... /usr/bin/grep -E 2023-11-28T17:44:29.489 INFO:tasks.workunit.client.0.smithi098.stdout:checking for ANSI C header files... yes 2023-11-28T17:44:29.693 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/types.h... yes 2023-11-28T17:44:29.904 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/stat.h... yes 2023-11-28T17:44:30.113 INFO:tasks.workunit.client.0.smithi098.stdout:checking for stdlib.h... yes 2023-11-28T17:44:31.194 INFO:tasks.workunit.client.0.smithi098.stdout:checking for string.h... yes 2023-11-28T17:44:31.407 INFO:tasks.workunit.client.0.smithi098.stdout:checking for memory.h... yes 2023-11-28T17:44:31.688 INFO:tasks.workunit.client.0.smithi098.stdout:checking for strings.h... yes 2023-11-28T17:44:31.899 INFO:tasks.workunit.client.0.smithi098.stdout:checking for inttypes.h... yes 2023-11-28T17:44:32.123 INFO:tasks.workunit.client.0.smithi098.stdout:checking for stdint.h... yes 2023-11-28T17:44:32.355 INFO:tasks.workunit.client.0.smithi098.stdout:checking for unistd.h... yes 2023-11-28T17:44:33.265 INFO:tasks.workunit.client.0.smithi098.stdout:checking minix/config.h usability... no 2023-11-28T17:44:33.463 INFO:tasks.workunit.client.0.smithi098.stdout:checking minix/config.h presence... no 2023-11-28T17:44:33.465 INFO:tasks.workunit.client.0.smithi098.stdout:checking for minix/config.h... no 2023-11-28T17:44:34.190 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2023-11-28T17:44:34.675 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gcc... (cached) gcc 2023-11-28T17:44:34.984 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether we are using the GNU C compiler... (cached) yes 2023-11-28T17:44:34.986 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether gcc accepts -g... (cached) yes 2023-11-28T17:44:34.988 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2023-11-28T17:44:34.990 INFO:tasks.workunit.client.0.smithi098.stdout:checking dependency style of gcc... (cached) gcc3 2023-11-28T17:44:35.287 INFO:tasks.workunit.client.0.smithi098.stdout:checking for main in -lm... yes 2023-11-28T17:44:35.423 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-11-28T17:44:35.423 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-11-28T17:44:35.424 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-28T17:44:35.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-28T17:44:35.432 INFO:tasks.ceph.mon.a:Started 2023-11-28T17:44:35.433 INFO:tasks.mon_thrash:Sending STOP to mon b 2023-11-28T17:44:35.433 INFO:tasks.ceph.mon.b:Sent signal 19 2023-11-28T17:44:35.433 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2023-11-28T17:44:35.546 INFO:tasks.workunit.client.0.smithi098.stdout:checking for main in -lpthread... yes 2023-11-28T17:44:35.556 INFO:tasks.workunit.client.0.smithi098.stdout:checking for ANSI C header files... (cached) yes 2023-11-28T17:44:35.764 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2023-11-28T17:44:36.224 INFO:tasks.workunit.client.0.smithi098.stdout:checking pthread.h usability... yes 2023-11-28T17:44:36.361 INFO:tasks.workunit.client.0.smithi098.stdout:checking pthread.h presence... yes 2023-11-28T17:44:36.362 INFO:tasks.workunit.client.0.smithi098.stdout:checking for pthread.h... yes 2023-11-28T17:44:36.652 INFO:tasks.workunit.client.0.smithi098.stdout:checking fcntl.h usability... yes 2023-11-28T17:44:37.338 INFO:tasks.workunit.client.0.smithi098.stdout:checking fcntl.h presence... yes 2023-11-28T17:44:37.340 INFO:tasks.workunit.client.0.smithi098.stdout:checking for fcntl.h... yes 2023-11-28T17:44:37.609 INFO:tasks.workunit.client.0.smithi098.stdout:checking limits.h usability... yes 2023-11-28T17:44:37.745 INFO:tasks.workunit.client.0.smithi098.stdout:checking limits.h presence... yes 2023-11-28T17:44:37.747 INFO:tasks.workunit.client.0.smithi098.stdout:checking for limits.h... yes 2023-11-28T17:44:37.797 INFO:tasks.workunit.client.0.smithi098.stdout:checking for stdint.h... (cached) yes 2023-11-28T17:44:38.337 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/time.h usability... yes 2023-11-28T17:44:38.459 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/time.h presence... yes 2023-11-28T17:44:38.461 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/time.h... yes 2023-11-28T17:44:38.504 INFO:tasks.workunit.client.0.smithi098.stdout:checking for unistd.h... (cached) yes 2023-11-28T17:44:38.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:44:38.537 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:44:38.687 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/vfs.h usability... yes 2023-11-28T17:44:38.798 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/vfs.h presence... yes 2023-11-28T17:44:38.800 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/vfs.h... yes 2023-11-28T17:44:39.947 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/limits.h usability... no 2023-11-28T17:44:40.100 INFO:tasks.workunit.client.0.smithi098.stdout:checking sys/limits.h presence... no 2023-11-28T17:44:40.102 INFO:tasks.workunit.client.0.smithi098.stdout:checking for sys/limits.h... no 2023-11-28T17:44:40.243 INFO:tasks.workunit.client.0.smithi098.stdout:checking for an ANSI C-conforming const... yes 2023-11-28T17:44:40.415 INFO:tasks.workunit.client.0.smithi098.stdout:checking whether time.h and sys/time.h may both be included... yes 2023-11-28T17:44:40.709 INFO:tasks.workunit.client.0.smithi098.stdout:checking for system... yes 2023-11-28T17:44:41.102 INFO:tasks.workunit.client.0.smithi098.stdout:checking for gettimeofday... yes 2023-11-28T17:44:41.426 INFO:tasks.workunit.client.0.smithi098.stdout:checking for mkdir... yes 2023-11-28T17:44:41.751 INFO:tasks.workunit.client.0.smithi098.stdout:checking for strerror... yes 2023-11-28T17:44:42.065 INFO:tasks.workunit.client.0.smithi098.stdout:checking for open64... yes 2023-11-28T17:44:42.878 INFO:tasks.workunit.client.0.smithi098.stdout:checking for stat64... yes 2023-11-28T17:44:43.191 INFO:tasks.workunit.client.0.smithi098.stdout:checking for fseeko64... yes 2023-11-28T17:44:43.621 INFO:tasks.workunit.client.0.smithi098.stdout:checking for lrand48_r... yes 2023-11-28T17:44:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:checking for srand48_r... yes 2023-11-28T17:44:44.042 INFO:tasks.workunit.client.0.smithi098.stdout:configure: creating ./config.status 2023-11-28T17:44:44.408 INFO:tasks.workunit.client.0.smithi098.stdout: /bin/sh ./config.status 2023-11-28T17:44:44.712 INFO:tasks.workunit.client.0.smithi098.stdout:config.status: creating Makefile 2023-11-28T17:44:44.810 INFO:tasks.workunit.client.0.smithi098.stdout:config.status: creating config.h 2023-11-28T17:44:44.922 INFO:tasks.workunit.client.0.smithi098.stdout:config.status: executing depfiles commands 2023-11-28T17:44:44.990 INFO:tasks.workunit.client.0.smithi098.stdout:config.status: executing default commands 2023-11-28T17:44:45.213 INFO:tasks.workunit.client.0.smithi098.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2023-11-28T17:44:45.613 INFO:tasks.workunit.client.0.smithi098.stdout:rm -f stamp-h1 2023-11-28T17:44:45.629 INFO:tasks.workunit.client.0.smithi098.stdout:touch config.h.in 2023-11-28T17:44:45.642 INFO:tasks.workunit.client.0.smithi098.stdout:cd . && /bin/sh ./config.status config.h 2023-11-28T17:44:46.679 INFO:tasks.workunit.client.0.smithi098.stdout:config.status: creating config.h 2023-11-28T17:44:46.862 INFO:tasks.workunit.client.0.smithi098.stdout:make all-am 2023-11-28T17:44:47.201 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2023-11-28T17:44:47.202 INFO:tasks.workunit.client.0.smithi098.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2023-11-28T17:44:47.540 INFO:tasks.ceph.mds.e.smithi098.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7fe3363c2640 time 2023-11-28T17:44:47.539813+0000 2023-11-28T17:44:47.540 INFO:tasks.ceph.mds.e.smithi098.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:44:47.541 INFO:tasks.ceph.mds.e.smithi098.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:44:47.541 INFO:tasks.ceph.mds.e.smithi098.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fe33bf6eef8] 2023-11-28T17:44:47.541 INFO:tasks.ceph.mds.e.smithi098.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fe33bf6f0b6] 2023-11-28T17:44:47.541 INFO:tasks.ceph.mds.e.smithi098.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5608ca82e329] 2023-11-28T17:44:47.541 INFO:tasks.ceph.mds.e.smithi098.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5608ca8235b5] 2023-11-28T17:44:47.541 INFO:tasks.ceph.mds.e.smithi098.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x5608ca7d285b] 2023-11-28T17:44:47.541 INFO:tasks.ceph.mds.e.smithi098.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5608ca70c521] 2023-11-28T17:44:47.541 INFO:tasks.ceph.mds.e.smithi098.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5608ca713508] 2023-11-28T17:44:47.541 INFO:tasks.ceph.mds.e.smithi098.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5608ca708edd] 2023-11-28T17:44:47.541 INFO:tasks.ceph.mds.e.smithi098.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5608ca7097bc] 2023-11-28T17:44:47.542 INFO:tasks.ceph.mds.e.smithi098.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5608ca6f5467] 2023-11-28T17:44:47.542 INFO:tasks.ceph.mds.e.smithi098.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x5608ca67948b] 2023-11-28T17:44:47.542 INFO:tasks.ceph.mds.e.smithi098.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5608ca67997c] 2023-11-28T17:44:47.542 INFO:tasks.ceph.mds.e.smithi098.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5608ca6634b5] 2023-11-28T17:44:47.542 INFO:tasks.ceph.mds.e.smithi098.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7fe33c16040a] 2023-11-28T17:44:47.542 INFO:tasks.ceph.mds.e.smithi098.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fe33c1f24a1] 2023-11-28T17:44:47.542 INFO:tasks.ceph.mds.e.smithi098.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7fe33b69f802] 2023-11-28T17:44:47.542 INFO:tasks.ceph.mds.e.smithi098.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7fe33b63f450] 2023-11-28T17:44:47.542 INFO:tasks.ceph.mds.e.smithi098.stderr:*** Caught signal (Aborted) ** 2023-11-28T17:44:47.542 INFO:tasks.ceph.mds.e.smithi098.stderr: in thread 7fe3363c2640 thread_name:ms_dispatch 2023-11-28T17:44:47.542 INFO:tasks.ceph.mds.e.smithi098.stderr:2023-11-28T17:44:47.539+0000 7fe3363c2640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7fe3363c2640 time 2023-11-28T17:44:47.539813+0000 2023-11-28T17:44:47.542 INFO:tasks.ceph.mds.e.smithi098.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:44:47.542 INFO:tasks.ceph.mds.e.smithi098.stderr: 2023-11-28T17:44:47.543 INFO:tasks.ceph.mds.e.smithi098.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:44:47.543 INFO:tasks.ceph.mds.e.smithi098.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fe33bf6eef8] 2023-11-28T17:44:47.543 INFO:tasks.ceph.mds.e.smithi098.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fe33bf6f0b6] 2023-11-28T17:44:47.543 INFO:tasks.ceph.mds.e.smithi098.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5608ca82e329] 2023-11-28T17:44:47.543 INFO:tasks.ceph.mds.e.smithi098.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5608ca8235b5] 2023-11-28T17:44:47.543 INFO:tasks.ceph.mds.e.smithi098.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x5608ca7d285b] 2023-11-28T17:44:47.543 INFO:tasks.ceph.mds.e.smithi098.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5608ca70c521] 2023-11-28T17:44:47.543 INFO:tasks.ceph.mds.e.smithi098.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5608ca713508] 2023-11-28T17:44:47.543 INFO:tasks.ceph.mds.e.smithi098.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5608ca708edd] 2023-11-28T17:44:47.543 INFO:tasks.ceph.mds.e.smithi098.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5608ca7097bc] 2023-11-28T17:44:47.543 INFO:tasks.ceph.mds.e.smithi098.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5608ca6f5467] 2023-11-28T17:44:47.543 INFO:tasks.ceph.mds.e.smithi098.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x5608ca67948b] 2023-11-28T17:44:47.543 INFO:tasks.ceph.mds.e.smithi098.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5608ca67997c] 2023-11-28T17:44:47.544 INFO:tasks.ceph.mds.e.smithi098.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5608ca6634b5] 2023-11-28T17:44:47.544 INFO:tasks.ceph.mds.e.smithi098.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7fe33c16040a] 2023-11-28T17:44:47.544 INFO:tasks.ceph.mds.e.smithi098.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fe33c1f24a1] 2023-11-28T17:44:47.544 INFO:tasks.ceph.mds.e.smithi098.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7fe33b69f802] 2023-11-28T17:44:47.544 INFO:tasks.ceph.mds.e.smithi098.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7fe33b63f450] 2023-11-28T17:44:47.544 INFO:tasks.ceph.mds.e.smithi098.stderr: 2023-11-28T17:44:47.544 INFO:tasks.ceph.mds.e.smithi098.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:44:47.544 INFO:tasks.ceph.mds.e.smithi098.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fe33b654db0] 2023-11-28T17:44:47.544 INFO:tasks.ceph.mds.e.smithi098.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fe33b6a154c] 2023-11-28T17:44:47.544 INFO:tasks.ceph.mds.e.smithi098.stderr: 3: raise() 2023-11-28T17:44:47.544 INFO:tasks.ceph.mds.e.smithi098.stderr: 4: abort() 2023-11-28T17:44:47.544 INFO:tasks.ceph.mds.e.smithi098.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fe33bf6ef52] 2023-11-28T17:44:47.544 INFO:tasks.ceph.mds.e.smithi098.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fe33bf6f0b6] 2023-11-28T17:44:47.544 INFO:tasks.ceph.mds.e.smithi098.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5608ca82e329] 2023-11-28T17:44:47.545 INFO:tasks.ceph.mds.e.smithi098.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5608ca8235b5] 2023-11-28T17:44:47.545 INFO:tasks.ceph.mds.e.smithi098.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x5608ca7d285b] 2023-11-28T17:44:47.545 INFO:tasks.ceph.mds.e.smithi098.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5608ca70c521] 2023-11-28T17:44:47.545 INFO:tasks.ceph.mds.e.smithi098.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5608ca713508] 2023-11-28T17:44:47.545 INFO:tasks.ceph.mds.e.smithi098.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5608ca708edd] 2023-11-28T17:44:47.545 INFO:tasks.ceph.mds.e.smithi098.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5608ca7097bc] 2023-11-28T17:44:47.545 INFO:tasks.ceph.mds.e.smithi098.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5608ca6f5467] 2023-11-28T17:44:47.545 INFO:tasks.ceph.mds.e.smithi098.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x5608ca67948b] 2023-11-28T17:44:47.545 INFO:tasks.ceph.mds.e.smithi098.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5608ca67997c] 2023-11-28T17:44:47.545 INFO:tasks.ceph.mds.e.smithi098.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5608ca6634b5] 2023-11-28T17:44:47.545 INFO:tasks.ceph.mds.e.smithi098.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7fe33c16040a] 2023-11-28T17:44:47.545 INFO:tasks.ceph.mds.e.smithi098.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fe33c1f24a1] 2023-11-28T17:44:47.545 INFO:tasks.ceph.mds.e.smithi098.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7fe33b69f802] 2023-11-28T17:44:47.545 INFO:tasks.ceph.mds.e.smithi098.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7fe33b63f450] 2023-11-28T17:44:47.546 INFO:tasks.ceph.mds.e.smithi098.stderr:2023-11-28T17:44:47.541+0000 7fe3363c2640 -1 *** Caught signal (Aborted) ** 2023-11-28T17:44:47.546 INFO:tasks.ceph.mds.e.smithi098.stderr: in thread 7fe3363c2640 thread_name:ms_dispatch 2023-11-28T17:44:47.546 INFO:tasks.ceph.mds.e.smithi098.stderr: 2023-11-28T17:44:47.546 INFO:tasks.ceph.mds.e.smithi098.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:44:47.546 INFO:tasks.ceph.mds.e.smithi098.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fe33b654db0] 2023-11-28T17:44:47.546 INFO:tasks.ceph.mds.e.smithi098.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fe33b6a154c] 2023-11-28T17:44:47.546 INFO:tasks.ceph.mds.e.smithi098.stderr: 3: raise() 2023-11-28T17:44:47.546 INFO:tasks.ceph.mds.e.smithi098.stderr: 4: abort() 2023-11-28T17:44:47.546 INFO:tasks.ceph.mds.e.smithi098.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fe33bf6ef52] 2023-11-28T17:44:47.546 INFO:tasks.ceph.mds.e.smithi098.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fe33bf6f0b6] 2023-11-28T17:44:47.547 INFO:tasks.ceph.mds.e.smithi098.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5608ca82e329] 2023-11-28T17:44:47.547 INFO:tasks.ceph.mds.e.smithi098.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5608ca8235b5] 2023-11-28T17:44:47.547 INFO:tasks.ceph.mds.e.smithi098.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x5608ca7d285b] 2023-11-28T17:44:47.547 INFO:tasks.ceph.mds.e.smithi098.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5608ca70c521] 2023-11-28T17:44:47.547 INFO:tasks.ceph.mds.e.smithi098.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5608ca713508] 2023-11-28T17:44:47.547 INFO:tasks.ceph.mds.e.smithi098.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5608ca708edd] 2023-11-28T17:44:47.547 INFO:tasks.ceph.mds.e.smithi098.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5608ca7097bc] 2023-11-28T17:44:47.547 INFO:tasks.ceph.mds.e.smithi098.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5608ca6f5467] 2023-11-28T17:44:47.547 INFO:tasks.ceph.mds.e.smithi098.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x5608ca67948b] 2023-11-28T17:44:47.547 INFO:tasks.ceph.mds.e.smithi098.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5608ca67997c] 2023-11-28T17:44:47.547 INFO:tasks.ceph.mds.e.smithi098.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5608ca6634b5] 2023-11-28T17:44:47.547 INFO:tasks.ceph.mds.e.smithi098.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7fe33c16040a] 2023-11-28T17:44:47.547 INFO:tasks.ceph.mds.e.smithi098.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fe33c1f24a1] 2023-11-28T17:44:47.548 INFO:tasks.ceph.mds.e.smithi098.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7fe33b69f802] 2023-11-28T17:44:47.548 INFO:tasks.ceph.mds.e.smithi098.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7fe33b63f450] 2023-11-28T17:44:47.548 INFO:tasks.ceph.mds.e.smithi098.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-28T17:44:47.548 INFO:tasks.ceph.mds.e.smithi098.stderr: 2023-11-28T17:44:47.566 INFO:tasks.ceph.mds.e.smithi098.stderr: -1> 2023-11-28T17:44:47.539+0000 7fe3363c2640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7fe3363c2640 time 2023-11-28T17:44:47.539813+0000 2023-11-28T17:44:47.566 INFO:tasks.ceph.mds.e.smithi098.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:44:47.566 INFO:tasks.ceph.mds.e.smithi098.stderr: 2023-11-28T17:44:47.566 INFO:tasks.ceph.mds.e.smithi098.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:44:47.567 INFO:tasks.ceph.mds.e.smithi098.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fe33bf6eef8] 2023-11-28T17:44:47.567 INFO:tasks.ceph.mds.e.smithi098.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fe33bf6f0b6] 2023-11-28T17:44:47.567 INFO:tasks.ceph.mds.e.smithi098.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5608ca82e329] 2023-11-28T17:44:47.567 INFO:tasks.ceph.mds.e.smithi098.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5608ca8235b5] 2023-11-28T17:44:47.567 INFO:tasks.ceph.mds.e.smithi098.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x5608ca7d285b] 2023-11-28T17:44:47.567 INFO:tasks.ceph.mds.e.smithi098.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5608ca70c521] 2023-11-28T17:44:47.567 INFO:tasks.ceph.mds.e.smithi098.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5608ca713508] 2023-11-28T17:44:47.567 INFO:tasks.ceph.mds.e.smithi098.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5608ca708edd] 2023-11-28T17:44:47.567 INFO:tasks.ceph.mds.e.smithi098.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5608ca7097bc] 2023-11-28T17:44:47.567 INFO:tasks.ceph.mds.e.smithi098.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5608ca6f5467] 2023-11-28T17:44:47.567 INFO:tasks.ceph.mds.e.smithi098.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x5608ca67948b] 2023-11-28T17:44:47.567 INFO:tasks.ceph.mds.e.smithi098.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5608ca67997c] 2023-11-28T17:44:47.568 INFO:tasks.ceph.mds.e.smithi098.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5608ca6634b5] 2023-11-28T17:44:47.568 INFO:tasks.ceph.mds.e.smithi098.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7fe33c16040a] 2023-11-28T17:44:47.568 INFO:tasks.ceph.mds.e.smithi098.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fe33c1f24a1] 2023-11-28T17:44:47.568 INFO:tasks.ceph.mds.e.smithi098.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7fe33b69f802] 2023-11-28T17:44:47.568 INFO:tasks.ceph.mds.e.smithi098.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7fe33b63f450] 2023-11-28T17:44:47.568 INFO:tasks.ceph.mds.e.smithi098.stderr: 2023-11-28T17:44:47.568 INFO:tasks.ceph.mds.e.smithi098.stderr: 0> 2023-11-28T17:44:47.541+0000 7fe3363c2640 -1 *** Caught signal (Aborted) ** 2023-11-28T17:44:47.568 INFO:tasks.ceph.mds.e.smithi098.stderr: in thread 7fe3363c2640 thread_name:ms_dispatch 2023-11-28T17:44:47.568 INFO:tasks.ceph.mds.e.smithi098.stderr: 2023-11-28T17:44:47.568 INFO:tasks.ceph.mds.e.smithi098.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:44:47.568 INFO:tasks.ceph.mds.e.smithi098.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fe33b654db0] 2023-11-28T17:44:47.569 INFO:tasks.ceph.mds.e.smithi098.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fe33b6a154c] 2023-11-28T17:44:47.569 INFO:tasks.ceph.mds.e.smithi098.stderr: 3: raise() 2023-11-28T17:44:47.569 INFO:tasks.ceph.mds.e.smithi098.stderr: 4: abort() 2023-11-28T17:44:47.569 INFO:tasks.ceph.mds.e.smithi098.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fe33bf6ef52] 2023-11-28T17:44:47.569 INFO:tasks.ceph.mds.e.smithi098.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fe33bf6f0b6] 2023-11-28T17:44:47.569 INFO:tasks.ceph.mds.e.smithi098.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5608ca82e329] 2023-11-28T17:44:47.569 INFO:tasks.ceph.mds.e.smithi098.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5608ca8235b5] 2023-11-28T17:44:47.569 INFO:tasks.ceph.mds.e.smithi098.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x5608ca7d285b] 2023-11-28T17:44:47.569 INFO:tasks.ceph.mds.e.smithi098.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5608ca70c521] 2023-11-28T17:44:47.569 INFO:tasks.ceph.mds.e.smithi098.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5608ca713508] 2023-11-28T17:44:47.569 INFO:tasks.ceph.mds.e.smithi098.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5608ca708edd] 2023-11-28T17:44:47.569 INFO:tasks.ceph.mds.e.smithi098.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5608ca7097bc] 2023-11-28T17:44:47.569 INFO:tasks.ceph.mds.e.smithi098.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5608ca6f5467] 2023-11-28T17:44:47.569 INFO:tasks.ceph.mds.e.smithi098.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x5608ca67948b] 2023-11-28T17:44:47.570 INFO:tasks.ceph.mds.e.smithi098.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5608ca67997c] 2023-11-28T17:44:47.570 INFO:tasks.ceph.mds.e.smithi098.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5608ca6634b5] 2023-11-28T17:44:47.570 INFO:tasks.ceph.mds.e.smithi098.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7fe33c16040a] 2023-11-28T17:44:47.570 INFO:tasks.ceph.mds.e.smithi098.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fe33c1f24a1] 2023-11-28T17:44:47.570 INFO:tasks.ceph.mds.e.smithi098.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7fe33b69f802] 2023-11-28T17:44:47.570 INFO:tasks.ceph.mds.e.smithi098.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7fe33b63f450] 2023-11-28T17:44:47.570 INFO:tasks.ceph.mds.e.smithi098.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-28T17:44:47.570 INFO:tasks.ceph.mds.e.smithi098.stderr: 2023-11-28T17:44:47.578 INFO:tasks.ceph.mds.e.smithi098.stderr: -1> 2023-11-28T17:44:47.539+0000 7fe3363c2640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7fe3363c2640 time 2023-11-28T17:44:47.539813+0000 2023-11-28T17:44:47.578 INFO:tasks.ceph.mds.e.smithi098.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:44:47.578 INFO:tasks.ceph.mds.e.smithi098.stderr: 2023-11-28T17:44:47.578 INFO:tasks.ceph.mds.e.smithi098.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:44:47.579 INFO:tasks.ceph.mds.e.smithi098.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fe33bf6eef8] 2023-11-28T17:44:47.579 INFO:tasks.ceph.mds.e.smithi098.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fe33bf6f0b6] 2023-11-28T17:44:47.579 INFO:tasks.ceph.mds.e.smithi098.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5608ca82e329] 2023-11-28T17:44:47.579 INFO:tasks.ceph.mds.e.smithi098.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5608ca8235b5] 2023-11-28T17:44:47.579 INFO:tasks.ceph.mds.e.smithi098.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x5608ca7d285b] 2023-11-28T17:44:47.579 INFO:tasks.ceph.mds.e.smithi098.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5608ca70c521] 2023-11-28T17:44:47.579 INFO:tasks.ceph.mds.e.smithi098.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5608ca713508] 2023-11-28T17:44:47.579 INFO:tasks.ceph.mds.e.smithi098.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5608ca708edd] 2023-11-28T17:44:47.579 INFO:tasks.ceph.mds.e.smithi098.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5608ca7097bc] 2023-11-28T17:44:47.579 INFO:tasks.ceph.mds.e.smithi098.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5608ca6f5467] 2023-11-28T17:44:47.579 INFO:tasks.ceph.mds.e.smithi098.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x5608ca67948b] 2023-11-28T17:44:47.579 INFO:tasks.ceph.mds.e.smithi098.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5608ca67997c] 2023-11-28T17:44:47.579 INFO:tasks.ceph.mds.e.smithi098.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5608ca6634b5] 2023-11-28T17:44:47.580 INFO:tasks.ceph.mds.e.smithi098.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7fe33c16040a] 2023-11-28T17:44:47.580 INFO:tasks.ceph.mds.e.smithi098.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fe33c1f24a1] 2023-11-28T17:44:47.580 INFO:tasks.ceph.mds.e.smithi098.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7fe33b69f802] 2023-11-28T17:44:47.580 INFO:tasks.ceph.mds.e.smithi098.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7fe33b63f450] 2023-11-28T17:44:47.580 INFO:tasks.ceph.mds.e.smithi098.stderr: 2023-11-28T17:44:47.580 INFO:tasks.ceph.mds.e.smithi098.stderr: 0> 2023-11-28T17:44:47.541+0000 7fe3363c2640 -1 *** Caught signal (Aborted) ** 2023-11-28T17:44:47.580 INFO:tasks.ceph.mds.e.smithi098.stderr: in thread 7fe3363c2640 thread_name:ms_dispatch 2023-11-28T17:44:47.580 INFO:tasks.ceph.mds.e.smithi098.stderr: 2023-11-28T17:44:47.580 INFO:tasks.ceph.mds.e.smithi098.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:44:47.580 INFO:tasks.ceph.mds.e.smithi098.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fe33b654db0] 2023-11-28T17:44:47.580 INFO:tasks.ceph.mds.e.smithi098.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fe33b6a154c] 2023-11-28T17:44:47.580 INFO:tasks.ceph.mds.e.smithi098.stderr: 3: raise() 2023-11-28T17:44:47.580 INFO:tasks.ceph.mds.e.smithi098.stderr: 4: abort() 2023-11-28T17:44:47.581 INFO:tasks.ceph.mds.e.smithi098.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fe33bf6ef52] 2023-11-28T17:44:47.581 INFO:tasks.ceph.mds.e.smithi098.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fe33bf6f0b6] 2023-11-28T17:44:47.581 INFO:tasks.ceph.mds.e.smithi098.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5608ca82e329] 2023-11-28T17:44:47.581 INFO:tasks.ceph.mds.e.smithi098.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5608ca8235b5] 2023-11-28T17:44:47.581 INFO:tasks.ceph.mds.e.smithi098.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x5608ca7d285b] 2023-11-28T17:44:47.581 INFO:tasks.ceph.mds.e.smithi098.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5608ca70c521] 2023-11-28T17:44:47.581 INFO:tasks.ceph.mds.e.smithi098.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5608ca713508] 2023-11-28T17:44:47.581 INFO:tasks.ceph.mds.e.smithi098.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5608ca708edd] 2023-11-28T17:44:47.581 INFO:tasks.ceph.mds.e.smithi098.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5608ca7097bc] 2023-11-28T17:44:47.581 INFO:tasks.ceph.mds.e.smithi098.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5608ca6f5467] 2023-11-28T17:44:47.581 INFO:tasks.ceph.mds.e.smithi098.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x5608ca67948b] 2023-11-28T17:44:47.581 INFO:tasks.ceph.mds.e.smithi098.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5608ca67997c] 2023-11-28T17:44:47.581 INFO:tasks.ceph.mds.e.smithi098.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5608ca6634b5] 2023-11-28T17:44:47.582 INFO:tasks.ceph.mds.e.smithi098.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7fe33c16040a] 2023-11-28T17:44:47.582 INFO:tasks.ceph.mds.e.smithi098.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fe33c1f24a1] 2023-11-28T17:44:47.582 INFO:tasks.ceph.mds.e.smithi098.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7fe33b69f802] 2023-11-28T17:44:47.582 INFO:tasks.ceph.mds.e.smithi098.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7fe33b63f450] 2023-11-28T17:44:47.582 INFO:tasks.ceph.mds.e.smithi098.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-28T17:44:47.582 INFO:tasks.ceph.mds.e.smithi098.stderr: 2023-11-28T17:44:47.931 INFO:tasks.ceph.mds.e.smithi098.stderr:daemon-helper: command crashed with signal 6 2023-11-28T17:44:50.434 INFO:tasks.mon_thrash:Sending CONT to mon b 2023-11-28T17:44:50.434 INFO:tasks.ceph.mon.b:Sent signal 18 2023-11-28T17:44:50.434 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:44:50.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:44:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.655+0000 7f7f5086b640 1 Processor -- start 2023-11-28T17:44:50.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.655+0000 7f7f5086b640 1 -- start start 2023-11-28T17:44:50.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.657+0000 7f7f5086b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7f4c157b60 0x7f7f4c15bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:50.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.657+0000 7f7f5086b640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7f4c0c4910 0x7f7f4c0c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:50.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.657+0000 7f7f5086b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c15b150 con 0x7f7f4c151d80 2023-11-28T17:44:50.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.657+0000 7f7f5086b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f4c0c5260 con 0x7f7f4c157b60 2023-11-28T17:44:50.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.657+0000 7f7f5086b640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f4c157f90 con 0x7f7f4c0c4910 2023-11-28T17:44:50.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.657+0000 7f7f43fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7f4c157b60 0x7f7f4c15bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:50.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.657+0000 7f7f43fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7f4c157b60 0x7f7f4c15bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55240/0 (socket says 172.21.15.98:55240) 2023-11-28T17:44:50.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.657+0000 7f7f43fff640 1 -- 172.21.15.98:0/3544976308 learned_addr learned my addr 172.21.15.98:0/3544976308 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:44:50.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.658+0000 7f7f43fff640 1 -- 172.21.15.98:0/3544976308 >> v2:172.21.15.98:3300/0 conn(0x7f7f4c157b60 msgr2=0x7f7f4c15bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:44:50.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.658+0000 7f7f43fff640 1 -- 172.21.15.98:0/3544976308 >> v2:172.21.15.98:3300/0 conn(0x7f7f4c157b60 msgr2=0x7f7f4c15bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:44:50.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.658+0000 7f7f43fff640 1 --2- 172.21.15.98:0/3544976308 >> v2:172.21.15.98:3300/0 conn(0x7f7f4c157b60 0x7f7f4c15bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:44:50.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.658+0000 7f7f43fff640 1 --2- 172.21.15.98:0/3544976308 >> v2:172.21.15.98:3300/0 conn(0x7f7f4c157b60 0x7f7f4c15bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:44:50.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.658+0000 7f7f4ad9d640 1 --2- 172.21.15.98:0/3544976308 >> v2:172.21.15.113:3300/0 conn(0x7f7f4c0c4910 0x7f7f4c0c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:50.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.658+0000 7f7f4ad9d640 1 -- 172.21.15.98:0/3544976308 >> v2:172.21.15.113:3300/0 conn(0x7f7f4c0c4910 msgr2=0x7f7f4c0c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:44:50.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.658+0000 7f7f4ad9d640 1 -- 172.21.15.98:0/3544976308 >> v2:172.21.15.113:3300/0 conn(0x7f7f4c0c4910 msgr2=0x7f7f4c0c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:44:50.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.658+0000 7f7f4ad9d640 1 --2- 172.21.15.98:0/3544976308 >> v2:172.21.15.113:3300/0 conn(0x7f7f4c0c4910 0x7f7f4c0c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:44:50.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.658+0000 7f7f4ad9d640 1 --2- 172.21.15.98:0/3544976308 >> v2:172.21.15.113:3300/0 conn(0x7f7f4c0c4910 0x7f7f4c0c4d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:44:50.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.858+0000 7f7f49d9b640 1 -- 172.21.15.98:0/3544976308 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408486703 0 0) 0x7f7f4c15b150 con 0x7f7f4c151d80 2023-11-28T17:44:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.859+0000 7f7f49d9b640 1 -- 172.21.15.98:0/3544976308 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f38003520 con 0x7f7f4c151d80 2023-11-28T17:44:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.859+0000 7f7f43fff640 1 --2- 172.21.15.98:0/3544976308 >> v2:172.21.15.98:3300/0 conn(0x7f7f4c157b60 0x7f7f4c15bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.859+0000 7f7f4ad9d640 1 --2- 172.21.15.98:0/3544976308 >> v2:172.21.15.113:3300/0 conn(0x7f7f4c0c4910 0x7f7f4c0c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.860+0000 7f7f49d9b640 1 -- 172.21.15.98:0/3544976308 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4187515774 0 0) 0x7f7f38003520 con 0x7f7f4c151d80 2023-11-28T17:44:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.860+0000 7f7f49d9b640 1 -- 172.21.15.98:0/3544976308 >> v2:172.21.15.113:3300/0 conn(0x7f7f4c0c4910 msgr2=0x7f7f4c0c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:44:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.860+0000 7f7f49d9b640 1 --2- 172.21.15.98:0/3544976308 >> v2:172.21.15.113:3300/0 conn(0x7f7f4c0c4910 0x7f7f4c0c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.860+0000 7f7f49d9b640 1 -- 172.21.15.98:0/3544976308 >> v2:172.21.15.98:3300/0 conn(0x7f7f4c157b60 msgr2=0x7f7f4c15bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:44:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.860+0000 7f7f49d9b640 1 --2- 172.21.15.98:0/3544976308 >> v2:172.21.15.98:3300/0 conn(0x7f7f4c157b60 0x7f7f4c15bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.860+0000 7f7f49d9b640 1 -- 172.21.15.98:0/3544976308 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c06e1f0 con 0x7f7f4c151d80 2023-11-28T17:44:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.860+0000 7f7f49d9b640 1 -- 172.21.15.98:0/3544976308 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f7f3c001670 con 0x7f7f4c151d80 2023-11-28T17:44:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.860+0000 7f7f49d9b640 1 -- 172.21.15.98:0/3544976308 >> v1:172.21.15.98:6789/0 conn(0x7f7f4c151d80 legacy=0x7f7f4c0c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.860+0000 7f7f49d9b640 1 --2- 172.21.15.98:0/3544976308 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7f380039c0 0x7f7f38003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.860+0000 7f7f49d9b640 1 --2- 172.21.15.98:0/3544976308 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f380047c0 0x7f7f38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.860+0000 7f7f49d9b640 1 --2- 172.21.15.98:0/3544976308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f38009970 0x7f7f38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.860+0000 7f7f49d9b640 1 -- 172.21.15.98:0/3544976308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c0c5260 con 0x7f7f38009970 2023-11-28T17:44:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.860+0000 7f7f49d9b640 1 -- 172.21.15.98:0/3544976308 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f380146f0 con 0x7f7f380047c0 2023-11-28T17:44:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.860+0000 7f7f49d9b640 1 -- 172.21.15.98:0/3544976308 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f380148b0 con 0x7f7f380039c0 2023-11-28T17:44:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.861+0000 7f7f43fff640 1 --2- 172.21.15.98:0/3544976308 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f380047c0 0x7f7f38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.861+0000 7f7f4a59c640 1 --2- 172.21.15.98:0/3544976308 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7f380039c0 0x7f7f38003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.861+0000 7f7f4ad9d640 1 --2- 172.21.15.98:0/3544976308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f38009970 0x7f7f38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.862+0000 7f7f43fff640 1 -- 172.21.15.98:0/3544976308 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7f380039c0 msgr2=0x7f7f38003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:44:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.862+0000 7f7f43fff640 1 --2- 172.21.15.98:0/3544976308 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7f380039c0 0x7f7f38003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.862+0000 7f7f43fff640 1 -- 172.21.15.98:0/3544976308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f38009970 msgr2=0x7f7f38005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:44:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.862+0000 7f7f43fff640 1 --2- 172.21.15.98:0/3544976308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f38009970 0x7f7f38005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.862+0000 7f7f43fff640 1 -- 172.21.15.98:0/3544976308 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f38003520 con 0x7f7f380047c0 2023-11-28T17:44:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.862+0000 7f7f4ad9d640 1 --2- 172.21.15.98:0/3544976308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f38009970 0x7f7f38005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:44:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.862+0000 7f7f4ad9d640 1 -- 172.21.15.98:0/3544976308 reap_dead start 2023-11-28T17:44:50.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.864+0000 7f7f43fff640 1 --2- 172.21.15.98:0/3544976308 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f380047c0 0x7f7f38004c40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7f34009a00 tx=0x7f7f380057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ab0bad926b5b997 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:50.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.865+0000 7f7f49d9b640 1 -- 172.21.15.98:0/3544976308 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3402cc60 con 0x7f7f380047c0 2023-11-28T17:44:50.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.865+0000 7f7f49d9b640 1 -- 172.21.15.98:0/3544976308 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3402cdc0 con 0x7f7f380047c0 2023-11-28T17:44:50.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.865+0000 7f7f49d9b640 1 -- 172.21.15.98:0/3544976308 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f34034660 con 0x7f7f380047c0 2023-11-28T17:44:50.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.866+0000 7f7f5086b640 1 -- 172.21.15.98:0/3544976308 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f380047c0 msgr2=0x7f7f38004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:50.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.866+0000 7f7f5086b640 1 --2- 172.21.15.98:0/3544976308 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f380047c0 0x7f7f38004c40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7f34009a00 tx=0x7f7f380057f0 comp rx=0 tx=0).stop 2023-11-28T17:44:50.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.866+0000 7f7f5086b640 1 -- 172.21.15.98:0/3544976308 shutdown_connections 2023-11-28T17:44:50.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.866+0000 7f7f5086b640 1 --2- 172.21.15.98:0/3544976308 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f380047c0 0x7f7f38004c40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:50.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.866+0000 7f7f5086b640 1 -- 172.21.15.98:0/3544976308 >> 172.21.15.98:0/3544976308 conn(0x7f7f4c06a4d0 msgr2=0x7f7f4c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:50.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.867+0000 7f7f5086b640 1 -- 172.21.15.98:0/3544976308 shutdown_connections 2023-11-28T17:44:50.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.867+0000 7f7f5086b640 1 -- 172.21.15.98:0/3544976308 wait complete. 2023-11-28T17:44:50.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.870+0000 7f7f5086b640 1 Processor -- start 2023-11-28T17:44:50.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.870+0000 7f7f5086b640 1 -- start start 2023-11-28T17:44:50.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.872+0000 7f7f5086b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f380047c0 0x7f7f4c0c3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:50.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.872+0000 7f7f5086b640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7f4c146630 0x7f7f4c155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:50.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.872+0000 7f7f5086b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f4c0717c0 0x7f7f4c156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:50.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.872+0000 7f7f5086b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c0c2d30 con 0x7f7f4c0717c0 2023-11-28T17:44:50.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.872+0000 7f7f5086b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c1eb1f0 con 0x7f7f380047c0 2023-11-28T17:44:50.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.872+0000 7f7f5086b640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c1eb4b0 con 0x7f7f4c146630 2023-11-28T17:44:50.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.872+0000 7f7f4ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f4c0717c0 0x7f7f4c156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:50.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.872+0000 7f7f4ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f4c0717c0 0x7f7f4c156110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55266/0 (socket says 172.21.15.98:55266) 2023-11-28T17:44:50.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.872+0000 7f7f4ad9d640 1 -- 172.21.15.98:0/2819292730 learned_addr learned my addr 172.21.15.98:0/2819292730 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:44:50.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.873+0000 7f7f4a59c640 1 --2- 172.21.15.98:0/2819292730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f380047c0 0x7f7f4c0c3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:50.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.873+0000 7f7f43fff640 1 --2- 172.21.15.98:0/2819292730 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7f4c146630 0x7f7f4c155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:50.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.873+0000 7f7f4ad9d640 1 -- 172.21.15.98:0/2819292730 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7f4c146630 msgr2=0x7f7f4c155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:50.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.873+0000 7f7f4ad9d640 1 --2- 172.21.15.98:0/2819292730 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7f4c146630 0x7f7f4c155bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:50.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.873+0000 7f7f4ad9d640 1 -- 172.21.15.98:0/2819292730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f380047c0 msgr2=0x7f7f4c0c3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:50.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.873+0000 7f7f4ad9d640 1 --2- 172.21.15.98:0/2819292730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f380047c0 0x7f7f4c0c3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:50.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.873+0000 7f7f4ad9d640 1 -- 172.21.15.98:0/2819292730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c1eb790 con 0x7f7f4c0717c0 2023-11-28T17:44:50.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.873+0000 7f7f4a59c640 1 --2- 172.21.15.98:0/2819292730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f380047c0 0x7f7f4c0c3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:44:50.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.873+0000 7f7f4ad9d640 1 --2- 172.21.15.98:0/2819292730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f4c0717c0 0x7f7f4c156110 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0c1c70 tx=0x7f7f44004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:50.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.874+0000 7f7f42ffd640 1 -- 172.21.15.98:0/2819292730 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4400a980 con 0x7f7f4c0717c0 2023-11-28T17:44:50.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.874+0000 7f7f42ffd640 1 -- 172.21.15.98:0/2819292730 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f44016460 con 0x7f7f4c0717c0 2023-11-28T17:44:50.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.874+0000 7f7f42ffd640 1 -- 172.21.15.98:0/2819292730 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f44016810 con 0x7f7f4c0717c0 2023-11-28T17:44:50.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.874+0000 7f7f5086b640 1 -- 172.21.15.98:0/2819292730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4c1eba70 con 0x7f7f4c0717c0 2023-11-28T17:44:50.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.874+0000 7f7f5086b640 1 -- 172.21.15.98:0/2819292730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4c0c3dc0 con 0x7f7f4c0717c0 2023-11-28T17:44:50.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.875+0000 7f7f5086b640 1 -- 172.21.15.98:0/2819292730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4c067bf0 con 0x7f7f4c0717c0 2023-11-28T17:44:50.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.876+0000 7f7f42ffd640 1 -- 172.21.15.98:0/2819292730 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f7f44016970 con 0x7f7f4c0717c0 2023-11-28T17:44:50.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.877+0000 7f7f42ffd640 1 --2- 172.21.15.98:0/2819292730 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7f2c06b2a0 0x7f7f2c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:50.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.878+0000 7f7f42ffd640 1 -- 172.21.15.98:0/2819292730 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7f4408aa30 con 0x7f7f4c0717c0 2023-11-28T17:44:50.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.878+0000 7f7f4a59c640 1 --2- 172.21.15.98:0/2819292730 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7f2c06b2a0 0x7f7f2c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:50.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.879+0000 7f7f4a59c640 1 --2- 172.21.15.98:0/2819292730 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7f2c06b2a0 0x7f7f2c06d760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c000fb0 tx=0x7f7f3c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:50.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:50.883+0000 7f7f42ffd640 1 -- 172.21.15.98:0/2819292730 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f44055780 con 0x7f7f4c0717c0 2023-11-28T17:44:51.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.148+0000 7f7f5086b640 1 -- 172.21.15.98:0/2819292730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7f4c1559d0 con 0x7f7f4c0717c0 2023-11-28T17:44:51.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.149+0000 7f7f42ffd640 1 -- 172.21.15.98:0/2819292730 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7f7f44058da0 con 0x7f7f4c0717c0 2023-11-28T17:44:51.150 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:44:51.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.151+0000 7f7f5086b640 1 -- 172.21.15.98:0/2819292730 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7f2c06b2a0 msgr2=0x7f7f2c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.151+0000 7f7f5086b640 1 --2- 172.21.15.98:0/2819292730 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7f2c06b2a0 0x7f7f2c06d760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c000fb0 tx=0x7f7f3c009040 comp rx=0 tx=0).stop 2023-11-28T17:44:51.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.151+0000 7f7f5086b640 1 -- 172.21.15.98:0/2819292730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f4c0717c0 msgr2=0x7f7f4c156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.151+0000 7f7f5086b640 1 --2- 172.21.15.98:0/2819292730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f4c0717c0 0x7f7f4c156110 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0c1c70 tx=0x7f7f44004700 comp rx=0 tx=0).stop 2023-11-28T17:44:51.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.151+0000 7f7f5086b640 1 -- 172.21.15.98:0/2819292730 shutdown_connections 2023-11-28T17:44:51.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.151+0000 7f7f5086b640 1 --2- 172.21.15.98:0/2819292730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f4c0717c0 0x7f7f4c156110 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.151+0000 7f7f5086b640 1 --2- 172.21.15.98:0/2819292730 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7f4c146630 0x7f7f4c155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.151+0000 7f7f5086b640 1 --2- 172.21.15.98:0/2819292730 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7f2c06b2a0 0x7f7f2c06d760 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.151+0000 7f7f5086b640 1 --2- 172.21.15.98:0/2819292730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f380047c0 0x7f7f4c0c3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.151+0000 7f7f5086b640 1 -- 172.21.15.98:0/2819292730 >> 172.21.15.98:0/2819292730 conn(0x7f7f4c06a4d0 msgr2=0x7f7f4c157f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:51.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.151+0000 7f7f5086b640 1 -- 172.21.15.98:0/2819292730 shutdown_connections 2023-11-28T17:44:51.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.152+0000 7f7f5086b640 1 -- 172.21.15.98:0/2819292730 wait complete. 2023-11-28T17:44:51.170 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:44:51.171 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:44:51.171 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:44:51.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.380+0000 7f4224e52640 1 Processor -- start 2023-11-28T17:44:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.380+0000 7f4224e52640 1 -- start start 2023-11-28T17:44:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.381+0000 7f4224e52640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4220153c80 0x7f422006b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.381+0000 7f4224e52640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422015d230 con 0x7f42200c69b0 2023-11-28T17:44:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.381+0000 7f4224e52640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422015d4f0 con 0x7f42200c7e30 2023-11-28T17:44:51.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.381+0000 7f4224e52640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422015e070 con 0x7f4220153c80 2023-11-28T17:44:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.381+0000 7f4216ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f42200c69b0 0x7f42200c7720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50850/0 (socket says 172.21.15.98:50850) 2023-11-28T17:44:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.381+0000 7f4216ffd640 1 -- 172.21.15.98:0/1880377561 learned_addr learned my addr 172.21.15.98:0/1880377561 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:44:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421f7fe640 1 --2- 172.21.15.98:0/1880377561 >> v2:172.21.15.113:3300/0 conn(0x7f4220153c80 0x7f422006b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421effd640 1 -- 172.21.15.98:0/1880377561 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471362741 0 0) 0x7f422015d230 con 0x7f42200c69b0 2023-11-28T17:44:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421effd640 1 -- 172.21.15.98:0/1880377561 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f420c003520 con 0x7f42200c69b0 2023-11-28T17:44:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421effd640 1 -- 172.21.15.98:0/1880377561 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3836633605 0 0) 0x7f420c003520 con 0x7f42200c69b0 2023-11-28T17:44:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421effd640 1 -- 172.21.15.98:0/1880377561 >> v2:172.21.15.113:3300/0 conn(0x7f4220153c80 msgr2=0x7f422006b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:44:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421effd640 1 --2- 172.21.15.98:0/1880377561 >> v2:172.21.15.113:3300/0 conn(0x7f4220153c80 0x7f422006b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421effd640 1 -- 172.21.15.98:0/1880377561 >> v1:172.21.15.113:6789/0 conn(0x7f42200c7e30 legacy=0x7f4220164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421effd640 1 -- 172.21.15.98:0/1880377561 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422015d7d0 con 0x7f42200c69b0 2023-11-28T17:44:51.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421effd640 1 -- 172.21.15.98:0/1880377561 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f4208001670 con 0x7f42200c69b0 2023-11-28T17:44:51.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421effd640 1 -- 172.21.15.98:0/1880377561 >> v1:172.21.15.98:6789/0 conn(0x7f42200c69b0 legacy=0x7f42200c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421effd640 1 --2- 172.21.15.98:0/1880377561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f420c003970 0x7f420c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421effd640 1 --2- 172.21.15.98:0/1880377561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f420c0047c0 0x7f420c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421effd640 1 --2- 172.21.15.98:0/1880377561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f420c005180 0x7f420c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421effd640 1 -- 172.21.15.98:0/1880377561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422015e070 con 0x7f420c005180 2023-11-28T17:44:51.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421effd640 1 -- 172.21.15.98:0/1880377561 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c00c0b0 con 0x7f420c0047c0 2023-11-28T17:44:51.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421effd640 1 -- 172.21.15.98:0/1880377561 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420c00c270 con 0x7f420c003970 2023-11-28T17:44:51.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421f7fe640 1 --2- 172.21.15.98:0/1880377561 >> v2:172.21.15.113:3300/0 conn(0x7f4220153c80 0x7f422006b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:44:51.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.382+0000 7f421ffff640 1 --2- 172.21.15.98:0/1880377561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f420c005180 0x7f420c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.383+0000 7f4216ffd640 1 --2- 172.21.15.98:0/1880377561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f420c0047c0 0x7f420c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.383+0000 7f421f7fe640 1 --2- 172.21.15.98:0/1880377561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f420c003970 0x7f420c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.383+0000 7f421ffff640 1 -- 172.21.15.98:0/1880377561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f420c003970 msgr2=0x7f420c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:44:51.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.383+0000 7f421ffff640 1 --2- 172.21.15.98:0/1880377561 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f420c003970 0x7f420c003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.383+0000 7f421ffff640 1 -- 172.21.15.98:0/1880377561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f420c0047c0 msgr2=0x7f420c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:44:51.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.383+0000 7f421ffff640 1 --2- 172.21.15.98:0/1880377561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f420c0047c0 0x7f420c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.383+0000 7f421ffff640 1 -- 172.21.15.98:0/1880377561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c003520 con 0x7f420c005180 2023-11-28T17:44:51.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.384+0000 7f421ffff640 1 -- 172.21.15.98:0/1880377561 reap_dead start 2023-11-28T17:44:51.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.384+0000 7f421ffff640 1 --2- 172.21.15.98:0/1880377561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f420c005180 0x7f420c00b980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4218001750 tx=0x7f420c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a8c8318f3e88eb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:51.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.384+0000 7f421effd640 1 -- 172.21.15.98:0/1880377561 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42180052d0 con 0x7f420c005180 2023-11-28T17:44:51.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.384+0000 7f421effd640 1 -- 172.21.15.98:0/1880377561 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421802e050 con 0x7f420c005180 2023-11-28T17:44:51.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.384+0000 7f421effd640 1 -- 172.21.15.98:0/1880377561 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4218003830 con 0x7f420c005180 2023-11-28T17:44:51.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.385+0000 7f4224e52640 1 -- 172.21.15.98:0/1880377561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f420c005180 msgr2=0x7f420c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.385+0000 7f4224e52640 1 --2- 172.21.15.98:0/1880377561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f420c005180 0x7f420c00b980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4218001750 tx=0x7f420c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.385+0000 7f4224e52640 1 -- 172.21.15.98:0/1880377561 shutdown_connections 2023-11-28T17:44:51.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.385+0000 7f4224e52640 1 --2- 172.21.15.98:0/1880377561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f420c005180 0x7f420c00b980 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.385+0000 7f4224e52640 1 -- 172.21.15.98:0/1880377561 >> 172.21.15.98:0/1880377561 conn(0x7f42200cf300 msgr2=0x7f42200cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:51.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.385+0000 7f4224e52640 1 -- 172.21.15.98:0/1880377561 shutdown_connections 2023-11-28T17:44:51.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.385+0000 7f4224e52640 1 -- 172.21.15.98:0/1880377561 wait complete. 2023-11-28T17:44:51.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.387+0000 7f4224e52640 1 Processor -- start 2023-11-28T17:44:51.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.387+0000 7f4224e52640 1 -- start start 2023-11-28T17:44:51.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f4224e52640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f420c005180 0x7f42200cf950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f4224e52640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4220156e10 0x7f42201ef760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f4224e52640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42200c2f90 0x7f42201efca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f4224e52640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4220156810 con 0x7f42200c2f90 2023-11-28T17:44:51.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f4224e52640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4220154b20 con 0x7f4220156e10 2023-11-28T17:44:51.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f4224e52640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4220154580 con 0x7f420c005180 2023-11-28T17:44:51.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f421ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42200c2f90 0x7f42201efca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f421ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42200c2f90 0x7f42201efca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55270/0 (socket says 172.21.15.98:55270) 2023-11-28T17:44:51.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f421ffff640 1 -- 172.21.15.98:0/1156373755 learned_addr learned my addr 172.21.15.98:0/1156373755 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:44:51.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f4216ffd640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4220156e10 0x7f42201ef760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f421f7fe640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f420c005180 0x7f42200cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f421ffff640 1 -- 172.21.15.98:0/1156373755 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f420c005180 msgr2=0x7f42200cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f421ffff640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f420c005180 0x7f42200cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f421ffff640 1 -- 172.21.15.98:0/1156373755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4220156e10 msgr2=0x7f42201ef760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f421ffff640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4220156e10 0x7f42201ef760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.388+0000 7f421ffff640 1 -- 172.21.15.98:0/1156373755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42201561e0 con 0x7f42200c2f90 2023-11-28T17:44:51.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.389+0000 7f421ffff640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42200c2f90 0x7f42201efca0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4218001870 tx=0x7f4218001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:51.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.389+0000 7f421cff9640 1 -- 172.21.15.98:0/1156373755 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4218039070 con 0x7f42200c2f90 2023-11-28T17:44:51.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.389+0000 7f421cff9640 1 -- 172.21.15.98:0/1156373755 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4218007a30 con 0x7f42200c2f90 2023-11-28T17:44:51.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.389+0000 7f4224e52640 1 -- 172.21.15.98:0/1156373755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4220154840 con 0x7f42200c2f90 2023-11-28T17:44:51.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.389+0000 7f421cff9640 1 -- 172.21.15.98:0/1156373755 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42180054e0 con 0x7f42200c2f90 2023-11-28T17:44:51.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.389+0000 7f4224e52640 1 -- 172.21.15.98:0/1156373755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42201558d0 con 0x7f42200c2f90 2023-11-28T17:44:51.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.390+0000 7f4224e52640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42201f2b60 0x7f42201f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.390+0000 7f4224e52640 1 -- 172.21.15.98:0/1156373755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4220068170 con 0x7f42201f2b60 2023-11-28T17:44:51.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.391+0000 7f421f7fe640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42201f2b60 0x7f42201f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.391+0000 7f421f7fe640 1 -- 172.21.15.98:0/1156373755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42201503f0 con 0x7f42201f2b60 2023-11-28T17:44:51.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.391+0000 7f421f7fe640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42201f2b60 0x7f42201f2f40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4210007c70 tx=0x7f421000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:51.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.392+0000 7f421cff9640 1 -- 172.21.15.98:0/1156373755 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421000e040 con 0x7f42201f2b60 2023-11-28T17:44:51.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.392+0000 7f421cff9640 1 -- 172.21.15.98:0/1156373755 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f421803c020 con 0x7f42200c2f90 2023-11-28T17:44:51.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.393+0000 7f421cff9640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f420006b340 0x7f420006d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.393+0000 7f421cff9640 1 -- 172.21.15.98:0/1156373755 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f42180aa090 con 0x7f42200c2f90 2023-11-28T17:44:51.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.393+0000 7f4216ffd640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f420006b340 0x7f420006d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.393+0000 7f421cff9640 1 -- 172.21.15.98:0/1156373755 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f4210007e80 con 0x7f42201f2b60 2023-11-28T17:44:51.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.393+0000 7f421cff9640 1 -- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42201f2b60 msgr2=0x7f42201f2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.393+0000 7f421cff9640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42201f2b60 0x7f42201f2f40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4210007c70 tx=0x7f421000f580 comp rx=0 tx=0).stop 2023-11-28T17:44:51.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.393+0000 7f421cff9640 1 -- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42201f2b60 msgr2=0x7f42201f2f40 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:44:51.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.393+0000 7f421cff9640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42201f2b60 0x7f42201f2f40 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4210007c70 tx=0x7f421000f580 comp rx=0 tx=0).stop 2023-11-28T17:44:51.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.394+0000 7f4216ffd640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f420006b340 0x7f420006d800 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4208000f60 tx=0x7f4208000ef0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.400+0000 7f4224e52640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42201f3480 0x7f422014b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.400+0000 7f4224e52640 1 -- 172.21.15.98:0/1156373755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42200c9eb0 con 0x7f42201f3480 2023-11-28T17:44:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.400+0000 7f421f7fe640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42201f3480 0x7f422014b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.401+0000 7f421f7fe640 1 -- 172.21.15.98:0/1156373755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f42200cbb10 con 0x7f42201f3480 2023-11-28T17:44:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.401+0000 7f421f7fe640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42201f3480 0x7f422014b080 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4210007c70 tx=0x7f4210004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.401+0000 7f421cff9640 1 -- 172.21.15.98:0/1156373755 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42100209c0 con 0x7f42201f3480 2023-11-28T17:44:51.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.402+0000 7f421cff9640 1 -- 172.21.15.98:0/1156373755 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4467 (secure 0 0 0) 0x7f4210024070 con 0x7f42201f3480 2023-11-28T17:44:51.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.402+0000 7f421cff9640 1 -- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42201f3480 msgr2=0x7f422014b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.402+0000 7f421cff9640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42201f3480 0x7f422014b080 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4210007c70 tx=0x7f4210004ab0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.402+0000 7f421cff9640 1 -- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42201f3480 msgr2=0x7f422014b080 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:44:51.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.402+0000 7f421cff9640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42201f3480 0x7f422014b080 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4210007c70 tx=0x7f4210004ab0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.403 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:44:51.403 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:44:51.403 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:44:51.404 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:44:51.404 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 36, 2023-11-28T17:44:51.404 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:44:51.404 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:44:51.404 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:44:51.404 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:44:51.404 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:51.404 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:44:51.404 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:44:51.404 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:44:51.404 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:44:51.405 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:44:51.405 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:44:51.405 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:44:51.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:44:51.405 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:44:51.405 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:44:51.405 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:44:51.405 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:44:51.405 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:44:51.406 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:44:51.406 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:51.406 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:44:51.406 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:44:51.406 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:44:51.406 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:44:51.406 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:44:51.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:44:51.406 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:44:51.406 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:44:51.406 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:44:51.407 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:44:51.407 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:44:51.407 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:44:51.407 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.407 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:44:51.407 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:44:51.407 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:44:51.407 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:44:51.407 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:44:51.407 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:44:51.407 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:44:51.408 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:44:51.408 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:44:51.408 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:44:51.408 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:44:51.408 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:44:51.408 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:44:51.408 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:44:51.408 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:44:51.408 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:44:51.408 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:44:51.408 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:44:51.409 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:44:51.409 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:44:51.409 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:44:51.409 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:44:51.409 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:44:51.409 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:44:51.409 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:44:51.409 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:44:51.409 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:44:51.409 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:51.409 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:44:51.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.410 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:44:51.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.410 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:44:51.410 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:44:51.410 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:44:51.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:44:51.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:44:51.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:44:51.411 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:44:51.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:44:51.412 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.412 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.412 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:44:51.413 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:44:51.413 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:44:51.413 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:44:51.413 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:44:51.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.413 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:44:51.414 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:44:51.414 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:44:51.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:44:51.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:44:51.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:44:51.415 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:44:51.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:44:51.415 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.415 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.416 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:44:51.416 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:44:51.416 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:44:51.416 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:44:51.416 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:44:51.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.417 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:44:51.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:44:51.417 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:44:51.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:44:51.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:44:51.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:44:51.418 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:44:51.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:44:51.421 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.421 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.422 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:44:51.422 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:44:51.422 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:44:51.422 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:44:51.422 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:44:51.422 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.422 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.423 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:44:51.423 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:44:51.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.423 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:51.423 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:51.423 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:44:51.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.423 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:51.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:44:51.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.424 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:51.424 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:51.424 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:44:51.424 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.424 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.425 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:44:51.425 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:44:51.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.402+0000 7f4224e52640 1 -- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f420006b340 msgr2=0x7f420006d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.402+0000 7f4224e52640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f420006b340 0x7f420006d800 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4208000f60 tx=0x7f4208000ef0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.402+0000 7f4224e52640 1 -- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42200c2f90 msgr2=0x7f42201efca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.402+0000 7f4224e52640 1 --2- 172.21.15.98:0/1156373755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42200c2f90 0x7f42201efca0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4218001870 tx=0x7f4218001120 comp rx=0 tx=0).stop 2023-11-28T17:44:51.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.402+0000 7f421ffff640 1 -- 172.21.15.98:0/1156373755 reap_dead start 2023-11-28T17:44:51.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.402+0000 7f421ffff640 1 -- 172.21.15.98:0/1156373755 reap_dead start 2023-11-28T17:44:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.402+0000 7f4224e52640 1 -- 172.21.15.98:0/1156373755 shutdown_connections 2023-11-28T17:44:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.402+0000 7f4224e52640 1 -- 172.21.15.98:0/1156373755 >> 172.21.15.98:0/1156373755 conn(0x7f42200cf300 msgr2=0x7f422014fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.402+0000 7f4224e52640 1 -- 172.21.15.98:0/1156373755 shutdown_connections 2023-11-28T17:44:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.402+0000 7f4224e52640 1 -- 172.21.15.98:0/1156373755 wait complete. 2023-11-28T17:44:51.428 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:44:51.428 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:44:51.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:44:51.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.638+0000 7f01f32d3640 1 Processor -- start 2023-11-28T17:44:51.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.638+0000 7f01f32d3640 1 -- start start 2023-11-28T17:44:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.640+0000 7f01f32d3640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f01ec1571d0 0x7f01ec06c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.640+0000 7f01f32d3640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec070790 0x7f01ec0c0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.640+0000 7f01f32d3640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ec06b860 con 0x7f01ec159cb0 2023-11-28T17:44:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.640+0000 7f01f32d3640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ec0c1e80 con 0x7f01ec1571d0 2023-11-28T17:44:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.640+0000 7f01f32d3640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec06c7f0 con 0x7f01ec070790 2023-11-28T17:44:51.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.641+0000 7f01f106f640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f01ec1571d0 0x7f01ec06c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.641+0000 7f01f086e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec070790 0x7f01ec0c0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.641+0000 7f01f106f640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f01ec1571d0 0x7f01ec06c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:47460/0 (socket says 172.21.15.98:47460) 2023-11-28T17:44:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.641+0000 7f01f106f640 1 -- 172.21.15.98:0/1867017074 learned_addr learned my addr 172.21.15.98:0/1867017074 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:44:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.641+0000 7f01f086e640 1 -- 172.21.15.98:0/1867017074 >> v2:172.21.15.113:3300/0 conn(0x7f01ec1571d0 msgr2=0x7f01ec06c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:44:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.641+0000 7f01f086e640 1 --2- 172.21.15.98:0/1867017074 >> v2:172.21.15.113:3300/0 conn(0x7f01ec1571d0 0x7f01ec06c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.641+0000 7f01f086e640 1 -- 172.21.15.98:0/1867017074 >> v1:172.21.15.113:6789/0 conn(0x7f01ec159cb0 legacy=0x7f01ec0caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.641+0000 7f01f086e640 1 -- 172.21.15.98:0/1867017074 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec0c05b0 con 0x7f01ec070790 2023-11-28T17:44:51.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.641+0000 7f01f106f640 1 --2- 172.21.15.98:0/1867017074 >> v2:172.21.15.113:3300/0 conn(0x7f01ec1571d0 0x7f01ec06c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:44:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.643+0000 7f01f086e640 1 --2- 172.21.15.98:0/1867017074 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec070790 0x7f01ec0c0070 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f01e00097a0 tx=0x7f01e002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a727e597bc21837 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.643+0000 7f01ebfff640 1 -- 172.21.15.98:0/1867017074 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e0030800 con 0x7f01ec070790 2023-11-28T17:44:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.643+0000 7f01ebfff640 1 -- 172.21.15.98:0/1867017074 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e0030960 con 0x7f01ec070790 2023-11-28T17:44:51.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.643+0000 7f01ebfff640 1 -- 172.21.15.98:0/1867017074 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e0030cf0 con 0x7f01ec070790 2023-11-28T17:44:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.644+0000 7f01f32d3640 1 -- 172.21.15.98:0/1867017074 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec070790 msgr2=0x7f01ec0c0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.644+0000 7f01f32d3640 1 --2- 172.21.15.98:0/1867017074 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec070790 0x7f01ec0c0070 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f01e00097a0 tx=0x7f01e002b940 comp rx=0 tx=0).stop 2023-11-28T17:44:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.644+0000 7f01f32d3640 1 -- 172.21.15.98:0/1867017074 shutdown_connections 2023-11-28T17:44:51.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.644+0000 7f01f32d3640 1 --2- 172.21.15.98:0/1867017074 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec070790 0x7f01ec0c0070 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.644+0000 7f01f32d3640 1 --2- 172.21.15.98:0/1867017074 >> v2:172.21.15.113:3300/0 conn(0x7f01ec1571d0 0x7f01ec06c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.644+0000 7f01f32d3640 1 -- 172.21.15.98:0/1867017074 >> 172.21.15.98:0/1867017074 conn(0x7f01ec06a4d0 msgr2=0x7f01ec06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.644+0000 7f01f32d3640 1 -- 172.21.15.98:0/1867017074 shutdown_connections 2023-11-28T17:44:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.644+0000 7f01f32d3640 1 -- 172.21.15.98:0/1867017074 wait complete. 2023-11-28T17:44:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.646+0000 7f01f32d3640 1 Processor -- start 2023-11-28T17:44:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.646+0000 7f01f32d3640 1 -- start start 2023-11-28T17:44:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f32d3640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec070790 0x7f01ec0ae4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f32d3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ec1571d0 0x7f01ec14e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f32d3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec159cb0 0x7f01ec071e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f32d3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec1efa50 con 0x7f01ec1571d0 2023-11-28T17:44:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f32d3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec1ef180 con 0x7f01ec159cb0 2023-11-28T17:44:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f32d3640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec1e0dc0 con 0x7f01ec070790 2023-11-28T17:44:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f086e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ec1571d0 0x7f01ec14e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f086e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ec1571d0 0x7f01ec14e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55302/0 (socket says 172.21.15.98:55302) 2023-11-28T17:44:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f086e640 1 -- 172.21.15.98:0/3263591216 learned_addr learned my addr 172.21.15.98:0/3263591216 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:44:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f106f640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec070790 0x7f01ec0ae4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f1870640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec159cb0 0x7f01ec071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f086e640 1 -- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec070790 msgr2=0x7f01ec0ae4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f086e640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01ec070790 0x7f01ec0ae4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f086e640 1 -- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec159cb0 msgr2=0x7f01ec071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f086e640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec159cb0 0x7f01ec071e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.648+0000 7f01f086e640 1 -- 172.21.15.98:0/3263591216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01dc009b30 con 0x7f01ec1571d0 2023-11-28T17:44:51.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.649+0000 7f01f086e640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ec1571d0 0x7f01ec14e990 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f01e00098c0 tx=0x7f01e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:51.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.649+0000 7f01e9ffb640 1 -- 172.21.15.98:0/3263591216 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e003e040 con 0x7f01ec1571d0 2023-11-28T17:44:51.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.649+0000 7f01e9ffb640 1 -- 172.21.15.98:0/3263591216 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e0039c90 con 0x7f01ec1571d0 2023-11-28T17:44:51.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.649+0000 7f01e9ffb640 1 -- 172.21.15.98:0/3263591216 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e00443f0 con 0x7f01ec1571d0 2023-11-28T17:44:51.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.649+0000 7f01f32d3640 1 -- 172.21.15.98:0/3263591216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e0009400 con 0x7f01ec1571d0 2023-11-28T17:44:51.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.649+0000 7f01f32d3640 1 -- 172.21.15.98:0/3263591216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ec0ae210 con 0x7f01ec1571d0 2023-11-28T17:44:51.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.650+0000 7f01f32d3640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec1507b0 0x7f01ec1ef3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.650+0000 7f01f32d3640 1 -- 172.21.15.98:0/3263591216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec0743f0 con 0x7f01ec1507b0 2023-11-28T17:44:51.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.651+0000 7f01f106f640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec1507b0 0x7f01ec1ef3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.651+0000 7f01e9ffb640 1 -- 172.21.15.98:0/3263591216 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f01e0043390 con 0x7f01ec1571d0 2023-11-28T17:44:51.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.652+0000 7f01f106f640 1 -- 172.21.15.98:0/3263591216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f01ec06cd90 con 0x7f01ec1507b0 2023-11-28T17:44:51.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.652+0000 7f01f106f640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec1507b0 0x7f01ec1ef3f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f01dc007de0 tx=0x7f01dc010590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:51.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.652+0000 7f01e9ffb640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f01d806b2a0 0x7f01d806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.652+0000 7f01e9ffb640 1 -- 172.21.15.98:0/3263591216 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f01e00abfb0 con 0x7f01ec1571d0 2023-11-28T17:44:51.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.652+0000 7f01f1870640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f01d806b2a0 0x7f01d806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.652+0000 7f01e9ffb640 1 -- 172.21.15.98:0/3263591216 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc00e050 con 0x7f01ec1507b0 2023-11-28T17:44:51.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.653+0000 7f01f1870640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f01d806b2a0 0x7f01d806d760 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f01ec14e860 tx=0x7f01e4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:51.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.653+0000 7f01e9ffb640 1 -- 172.21.15.98:0/3263591216 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f01dc00f040 con 0x7f01ec1507b0 2023-11-28T17:44:51.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.653+0000 7f01e9ffb640 1 -- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec1507b0 msgr2=0x7f01ec1ef3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.653+0000 7f01e9ffb640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec1507b0 0x7f01ec1ef3f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f01dc007de0 tx=0x7f01dc010590 comp rx=0 tx=0).stop 2023-11-28T17:44:51.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.653+0000 7f01e9ffb640 1 -- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec1507b0 msgr2=0x7f01ec1ef3f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:44:51.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.653+0000 7f01e9ffb640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec1507b0 0x7f01ec1ef3f0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f01dc007de0 tx=0x7f01dc010590 comp rx=0 tx=0).stop 2023-11-28T17:44:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.660+0000 7f01f32d3640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec1efe70 0x7f01ec14d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.660+0000 7f01f32d3640 1 -- 172.21.15.98:0/3263591216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec15aab0 con 0x7f01ec1efe70 2023-11-28T17:44:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.661+0000 7f01f106f640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec1efe70 0x7f01ec14d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.662+0000 7f01f106f640 1 -- 172.21.15.98:0/3263591216 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f01ec15a7d0 con 0x7f01ec1efe70 2023-11-28T17:44:51.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.662+0000 7f01f106f640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec1efe70 0x7f01ec14d4a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f01dc007de0 tx=0x7f01dc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.663+0000 7f01e9ffb640 1 -- 172.21.15.98:0/3263591216 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc021320 con 0x7f01ec1efe70 2023-11-28T17:44:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.663+0000 7f01e9ffb640 1 -- 172.21.15.98:0/3263591216 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4465 (secure 0 0 0) 0x7f01dc015c50 con 0x7f01ec1efe70 2023-11-28T17:44:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.663+0000 7f01e9ffb640 1 -- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec1efe70 msgr2=0x7f01ec14d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.663+0000 7f01e9ffb640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec1efe70 0x7f01ec14d4a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f01dc007de0 tx=0x7f01dc004a40 comp rx=0 tx=0).stop 2023-11-28T17:44:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.663+0000 7f01e9ffb640 1 -- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec1efe70 msgr2=0x7f01ec14d4a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:44:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.663+0000 7f01e9ffb640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01ec1efe70 0x7f01ec14d4a0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f01dc007de0 tx=0x7f01dc004a40 comp rx=0 tx=0).stop 2023-11-28T17:44:51.665 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:44:51.665 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:44:51.665 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:44:51.665 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:44:51.665 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 36, 2023-11-28T17:44:51.665 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:44:51.665 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:44:51.665 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:44:51.666 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:44:51.666 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:51.666 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:44:51.666 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:44:51.666 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:44:51.666 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:44:51.666 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:44:51.666 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:44:51.666 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:44:51.666 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:44:51.666 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:44:51.667 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:44:51.667 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:44:51.667 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:44:51.667 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:44:51.667 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:44:51.667 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:51.667 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:44:51.667 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:44:51.667 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:44:51.667 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:44:51.667 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:44:51.668 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:44:51.668 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:44:51.668 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:44:51.668 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:44:51.668 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:44:51.668 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:44:51.668 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:44:51.668 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.668 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:44:51.668 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:44:51.668 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:44:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:44:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:44:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:44:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:44:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:44:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:44:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:44:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:44:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:44:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:44:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:44:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:44:51.669 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:44:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:44:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:44:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:44:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:44:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:44:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:44:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:44:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:44:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:44:51.670 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:44:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:44:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:44:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:44:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:44:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:44:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:44:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:44:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:44:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:44:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:44:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:44:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:44:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:44:51.672 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:44:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:44:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:44:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:44:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:44:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:44:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:44:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:44:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:44:51.673 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:44:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:44:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:44:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:44:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:44:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:44:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:44:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:44:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:44:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:44:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:44:51.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:44:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:44:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:44:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:44:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:44:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:44:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:44:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:44:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:44:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:44:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:44:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:51.677 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:44:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:44:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:44:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.678 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:44:51.679 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:44:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.664+0000 7f01f32d3640 1 -- 172.21.15.98:0/3263591216 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f01d806b2a0 msgr2=0x7f01d806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.664+0000 7f01f32d3640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f01d806b2a0 0x7f01d806d760 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f01ec14e860 tx=0x7f01e4009040 comp rx=0 tx=0).stop 2023-11-28T17:44:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.664+0000 7f01f32d3640 1 -- 172.21.15.98:0/3263591216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ec1571d0 msgr2=0x7f01ec14e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.664+0000 7f01f32d3640 1 --2- 172.21.15.98:0/3263591216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01ec1571d0 0x7f01ec14e990 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f01e00098c0 tx=0x7f01e0004730 comp rx=0 tx=0).stop 2023-11-28T17:44:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.664+0000 7f01f1870640 1 -- 172.21.15.98:0/3263591216 reap_dead start 2023-11-28T17:44:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.664+0000 7f01f1870640 1 -- 172.21.15.98:0/3263591216 reap_dead start 2023-11-28T17:44:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.664+0000 7f01f32d3640 1 -- 172.21.15.98:0/3263591216 shutdown_connections 2023-11-28T17:44:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.664+0000 7f01f32d3640 1 -- 172.21.15.98:0/3263591216 >> 172.21.15.98:0/3263591216 conn(0x7f01ec06a4d0 msgr2=0x7f01ec157740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.664+0000 7f01f32d3640 1 -- 172.21.15.98:0/3263591216 shutdown_connections 2023-11-28T17:44:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.664+0000 7f01f32d3640 1 -- 172.21.15.98:0/3263591216 wait complete. 2023-11-28T17:44:51.680 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~0s 2023-11-28T17:44:51.682 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:44:51.682 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:44:51.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:44:51.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.892+0000 7f29b23db640 1 Processor -- start 2023-11-28T17:44:51.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.892+0000 7f29b23db640 1 -- start start 2023-11-28T17:44:51.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.894+0000 7f29b23db640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29ac06b160 0x7f29ac0c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.894+0000 7f29b23db640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f29ac0c28c0 0x7f29ac162390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.894+0000 7f29b23db640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ac0c2d70 con 0x7f29ac0c24e0 2023-11-28T17:44:51.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.894+0000 7f29b23db640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29ac0c6300 con 0x7f29ac06b160 2023-11-28T17:44:51.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.894+0000 7f29b23db640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29ac0c7430 con 0x7f29ac0c28c0 2023-11-28T17:44:51.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.894+0000 7f29b0bd8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29ac06b160 0x7f29ac0c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.894+0000 7f29b0bd8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29ac06b160 0x7f29ac0c6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55312/0 (socket says 172.21.15.98:55312) 2023-11-28T17:44:51.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.894+0000 7f29b0bd8640 1 -- 172.21.15.98:0/1124862094 learned_addr learned my addr 172.21.15.98:0/1124862094 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:44:51.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.894+0000 7f29b1bda640 1 --2- 172.21.15.98:0/1124862094 >> v2:172.21.15.113:3300/0 conn(0x7f29ac0c28c0 0x7f29ac162390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.895+0000 7f29b0bd8640 1 -- 172.21.15.98:0/1124862094 >> v2:172.21.15.113:3300/0 conn(0x7f29ac0c28c0 msgr2=0x7f29ac162390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:44:51.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.895+0000 7f29b0bd8640 1 --2- 172.21.15.98:0/1124862094 >> v2:172.21.15.113:3300/0 conn(0x7f29ac0c28c0 0x7f29ac162390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.895+0000 7f29b0bd8640 1 -- 172.21.15.98:0/1124862094 >> v1:172.21.15.113:6789/0 conn(0x7f29ac0c24e0 legacy=0x7f29ac157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:44:51.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.895+0000 7f29b0bd8640 1 -- 172.21.15.98:0/1124862094 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ac0c7710 con 0x7f29ac06b160 2023-11-28T17:44:51.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.896+0000 7f29b0bd8640 1 --2- 172.21.15.98:0/1124862094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ac06b160 0x7f29ac0c6dc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f29ac067590 tx=0x7f299c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8277f2c317e846bd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:51.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.896+0000 7f29a3fff640 1 -- 172.21.15.98:0/1124862094 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c0317e0 con 0x7f29ac06b160 2023-11-28T17:44:51.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.896+0000 7f29a3fff640 1 -- 172.21.15.98:0/1124862094 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299c031940 con 0x7f29ac06b160 2023-11-28T17:44:51.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.896+0000 7f29a3fff640 1 -- 172.21.15.98:0/1124862094 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c031cd0 con 0x7f29ac06b160 2023-11-28T17:44:51.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.897+0000 7f29b23db640 1 -- 172.21.15.98:0/1124862094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ac06b160 msgr2=0x7f29ac0c6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.897+0000 7f29b23db640 1 --2- 172.21.15.98:0/1124862094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ac06b160 0x7f29ac0c6dc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f29ac067590 tx=0x7f299c02b880 comp rx=0 tx=0).stop 2023-11-28T17:44:51.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.897+0000 7f29b23db640 1 -- 172.21.15.98:0/1124862094 shutdown_connections 2023-11-28T17:44:51.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.897+0000 7f29b23db640 1 --2- 172.21.15.98:0/1124862094 >> v2:172.21.15.113:3300/0 conn(0x7f29ac0c28c0 0x7f29ac162390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.897+0000 7f29b23db640 1 --2- 172.21.15.98:0/1124862094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ac06b160 0x7f29ac0c6dc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.897+0000 7f29b23db640 1 -- 172.21.15.98:0/1124862094 >> 172.21.15.98:0/1124862094 conn(0x7f29ac0cafe0 msgr2=0x7f29ac0cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:51.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.897+0000 7f29b23db640 1 -- 172.21.15.98:0/1124862094 shutdown_connections 2023-11-28T17:44:51.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.897+0000 7f29b23db640 1 -- 172.21.15.98:0/1124862094 wait complete. 2023-11-28T17:44:51.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.899+0000 7f29b23db640 1 Processor -- start 2023-11-28T17:44:51.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.899+0000 7f29b23db640 1 -- start start 2023-11-28T17:44:51.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.900+0000 7f29b23db640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29ac06b160 0x7f29ac1ed730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.900+0000 7f29b23db640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ac0c24e0 0x7f29ac159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.900+0000 7f29b23db640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac0c28c0 0x7f29ac154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.900+0000 7f29b23db640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac146480 con 0x7f29ac0c24e0 2023-11-28T17:44:51.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.900+0000 7f29b23db640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac146760 con 0x7f29ac06b160 2023-11-28T17:44:51.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.900+0000 7f29b23db640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac146a40 con 0x7f29ac0c28c0 2023-11-28T17:44:51.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.900+0000 7f29b0bd8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ac0c24e0 0x7f29ac159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.900+0000 7f29b0bd8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ac0c24e0 0x7f29ac159e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55322/0 (socket says 172.21.15.98:55322) 2023-11-28T17:44:51.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.900+0000 7f29b0bd8640 1 -- 172.21.15.98:0/3495609294 learned_addr learned my addr 172.21.15.98:0/3495609294 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:44:51.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.901+0000 7f29b13d9640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29ac06b160 0x7f29ac1ed730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.901+0000 7f29b1bda640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac0c28c0 0x7f29ac154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.901+0000 7f29b0bd8640 1 -- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac0c28c0 msgr2=0x7f29ac154c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.901+0000 7f29b0bd8640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac0c28c0 0x7f29ac154c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.901+0000 7f29b0bd8640 1 -- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29ac06b160 msgr2=0x7f29ac1ed730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.901+0000 7f29b0bd8640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29ac06b160 0x7f29ac1ed730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.901+0000 7f29b0bd8640 1 -- 172.21.15.98:0/3495609294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c0094d0 con 0x7f29ac0c24e0 2023-11-28T17:44:51.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.901+0000 7f29b0bd8640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ac0c24e0 0x7f29ac159e70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f299c000c00 tx=0x7f299c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:51.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.901+0000 7f29a1ffb640 1 -- 172.21.15.98:0/3495609294 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c032880 con 0x7f29ac0c24e0 2023-11-28T17:44:51.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.901+0000 7f29a1ffb640 1 -- 172.21.15.98:0/3495609294 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299c032a00 con 0x7f29ac0c24e0 2023-11-28T17:44:51.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.901+0000 7f29a1ffb640 1 -- 172.21.15.98:0/3495609294 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c032db0 con 0x7f29ac0c24e0 2023-11-28T17:44:51.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.901+0000 7f29b23db640 1 -- 172.21.15.98:0/3495609294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ac146d20 con 0x7f29ac0c24e0 2023-11-28T17:44:51.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.902+0000 7f29b23db640 1 -- 172.21.15.98:0/3495609294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ac1472e0 con 0x7f29ac0c24e0 2023-11-28T17:44:51.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.902+0000 7f29b23db640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac15bc60 0x7f29ac155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.902+0000 7f29b23db640 1 -- 172.21.15.98:0/3495609294 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac074310 con 0x7f29ac15bc60 2023-11-28T17:44:51.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.903+0000 7f29b13d9640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac15bc60 0x7f29ac155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.904+0000 7f29a1ffb640 1 -- 172.21.15.98:0/3495609294 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f299c04d3f0 con 0x7f29ac0c24e0 2023-11-28T17:44:51.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.904+0000 7f29b13d9640 1 -- 172.21.15.98:0/3495609294 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29ac0c3630 con 0x7f29ac15bc60 2023-11-28T17:44:51.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.904+0000 7f29b13d9640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac15bc60 0x7f29ac155150 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f29a800e4d0 tx=0x7f29a800e990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:51.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.905+0000 7f29a1ffb640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f299006b2a0 0x7f299006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.905+0000 7f29a1ffb640 1 -- 172.21.15.98:0/3495609294 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f299c0ab930 con 0x7f29ac0c24e0 2023-11-28T17:44:51.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.905+0000 7f29b1bda640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f299006b2a0 0x7f299006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.905+0000 7f29a1ffb640 1 -- 172.21.15.98:0/3495609294 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a8007240 con 0x7f29ac15bc60 2023-11-28T17:44:51.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.905+0000 7f29a1ffb640 1 -- 172.21.15.98:0/3495609294 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f29a80073a0 con 0x7f29ac15bc60 2023-11-28T17:44:51.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.905+0000 7f29a1ffb640 1 -- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac15bc60 msgr2=0x7f29ac155150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.905+0000 7f29a1ffb640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac15bc60 0x7f29ac155150 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f29a800e4d0 tx=0x7f29a800e990 comp rx=0 tx=0).stop 2023-11-28T17:44:51.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.905+0000 7f29a1ffb640 1 -- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac15bc60 msgr2=0x7f29ac155150 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:44:51.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.905+0000 7f29a1ffb640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac15bc60 0x7f29ac155150 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.906+0000 7f29b1bda640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f299006b2a0 0x7f299006d760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009e10 tx=0x7f29a400b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:51.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.913+0000 7f29b23db640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac148f10 0x7f29ac14a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:51.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.913+0000 7f29b23db640 1 -- 172.21.15.98:0/3495609294 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac146eb0 con 0x7f29ac148f10 2023-11-28T17:44:51.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.913+0000 7f29b13d9640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac148f10 0x7f29ac14a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:51.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.914+0000 7f29b13d9640 1 -- 172.21.15.98:0/3495609294 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f29ac15b5c0 con 0x7f29ac148f10 2023-11-28T17:44:51.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.914+0000 7f29b13d9640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac148f10 0x7f29ac14a710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f29a800e4d0 tx=0x7f29a8003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:51.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.915+0000 7f29a1ffb640 1 -- 172.21.15.98:0/3495609294 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a801ed10 con 0x7f29ac148f10 2023-11-28T17:44:51.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.915+0000 7f29a1ffb640 1 -- 172.21.15.98:0/3495609294 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4975 (secure 0 0 0) 0x7f29a801ee70 con 0x7f29ac148f10 2023-11-28T17:44:51.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.915+0000 7f29a1ffb640 1 -- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac148f10 msgr2=0x7f29ac14a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.915+0000 7f29a1ffb640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac148f10 0x7f29ac14a710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f29a800e4d0 tx=0x7f29a8003760 comp rx=0 tx=0).stop 2023-11-28T17:44:51.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.915+0000 7f29a1ffb640 1 -- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac148f10 msgr2=0x7f29ac14a710 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:44:51.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.915+0000 7f29a1ffb640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29ac148f10 0x7f29ac14a710 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f29a800e4d0 tx=0x7f29a8003760 comp rx=0 tx=0).stop 2023-11-28T17:44:51.918 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:44:51.918 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:44:51.918 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:44:51.918 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:44:51.919 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 36, 2023-11-28T17:44:51.919 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:44:51.919 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:44:51.919 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:44:51.919 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:44:51.920 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:51.920 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2023-11-28T17:44:51.920 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:44:51.920 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:44:51.920 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:44:51.921 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:44:51.921 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:44:51.921 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:44:51.921 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:44:51.921 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:44:51.921 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:44:51.922 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:44:51.922 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:44:51.922 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:44:51.922 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:44:51.922 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:51.923 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:44:51.923 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:44:51.923 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:44:51.923 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:44:51.923 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:44:51.923 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:44:51.924 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:44:51.924 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:44:51.924 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:44:51.924 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:44:51.925 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:44:51.925 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:44:51.926 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.926 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:44:51.926 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:44:51.926 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:44:51.926 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:44:51.927 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:44:51.927 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:44:51.927 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:44:51.927 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:44:51.928 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:44:51.928 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:44:51.928 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:44:51.928 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:44:51.928 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:44:51.928 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:44:51.929 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:44:51.929 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:44:51.929 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:44:51.929 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:44:51.929 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:44:51.930 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:44:51.930 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:44:51.930 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:44:51.930 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:44:51.930 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:44:51.930 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:44:51.931 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:44:51.931 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:44:51.931 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:51.931 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:44:51.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.931 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:44:51.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.932 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:44:51.932 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:44:51.932 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:44:51.932 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:44:51.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.933 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:44:51.933 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:44:51.933 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:44:51.934 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:44:51.934 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.934 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.934 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.935 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:44:51.935 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:44:51.935 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:44:51.935 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:44:51.935 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:44:51.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.936 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:44:51.936 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:44:51.936 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:44:51.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:44:51.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:44:51.937 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:44:51.937 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:44:51.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:44:51.938 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.938 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:44:51.938 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:44:51.938 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:44:51.938 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:44:51.939 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:44:51.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.939 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:44:51.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:44:51.939 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:44:51.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:44:51.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:44:51.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:44:51.939 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.940 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:44:51.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:44:51.940 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:44:51.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.940 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:44:51.940 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:44:51.940 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:44:51.940 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:44:51.940 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:44:51.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.941 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.941 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:44:51.941 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:44:51.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.941 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:51.941 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:51.941 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:44:51.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.941 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:51.941 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:44:51.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.942 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:51.942 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:51.942 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:44:51.942 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.942 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:51.942 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:44:51.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.942 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:51.942 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:51.942 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 8 2023-11-28T17:44:51.942 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.943 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:51.943 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:44:51.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.943 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:51.943 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:51.943 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:44:51.943 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.943 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:44:51.943 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:44:51.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:44:51.943 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:44:51.943 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:44:51.943 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:44:51.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:44:51.944 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:44:51.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:44:51.944 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:44:51.944 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:44:51.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.916+0000 7f29b23db640 1 -- 172.21.15.98:0/3495609294 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f299006b2a0 msgr2=0x7f299006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.916+0000 7f29b23db640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f299006b2a0 0x7f299006d760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009e10 tx=0x7f29a400b040 comp rx=0 tx=0).stop 2023-11-28T17:44:51.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.916+0000 7f29b23db640 1 -- 172.21.15.98:0/3495609294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ac0c24e0 msgr2=0x7f29ac159e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:51.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.916+0000 7f29b23db640 1 --2- 172.21.15.98:0/3495609294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ac0c24e0 0x7f29ac159e70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f299c000c00 tx=0x7f299c0046c0 comp rx=0 tx=0).stop 2023-11-28T17:44:51.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.916+0000 7f29b1bda640 1 -- 172.21.15.98:0/3495609294 reap_dead start 2023-11-28T17:44:51.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.916+0000 7f29b1bda640 1 -- 172.21.15.98:0/3495609294 reap_dead start 2023-11-28T17:44:51.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.916+0000 7f29b23db640 1 -- 172.21.15.98:0/3495609294 shutdown_connections 2023-11-28T17:44:51.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.916+0000 7f29b23db640 1 -- 172.21.15.98:0/3495609294 >> 172.21.15.98:0/3495609294 conn(0x7f29ac0cafe0 msgr2=0x7f29ac150740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:51.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.916+0000 7f29b23db640 1 -- 172.21.15.98:0/3495609294 shutdown_connections 2023-11-28T17:44:51.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:51.916+0000 7f29b23db640 1 -- 172.21.15.98:0/3495609294 wait complete. 2023-11-28T17:44:51.946 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:44:51.946 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-11-28T17:44:51.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-11-28T17:44:52.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.158+0000 7fd47ef1d640 1 Processor -- start 2023-11-28T17:44:52.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.158+0000 7fd47ef1d640 1 -- start start 2023-11-28T17:44:52.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.159+0000 7fd47ef1d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd478157b60 0x7fd47815bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:52.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.160+0000 7fd47ef1d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd4780c4910 0x7fd4780c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:52.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.160+0000 7fd47ef1d640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47815b150 con 0x7fd478151d80 2023-11-28T17:44:52.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.160+0000 7fd47ef1d640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd478157f90 con 0x7fd478157b60 2023-11-28T17:44:52.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.160+0000 7fd47ef1d640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47815c150 con 0x7fd4780c4910 2023-11-28T17:44:52.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.160+0000 7fd47d4ba640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd4780c4910 0x7fd4780c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:52.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.160+0000 7fd46ffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd478157b60 0x7fd47815bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:52.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.160+0000 7fd46ffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fd478157b60 0x7fd47815bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:47512/0 (socket says 172.21.15.98:47512) 2023-11-28T17:44:52.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.160+0000 7fd46ffff640 1 -- 172.21.15.98:0/2919469693 learned_addr learned my addr 172.21.15.98:0/2919469693 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:44:52.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.160+0000 7fd47d4ba640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd4780c4910 0x7fd4780c4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:57462/0 (socket says 172.21.15.98:57462) 2023-11-28T17:44:52.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.161+0000 7fd47d4ba640 1 -- 172.21.15.98:0/2919469693 >> v2:172.21.15.113:3300/0 conn(0x7fd478157b60 msgr2=0x7fd47815bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:44:52.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.161+0000 7fd47d4ba640 1 --2- 172.21.15.98:0/2919469693 >> v2:172.21.15.113:3300/0 conn(0x7fd478157b60 0x7fd47815bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:52.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.161+0000 7fd47d4ba640 1 -- 172.21.15.98:0/2919469693 >> v1:172.21.15.113:6789/0 conn(0x7fd478151d80 legacy=0x7fd4780c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:52.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.161+0000 7fd47d4ba640 1 -- 172.21.15.98:0/2919469693 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47806e1f0 con 0x7fd4780c4910 2023-11-28T17:44:52.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.162+0000 7fd47d4ba640 1 --2- 172.21.15.98:0/2919469693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd4780c4910 0x7fd4780c4d40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd470009d90 tx=0x7fd47002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=464621f133a57158 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:52.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.163+0000 7fd46f7fe640 1 -- 172.21.15.98:0/2919469693 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47002cce0 con 0x7fd4780c4910 2023-11-28T17:44:52.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.163+0000 7fd46f7fe640 1 -- 172.21.15.98:0/2919469693 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47002ce40 con 0x7fd4780c4910 2023-11-28T17:44:52.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.163+0000 7fd46f7fe640 1 -- 172.21.15.98:0/2919469693 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd470034660 con 0x7fd4780c4910 2023-11-28T17:44:52.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.163+0000 7fd47ef1d640 1 -- 172.21.15.98:0/2919469693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd4780c4910 msgr2=0x7fd4780c4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:52.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.163+0000 7fd47ef1d640 1 --2- 172.21.15.98:0/2919469693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd4780c4910 0x7fd4780c4d40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd470009d90 tx=0x7fd47002bf30 comp rx=0 tx=0).stop 2023-11-28T17:44:52.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.163+0000 7fd47ef1d640 1 -- 172.21.15.98:0/2919469693 shutdown_connections 2023-11-28T17:44:52.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.163+0000 7fd47ef1d640 1 --2- 172.21.15.98:0/2919469693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd4780c4910 0x7fd4780c4d40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:52.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.163+0000 7fd47ef1d640 1 --2- 172.21.15.98:0/2919469693 >> v2:172.21.15.113:3300/0 conn(0x7fd478157b60 0x7fd47815bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:52.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.163+0000 7fd47ef1d640 1 -- 172.21.15.98:0/2919469693 >> 172.21.15.98:0/2919469693 conn(0x7fd47806a4d0 msgr2=0x7fd47806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:52.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.164+0000 7fd47ef1d640 1 -- 172.21.15.98:0/2919469693 shutdown_connections 2023-11-28T17:44:52.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.164+0000 7fd47ef1d640 1 -- 172.21.15.98:0/2919469693 wait complete. 2023-11-28T17:44:52.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.166+0000 7fd47ef1d640 1 Processor -- start 2023-11-28T17:44:52.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.166+0000 7fd47ef1d640 1 -- start start 2023-11-28T17:44:52.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.167+0000 7fd47ef1d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4780c4910 0x7fd4780c38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:52.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.167+0000 7fd47ef1d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd478151d80 0x7fd4781f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:52.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.167+0000 7fd47ef1d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd478157b60 0x7fd478150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:52.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.167+0000 7fd47ef1d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4781f3450 con 0x7fd4780c4910 2023-11-28T17:44:52.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.167+0000 7fd47ef1d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4781f3e20 con 0x7fd478151d80 2023-11-28T17:44:52.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.167+0000 7fd47ef1d640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4780c3f50 con 0x7fd478157b60 2023-11-28T17:44:52.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.167+0000 7fd47ccb9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4780c4910 0x7fd4780c38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:52.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.167+0000 7fd47ccb9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4780c4910 0x7fd4780c38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55332/0 (socket says 172.21.15.98:55332) 2023-11-28T17:44:52.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.167+0000 7fd47ccb9640 1 -- 172.21.15.98:0/3187440862 learned_addr learned my addr 172.21.15.98:0/3187440862 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:44:52.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.168+0000 7fd46ffff640 1 --2- 172.21.15.98:0/3187440862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd478151d80 0x7fd4781f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:52.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.168+0000 7fd47d4ba640 1 --2- 172.21.15.98:0/3187440862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd478157b60 0x7fd478150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:52.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.168+0000 7fd47ccb9640 1 -- 172.21.15.98:0/3187440862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd478157b60 msgr2=0x7fd478150ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:52.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.168+0000 7fd47ccb9640 1 --2- 172.21.15.98:0/3187440862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd478157b60 0x7fd478150ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:52.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.168+0000 7fd47ccb9640 1 -- 172.21.15.98:0/3187440862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd478151d80 msgr2=0x7fd4781f38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:52.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.168+0000 7fd47ccb9640 1 --2- 172.21.15.98:0/3187440862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd478151d80 0x7fd4781f38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:52.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.168+0000 7fd47ccb9640 1 -- 172.21.15.98:0/3187440862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd470009a70 con 0x7fd4780c4910 2023-11-28T17:44:52.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.168+0000 7fd47d4ba640 1 --2- 172.21.15.98:0/3187440862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd478157b60 0x7fd478150ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:44:52.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.168+0000 7fd47ccb9640 1 --2- 172.21.15.98:0/3187440862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4780c4910 0x7fd4780c38a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd47400c450 tx=0x7fd47400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:52.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.168+0000 7fd46d7fa640 1 -- 172.21.15.98:0/3187440862 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4740052d0 con 0x7fd4780c4910 2023-11-28T17:44:52.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.168+0000 7fd46d7fa640 1 -- 172.21.15.98:0/3187440862 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd474005430 con 0x7fd4780c4910 2023-11-28T17:44:52.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.168+0000 7fd47ef1d640 1 -- 172.21.15.98:0/3187440862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4780c4230 con 0x7fd4780c4910 2023-11-28T17:44:52.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.168+0000 7fd46d7fa640 1 -- 172.21.15.98:0/3187440862 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd474012870 con 0x7fd4780c4910 2023-11-28T17:44:52.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.169+0000 7fd47ef1d640 1 -- 172.21.15.98:0/3187440862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4781f30e0 con 0x7fd4780c4910 2023-11-28T17:44:52.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.170+0000 7fd47ef1d640 1 -- 172.21.15.98:0/3187440862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4781f32a0 con 0x7fd4780c4910 2023-11-28T17:44:52.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.170+0000 7fd46d7fa640 1 -- 172.21.15.98:0/3187440862 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fd4740129d0 con 0x7fd4780c4910 2023-11-28T17:44:52.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.171+0000 7fd46d7fa640 1 --2- 172.21.15.98:0/3187440862 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd46406b2a0 0x7fd46406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:44:52.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.172+0000 7fd46d7fa640 1 -- 172.21.15.98:0/3187440862 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd474089780 con 0x7fd4780c4910 2023-11-28T17:44:52.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.172+0000 7fd46ffff640 1 --2- 172.21.15.98:0/3187440862 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd46406b2a0 0x7fd46406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:44:52.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.172+0000 7fd46ffff640 1 --2- 172.21.15.98:0/3187440862 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd46406b2a0 0x7fd46406d760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd4781ed620 tx=0x7fd468008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:44:52.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.176+0000 7fd46d7fa640 1 -- 172.21.15.98:0/3187440862 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd474054570 con 0x7fd4780c4910 2023-11-28T17:44:52.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.444+0000 7fd47ef1d640 1 -- 172.21.15.98:0/3187440862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fd47806b2d0 con 0x7fd4780c4910 2023-11-28T17:44:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.447+0000 7fd46d7fa640 1 -- 172.21.15.98:0/3187440862 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fd474057b90 con 0x7fd4780c4910 2023-11-28T17:44:52.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.449+0000 7fd47ef1d640 1 -- 172.21.15.98:0/3187440862 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd46406b2a0 msgr2=0x7fd46406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:52.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.449+0000 7fd47ef1d640 1 --2- 172.21.15.98:0/3187440862 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd46406b2a0 0x7fd46406d760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd4781ed620 tx=0x7fd468008040 comp rx=0 tx=0).stop 2023-11-28T17:44:52.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.449+0000 7fd47ef1d640 1 -- 172.21.15.98:0/3187440862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4780c4910 msgr2=0x7fd4780c38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:44:52.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.449+0000 7fd47ef1d640 1 --2- 172.21.15.98:0/3187440862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4780c4910 0x7fd4780c38a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd47400c450 tx=0x7fd47400c910 comp rx=0 tx=0).stop 2023-11-28T17:44:52.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.449+0000 7fd47ef1d640 1 -- 172.21.15.98:0/3187440862 shutdown_connections 2023-11-28T17:44:52.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.449+0000 7fd47ef1d640 1 --2- 172.21.15.98:0/3187440862 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fd478157b60 0x7fd478150ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:52.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.449+0000 7fd47ef1d640 1 --2- 172.21.15.98:0/3187440862 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fd46406b2a0 0x7fd46406d760 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:52.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.449+0000 7fd47ef1d640 1 --2- 172.21.15.98:0/3187440862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd478151d80 0x7fd4781f38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:52.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.449+0000 7fd47ef1d640 1 --2- 172.21.15.98:0/3187440862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4780c4910 0x7fd4780c38a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:44:52.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.449+0000 7fd47ef1d640 1 -- 172.21.15.98:0/3187440862 >> 172.21.15.98:0/3187440862 conn(0x7fd47806a4d0 msgr2=0x7fd4780c8490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:44:52.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.450+0000 7fd47ef1d640 1 -- 172.21.15.98:0/3187440862 shutdown_connections 2023-11-28T17:44:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:44:52.450+0000 7fd47ef1d640 1 -- 172.21.15.98:0/3187440862 wait complete. 2023-11-28T17:44:52.472 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2023-11-28T17:44:58.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~7s 2023-11-28T17:45:02.473 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:45:02.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:45:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.693+0000 7f91f998a640 1 Processor -- start 2023-11-28T17:45:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.693+0000 7f91f998a640 1 -- start start 2023-11-28T17:45:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.695+0000 7f91f998a640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f91f4157b60 0x7f91f415bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.695+0000 7f91f998a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f415b150 con 0x7f91f40c4910 2023-11-28T17:45:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.695+0000 7f91f998a640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f406e1f0 con 0x7f91f4151d80 2023-11-28T17:45:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.695+0000 7f91f998a640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f4157f90 con 0x7f91f4157b60 2023-11-28T17:45:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.695+0000 7f91f37fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f91f40c4910 0x7f91f41624e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49702/0 (socket says 172.21.15.98:49702) 2023-11-28T17:45:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.695+0000 7f91f37fe640 1 -- 172.21.15.98:0/2922502673 learned_addr learned my addr 172.21.15.98:0/2922502673 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:45:02.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.696+0000 7f91ebfff640 1 --2- 172.21.15.98:0/2922502673 >> v2:172.21.15.113:3300/0 conn(0x7f91f4157b60 0x7f91f415bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:02.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.696+0000 7f91f27fc640 1 -- 172.21.15.98:0/2922502673 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516035851 0 0) 0x7f91f415b150 con 0x7f91f40c4910 2023-11-28T17:45:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.696+0000 7f91f27fc640 1 -- 172.21.15.98:0/2922502673 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e0003520 con 0x7f91f40c4910 2023-11-28T17:45:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.696+0000 7f91f27fc640 1 -- 172.21.15.98:0/2922502673 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1101429032 0 0) 0x7f91e0003520 con 0x7f91f40c4910 2023-11-28T17:45:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.696+0000 7f91ebfff640 1 -- 172.21.15.98:0/2922502673 >> v1:172.21.15.113:6789/0 conn(0x7f91f4151d80 legacy=0x7f91f40c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.696+0000 7f91ebfff640 1 -- 172.21.15.98:0/2922502673 >> v1:172.21.15.98:6789/0 conn(0x7f91f40c4910 legacy=0x7f91f41624e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.696+0000 7f91ebfff640 1 -- 172.21.15.98:0/2922502673 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f406e4d0 con 0x7f91f4157b60 2023-11-28T17:45:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.698+0000 7f91ebfff640 1 --2- 172.21.15.98:0/2922502673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91f4157b60 0x7f91f415bc10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0097a0 tx=0x7f91dc02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=698061e931383cd0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.698+0000 7f91f27fc640 1 -- 172.21.15.98:0/2922502673 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ec0019f0 con 0x7f91f4157b60 2023-11-28T17:45:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.698+0000 7f91f27fc640 1 -- 172.21.15.98:0/2922502673 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91dc031590 con 0x7f91f4157b60 2023-11-28T17:45:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.698+0000 7f91f27fc640 1 -- 172.21.15.98:0/2922502673 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc031920 con 0x7f91f4157b60 2023-11-28T17:45:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.699+0000 7f91f998a640 1 -- 172.21.15.98:0/2922502673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91f4157b60 msgr2=0x7f91f415bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.699+0000 7f91f998a640 1 --2- 172.21.15.98:0/2922502673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91f4157b60 0x7f91f415bc10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0097a0 tx=0x7f91dc02b8b0 comp rx=0 tx=0).stop 2023-11-28T17:45:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.699+0000 7f91f998a640 1 -- 172.21.15.98:0/2922502673 shutdown_connections 2023-11-28T17:45:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.699+0000 7f91f998a640 1 --2- 172.21.15.98:0/2922502673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91f4157b60 0x7f91f415bc10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.699+0000 7f91f998a640 1 -- 172.21.15.98:0/2922502673 >> 172.21.15.98:0/2922502673 conn(0x7f91f406a4d0 msgr2=0x7f91f406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.699+0000 7f91f998a640 1 -- 172.21.15.98:0/2922502673 shutdown_connections 2023-11-28T17:45:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.699+0000 7f91f998a640 1 -- 172.21.15.98:0/2922502673 wait complete. 2023-11-28T17:45:02.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.701+0000 7f91f998a640 1 Processor -- start 2023-11-28T17:45:02.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.701+0000 7f91f998a640 1 -- start start 2023-11-28T17:45:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.702+0000 7f91f998a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91f40c4910 0x7f91f4070810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.702+0000 7f91f998a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91f4151d80 0x7f91f40cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.702+0000 7f91f998a640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91f4157b60 0x7f91f4159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.702+0000 7f91f998a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f40c3470 con 0x7f91f40c4910 2023-11-28T17:45:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.702+0000 7f91f998a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f4070d50 con 0x7f91f4151d80 2023-11-28T17:45:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.702+0000 7f91f998a640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f4158ec0 con 0x7f91f4157b60 2023-11-28T17:45:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.702+0000 7f91f2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91f40c4910 0x7f91f4070810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.702+0000 7f91f2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91f40c4910 0x7f91f4070810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34014/0 (socket says 172.21.15.98:34014) 2023-11-28T17:45:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.702+0000 7f91f2ffd640 1 -- 172.21.15.98:0/2072641336 learned_addr learned my addr 172.21.15.98:0/2072641336 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.703+0000 7f91ebfff640 1 --2- 172.21.15.98:0/2072641336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91f4151d80 0x7f91f40cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.703+0000 7f91f2ffd640 1 -- 172.21.15.98:0/2072641336 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91f4157b60 msgr2=0x7f91f4159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.703+0000 7f91f2ffd640 1 --2- 172.21.15.98:0/2072641336 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91f4157b60 0x7f91f4159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.703+0000 7f91f2ffd640 1 -- 172.21.15.98:0/2072641336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91f4151d80 msgr2=0x7f91f40cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.703+0000 7f91f2ffd640 1 --2- 172.21.15.98:0/2072641336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91f4151d80 0x7f91f40cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.703+0000 7f91f2ffd640 1 -- 172.21.15.98:0/2072641336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc009400 con 0x7f91f40c4910 2023-11-28T17:45:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.703+0000 7f91f2ffd640 1 --2- 172.21.15.98:0/2072641336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91f40c4910 0x7f91f4070810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f91e400c450 tx=0x7f91e400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.703+0000 7f91f0ff9640 1 -- 172.21.15.98:0/2072641336 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e40052d0 con 0x7f91f40c4910 2023-11-28T17:45:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.703+0000 7f91f0ff9640 1 -- 172.21.15.98:0/2072641336 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e4005430 con 0x7f91f40c4910 2023-11-28T17:45:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.703+0000 7f91f0ff9640 1 -- 172.21.15.98:0/2072641336 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e4012870 con 0x7f91f40c4910 2023-11-28T17:45:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.703+0000 7f91f998a640 1 -- 172.21.15.98:0/2072641336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f40c36c0 con 0x7f91f40c4910 2023-11-28T17:45:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.703+0000 7f91f998a640 1 -- 172.21.15.98:0/2072641336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f40c3c80 con 0x7f91f40c4910 2023-11-28T17:45:02.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.705+0000 7f91f998a640 1 -- 172.21.15.98:0/2072641336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f4067bf0 con 0x7f91f40c4910 2023-11-28T17:45:02.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.705+0000 7f91f0ff9640 1 -- 172.21.15.98:0/2072641336 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f91e40129d0 con 0x7f91f40c4910 2023-11-28T17:45:02.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.706+0000 7f91f0ff9640 1 --2- 172.21.15.98:0/2072641336 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f91d406b2a0 0x7f91d406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:02.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.706+0000 7f91f0ff9640 1 -- 172.21.15.98:0/2072641336 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f91e4089790 con 0x7f91f40c4910 2023-11-28T17:45:02.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.707+0000 7f91ebfff640 1 --2- 172.21.15.98:0/2072641336 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f91d406b2a0 0x7f91d406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:02.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.707+0000 7f91ebfff640 1 --2- 172.21.15.98:0/2072641336 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f91d406b2a0 0x7f91d406d760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f91dc036000 tx=0x7f91dc03d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:02.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.712+0000 7f91f0ff9640 1 -- 172.21.15.98:0/2072641336 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91e4054580 con 0x7f91f40c4910 2023-11-28T17:45:02.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.977+0000 7f91f998a640 1 -- 172.21.15.98:0/2072641336 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f91f4152160 con 0x7f91f40c4910 2023-11-28T17:45:02.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.978+0000 7f91f0ff9640 1 -- 172.21.15.98:0/2072641336 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f91e4057ba0 con 0x7f91f40c4910 2023-11-28T17:45:02.979 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:45:02.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.981+0000 7f91f998a640 1 -- 172.21.15.98:0/2072641336 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f91d406b2a0 msgr2=0x7f91d406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:02.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.981+0000 7f91f998a640 1 --2- 172.21.15.98:0/2072641336 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f91d406b2a0 0x7f91d406d760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f91dc036000 tx=0x7f91dc03d040 comp rx=0 tx=0).stop 2023-11-28T17:45:02.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.981+0000 7f91f998a640 1 -- 172.21.15.98:0/2072641336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91f40c4910 msgr2=0x7f91f4070810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:02.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.981+0000 7f91f998a640 1 --2- 172.21.15.98:0/2072641336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91f40c4910 0x7f91f4070810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f91e400c450 tx=0x7f91e400c910 comp rx=0 tx=0).stop 2023-11-28T17:45:02.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.981+0000 7f91f998a640 1 -- 172.21.15.98:0/2072641336 shutdown_connections 2023-11-28T17:45:02.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.981+0000 7f91f998a640 1 --2- 172.21.15.98:0/2072641336 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f91f4157b60 0x7f91f4159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:02.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.981+0000 7f91f998a640 1 --2- 172.21.15.98:0/2072641336 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f91d406b2a0 0x7f91d406d760 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:02.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.981+0000 7f91f998a640 1 --2- 172.21.15.98:0/2072641336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91f4151d80 0x7f91f40cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:02.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.981+0000 7f91f998a640 1 --2- 172.21.15.98:0/2072641336 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91f40c4910 0x7f91f4070810 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:02.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.981+0000 7f91f998a640 1 -- 172.21.15.98:0/2072641336 >> 172.21.15.98:0/2072641336 conn(0x7f91f406a4d0 msgr2=0x7f91f40c1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:02.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.981+0000 7f91f998a640 1 -- 172.21.15.98:0/2072641336 shutdown_connections 2023-11-28T17:45:02.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:02.981+0000 7f91f998a640 1 -- 172.21.15.98:0/2072641336 wait complete. 2023-11-28T17:45:03.002 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:45:03.002 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-11-28T17:45:03.003 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:45:03.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:45:03.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.217+0000 7fc9d6678640 1 Processor -- start 2023-11-28T17:45:03.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.217+0000 7fc9d6678640 1 -- start start 2023-11-28T17:45:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.219+0000 7fc9d6678640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9d006a4d0 0x7fc9d015dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.219+0000 7fc9d6678640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc9d0070a90 0x7fc9d006a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.219+0000 7fc9d6678640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d00b03b0 con 0x7fc9d006b160 2023-11-28T17:45:03.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.219+0000 7fc9d6678640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d006adf0 con 0x7fc9d0070a90 2023-11-28T17:45:03.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.219+0000 7fc9d6678640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d006d1e0 con 0x7fc9d006a4d0 2023-11-28T17:45:03.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.219+0000 7fc9d5676640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9d006a4d0 0x7fc9d015dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.219+0000 7fc9d5e77640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc9d0070a90 0x7fc9d006a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.219+0000 7fc9d5676640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9d006a4d0 0x7fc9d015dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:34818/0 (socket says 172.21.15.98:34818) 2023-11-28T17:45:03.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.219+0000 7fc9d5676640 1 -- 172.21.15.98:0/1588094880 learned_addr learned my addr 172.21.15.98:0/1588094880 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:03.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.220+0000 7fc9d5e77640 1 -- 172.21.15.98:0/1588094880 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9d006a4d0 msgr2=0x7fc9d015dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:03.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.220+0000 7fc9d5e77640 1 --2- 172.21.15.98:0/1588094880 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9d006a4d0 0x7fc9d015dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.220+0000 7fc9d5e77640 1 -- 172.21.15.98:0/1588094880 >> v1:172.21.15.113:6789/0 conn(0x7fc9d006b160 legacy=0x7fc9d0070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.220+0000 7fc9d5e77640 1 -- 172.21.15.98:0/1588094880 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d006c2f0 con 0x7fc9d0070a90 2023-11-28T17:45:03.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.221+0000 7fc9d5e77640 1 --2- 172.21.15.98:0/1588094880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9d0070a90 0x7fc9d006a8b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009d90 tx=0x7fc9c802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2bbd35f4777bca2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:03.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.222+0000 7fc9c7fff640 1 -- 172.21.15.98:0/1588094880 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c802cce0 con 0x7fc9d0070a90 2023-11-28T17:45:03.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.222+0000 7fc9c7fff640 1 -- 172.21.15.98:0/1588094880 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c802ce40 con 0x7fc9d0070a90 2023-11-28T17:45:03.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.222+0000 7fc9c7fff640 1 -- 172.21.15.98:0/1588094880 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8034660 con 0x7fc9d0070a90 2023-11-28T17:45:03.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.222+0000 7fc9d6678640 1 -- 172.21.15.98:0/1588094880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9d0070a90 msgr2=0x7fc9d006a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.222+0000 7fc9d6678640 1 --2- 172.21.15.98:0/1588094880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9d0070a90 0x7fc9d006a8b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009d90 tx=0x7fc9c802bf30 comp rx=0 tx=0).stop 2023-11-28T17:45:03.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.223+0000 7fc9d6678640 1 -- 172.21.15.98:0/1588094880 shutdown_connections 2023-11-28T17:45:03.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.223+0000 7fc9d6678640 1 --2- 172.21.15.98:0/1588094880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9d0070a90 0x7fc9d006a8b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.223+0000 7fc9d6678640 1 --2- 172.21.15.98:0/1588094880 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9d006a4d0 0x7fc9d015dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.223+0000 7fc9d6678640 1 -- 172.21.15.98:0/1588094880 >> 172.21.15.98:0/1588094880 conn(0x7fc9d00cafe0 msgr2=0x7fc9d00cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:03.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.223+0000 7fc9d6678640 1 -- 172.21.15.98:0/1588094880 shutdown_connections 2023-11-28T17:45:03.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.223+0000 7fc9d6678640 1 -- 172.21.15.98:0/1588094880 wait complete. 2023-11-28T17:45:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.225+0000 7fc9d6678640 1 Processor -- start 2023-11-28T17:45:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.225+0000 7fc9d6678640 1 -- start start 2023-11-28T17:45:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.226+0000 7fc9d6678640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9d006a4d0 0x7fc9d00c4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.226+0000 7fc9d6678640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d006b160 0x7fc9d015cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.226+0000 7fc9d6678640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9d0070a90 0x7fc9d0159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.226+0000 7fc9d6678640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d0159710 con 0x7fc9d006b160 2023-11-28T17:45:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.226+0000 7fc9d6678640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d00c4e60 con 0x7fc9d0070a90 2023-11-28T17:45:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.226+0000 7fc9d6678640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d0157ae0 con 0x7fc9d006a4d0 2023-11-28T17:45:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.226+0000 7fc9d4e75640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d006b160 0x7fc9d015cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.226+0000 7fc9d4e75640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d006b160 0x7fc9d015cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34020/0 (socket says 172.21.15.98:34020) 2023-11-28T17:45:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.226+0000 7fc9d4e75640 1 -- 172.21.15.98:0/2458015795 learned_addr learned my addr 172.21.15.98:0/2458015795 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.227+0000 7fc9d5676640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9d006a4d0 0x7fc9d00c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.227+0000 7fc9d5e77640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9d0070a90 0x7fc9d0159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.227+0000 7fc9d4e75640 1 -- 172.21.15.98:0/2458015795 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9d006a4d0 msgr2=0x7fc9d00c4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.227+0000 7fc9d4e75640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9d006a4d0 0x7fc9d00c4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.227+0000 7fc9d4e75640 1 -- 172.21.15.98:0/2458015795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9d0070a90 msgr2=0x7fc9d0159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.227+0000 7fc9d4e75640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9d0070a90 0x7fc9d0159b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.227+0000 7fc9d4e75640 1 -- 172.21.15.98:0/2458015795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c8009a70 con 0x7fc9d006b160 2023-11-28T17:45:03.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.227+0000 7fc9d5676640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc9d006a4d0 0x7fc9d00c4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:45:03.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.227+0000 7fc9d5e77640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc9d0070a90 0x7fc9d0159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:45:03.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.227+0000 7fc9d4e75640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d006b160 0x7fc9d015cc20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc9c000c480 tx=0x7fc9c000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:03.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.227+0000 7fc9c5ffb640 1 -- 172.21.15.98:0/2458015795 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c00052d0 con 0x7fc9d006b160 2023-11-28T17:45:03.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.228+0000 7fc9c5ffb640 1 -- 172.21.15.98:0/2458015795 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c0005430 con 0x7fc9d006b160 2023-11-28T17:45:03.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.228+0000 7fc9d6678640 1 -- 172.21.15.98:0/2458015795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d01590e0 con 0x7fc9d006b160 2023-11-28T17:45:03.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.228+0000 7fc9c5ffb640 1 -- 172.21.15.98:0/2458015795 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c0012870 con 0x7fc9d006b160 2023-11-28T17:45:03.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.228+0000 7fc9d6678640 1 -- 172.21.15.98:0/2458015795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d0158a00 con 0x7fc9d006b160 2023-11-28T17:45:03.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.229+0000 7fc9d6678640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d0159270 0x7fc9d015a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.229+0000 7fc9d6678640 1 -- 172.21.15.98:0/2458015795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d0074310 con 0x7fc9d0159270 2023-11-28T17:45:03.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.229+0000 7fc9d5676640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d0159270 0x7fc9d015a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.229+0000 7fc9d5676640 1 -- 172.21.15.98:0/2458015795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9d006cb90 con 0x7fc9d0159270 2023-11-28T17:45:03.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.230+0000 7fc9c5ffb640 1 -- 172.21.15.98:0/2458015795 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fc9c00129d0 con 0x7fc9d006b160 2023-11-28T17:45:03.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.230+0000 7fc9d5676640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d0159270 0x7fc9d015a070 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc011830 tx=0x7fc9cc011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:03.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.231+0000 7fc9c5ffb640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc9b406b2a0 0x7fc9b406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.231+0000 7fc9c5ffb640 1 -- 172.21.15.98:0/2458015795 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc9c0089780 con 0x7fc9d006b160 2023-11-28T17:45:03.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.231+0000 7fc9d5e77640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc9b406b2a0 0x7fc9b406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.231+0000 7fc9c5ffb640 1 -- 172.21.15.98:0/2458015795 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc00fa30 con 0x7fc9d0159270 2023-11-28T17:45:03.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.231+0000 7fc9c5ffb640 1 -- 172.21.15.98:0/2458015795 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fc9cc00fb90 con 0x7fc9d0159270 2023-11-28T17:45:03.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.231+0000 7fc9c5ffb640 1 -- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d0159270 msgr2=0x7fc9d015a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.231+0000 7fc9c5ffb640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d0159270 0x7fc9d015a070 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc011830 tx=0x7fc9cc011cf0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.231+0000 7fc9c5ffb640 1 -- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d0159270 msgr2=0x7fc9d015a070 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:03.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.231+0000 7fc9c5ffb640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d0159270 0x7fc9d015a070 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc011830 tx=0x7fc9cc011cf0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.232+0000 7fc9d5e77640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc9b406b2a0 0x7fc9b406d760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8036000 tx=0x7fc9c8030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.238+0000 7fc9d6678640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d014c5f0 0x7fc9d014ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.238+0000 7fc9d6678640 1 -- 172.21.15.98:0/2458015795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d006c8b0 con 0x7fc9d014c5f0 2023-11-28T17:45:03.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.239+0000 7fc9d5676640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d014c5f0 0x7fc9d014ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.239+0000 7fc9d5676640 1 -- 172.21.15.98:0/2458015795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc9d006ce70 con 0x7fc9d014c5f0 2023-11-28T17:45:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.239+0000 7fc9d5676640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d014c5f0 0x7fc9d014ca00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc011830 tx=0x7fc9cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:03.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.239+0000 7fc9c5ffb640 1 -- 172.21.15.98:0/2458015795 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc021040 con 0x7fc9d014c5f0 2023-11-28T17:45:03.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.240+0000 7fc9c5ffb640 1 -- 172.21.15.98:0/2458015795 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4468 (secure 0 0 0) 0x7fc9cc026070 con 0x7fc9d014c5f0 2023-11-28T17:45:03.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.240+0000 7fc9c5ffb640 1 -- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d014c5f0 msgr2=0x7fc9d014ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.240+0000 7fc9c5ffb640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d014c5f0 0x7fc9d014ca00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc011830 tx=0x7fc9cc0046c0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.240+0000 7fc9c5ffb640 1 -- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d014c5f0 msgr2=0x7fc9d014ca00 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:03.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.240+0000 7fc9c5ffb640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d014c5f0 0x7fc9d014ca00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.241 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:45:03.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:45:03.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:45:03.242 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:45:03.242 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 36, 2023-11-28T17:45:03.242 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:45:03.242 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:45:03.242 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:45:03.242 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:45:03.242 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:03.242 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 12, 2023-11-28T17:45:03.242 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:03.242 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:45:03.242 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:45:03.243 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:03.243 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:03.243 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:03.243 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:03.243 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:03.243 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:03.243 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:03.243 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:03.243 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:03.243 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:03.243 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:03.243 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:45:03.243 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:45:03.243 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:03.244 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:03.244 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:03.244 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:03.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:03.244 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:03.244 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:03.244 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:03.244 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:03.244 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:03.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.244 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:45:03.244 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:45:03.245 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:45:03.245 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:45:03.245 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:45:03.245 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:45:03.245 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:03.245 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:03.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:45:03.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:45:03.245 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:45:03.245 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:45:03.245 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:45:03.245 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:45:03.245 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:45:03.246 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:03.246 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:45:03.246 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:03.246 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:03.246 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:03.246 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:03.246 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:03.246 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:03.246 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:03.246 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:03.246 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:03.247 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:03.247 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:03.247 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:45:03.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.247 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:45:03.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:45:03.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:45:03.247 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:03.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:03.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:45:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:45:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:45:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:45:03.248 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:03.249 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:03.249 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:03.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:45:03.249 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:45:03.249 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:03.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:03.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:03.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:45:03.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:03.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:45:03.250 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.250 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.250 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:45:03.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:45:03.251 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:03.251 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:03.251 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:03.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:45:03.251 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:45:03.251 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:03.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:03.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:03.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:45:03.252 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:03.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:45:03.252 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.252 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.252 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:45:03.253 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:45:03.253 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:03.253 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:03.253 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:03.253 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.253 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.254 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:45:03.254 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:45:03.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.254 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:03.254 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:03.254 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:45:03.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.254 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:03.254 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:45:03.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.254 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:03.255 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:03.255 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:45:03.255 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.255 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.255 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:45:03.255 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:45:03.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.240+0000 7fc9d6678640 1 -- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc9b406b2a0 msgr2=0x7fc9b406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.240+0000 7fc9d6678640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc9b406b2a0 0x7fc9b406d760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8036000 tx=0x7fc9c8030000 comp rx=0 tx=0).stop 2023-11-28T17:45:03.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.240+0000 7fc9d6678640 1 -- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d006b160 msgr2=0x7fc9d015cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.240+0000 7fc9d6678640 1 --2- 172.21.15.98:0/2458015795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9d006b160 0x7fc9d015cc20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc9c000c480 tx=0x7fc9c000c940 comp rx=0 tx=0).stop 2023-11-28T17:45:03.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.240+0000 7fc9d5e77640 1 -- 172.21.15.98:0/2458015795 reap_dead start 2023-11-28T17:45:03.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.240+0000 7fc9d5e77640 1 -- 172.21.15.98:0/2458015795 reap_dead start 2023-11-28T17:45:03.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.240+0000 7fc9d6678640 1 -- 172.21.15.98:0/2458015795 shutdown_connections 2023-11-28T17:45:03.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.240+0000 7fc9d6678640 1 -- 172.21.15.98:0/2458015795 >> 172.21.15.98:0/2458015795 conn(0x7fc9d00cafe0 msgr2=0x7fc9d00ca820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:03.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.240+0000 7fc9d6678640 1 -- 172.21.15.98:0/2458015795 shutdown_connections 2023-11-28T17:45:03.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.240+0000 7fc9d6678640 1 -- 172.21.15.98:0/2458015795 wait complete. 2023-11-28T17:45:03.259 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:45:03.259 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:45:03.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:45:03.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.463+0000 7fa5c781c640 1 Processor -- start 2023-11-28T17:45:03.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.464+0000 7fa5c781c640 1 -- start start 2023-11-28T17:45:03.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c781c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa5c0153fb0 0x7fa5c00c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c781c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa5c00b5270 0x7fa5c00ca300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c781c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c00ca840 0x7fa5c00c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c781c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c00cb730 con 0x7fa5c00b5270 2023-11-28T17:45:03.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c781c640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c015db80 con 0x7fa5c0153fb0 2023-11-28T17:45:03.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c781c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c015e0c0 con 0x7fa5c00ca840 2023-11-28T17:45:03.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c4db7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa5c00b5270 0x7fa5c00ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c4db7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa5c00b5270 0x7fa5c00ca300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34042/0 (socket says 172.21.15.98:34042) 2023-11-28T17:45:03.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c4db7640 1 -- 172.21.15.98:0/4075856312 learned_addr learned my addr 172.21.15.98:0/4075856312 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:03.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c55b8640 1 --2- 172.21.15.98:0/4075856312 >> v2:172.21.15.113:3300/0 conn(0x7fa5c0153fb0 0x7fa5c00c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c5db9640 1 --2- 172.21.15.98:0/4075856312 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c00ca840 0x7fa5c00c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c4db7640 1 -- 172.21.15.98:0/4075856312 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c00ca840 msgr2=0x7fa5c00c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:03.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c4db7640 1 --2- 172.21.15.98:0/4075856312 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c00ca840 0x7fa5c00c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c4db7640 1 -- 172.21.15.98:0/4075856312 >> v2:172.21.15.113:3300/0 conn(0x7fa5c0153fb0 msgr2=0x7fa5c00c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:03.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c4db7640 1 --2- 172.21.15.98:0/4075856312 >> v2:172.21.15.113:3300/0 conn(0x7fa5c0153fb0 0x7fa5c00c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c4db7640 1 -- 172.21.15.98:0/4075856312 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c00d00a0 con 0x7fa5c00b5270 2023-11-28T17:45:03.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c5db9640 1 --2- 172.21.15.98:0/4075856312 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c00ca840 0x7fa5c00c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:45:03.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.465+0000 7fa5c55b8640 1 --2- 172.21.15.98:0/4075856312 >> v2:172.21.15.113:3300/0 conn(0x7fa5c0153fb0 0x7fa5c00c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:45:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.466+0000 7fa5c4db7640 1 --2- 172.21.15.98:0/4075856312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c00b5270 0x7fa5c00ca300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00742d0 tx=0x7fa5ac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1cda2a8ffb03303 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:03.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.467+0000 7fa5bffff640 1 -- 172.21.15.98:0/4075856312 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac0317e0 con 0x7fa5c00b5270 2023-11-28T17:45:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.467+0000 7fa5bffff640 1 -- 172.21.15.98:0/4075856312 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ac031940 con 0x7fa5c00b5270 2023-11-28T17:45:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.467+0000 7fa5bffff640 1 -- 172.21.15.98:0/4075856312 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac031cd0 con 0x7fa5c00b5270 2023-11-28T17:45:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.467+0000 7fa5c781c640 1 -- 172.21.15.98:0/4075856312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c00b5270 msgr2=0x7fa5c00ca300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.467+0000 7fa5c781c640 1 --2- 172.21.15.98:0/4075856312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c00b5270 0x7fa5c00ca300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00742d0 tx=0x7fa5ac02b880 comp rx=0 tx=0).stop 2023-11-28T17:45:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.468+0000 7fa5c781c640 1 -- 172.21.15.98:0/4075856312 shutdown_connections 2023-11-28T17:45:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.468+0000 7fa5c781c640 1 --2- 172.21.15.98:0/4075856312 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c00ca840 0x7fa5c00c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.468+0000 7fa5c781c640 1 --2- 172.21.15.98:0/4075856312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c00b5270 0x7fa5c00ca300 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.468+0000 7fa5c781c640 1 --2- 172.21.15.98:0/4075856312 >> v2:172.21.15.113:3300/0 conn(0x7fa5c0153fb0 0x7fa5c00c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.468+0000 7fa5c781c640 1 -- 172.21.15.98:0/4075856312 >> 172.21.15.98:0/4075856312 conn(0x7fa5c00cf4a0 msgr2=0x7fa5c00cf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.468+0000 7fa5c781c640 1 -- 172.21.15.98:0/4075856312 shutdown_connections 2023-11-28T17:45:03.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.468+0000 7fa5c781c640 1 -- 172.21.15.98:0/4075856312 wait complete. 2023-11-28T17:45:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.469+0000 7fa5c781c640 1 Processor -- start 2023-11-28T17:45:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.469+0000 7fa5c781c640 1 -- start start 2023-11-28T17:45:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.470+0000 7fa5c781c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00ca840 0x7fa5c015b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.470+0000 7fa5c781c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c0153fb0 0x7fa5c00ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.470+0000 7fa5c781c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c015bae0 0x7fa5c00ceb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.470+0000 7fa5c781c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c01562d0 con 0x7fa5c0153fb0 2023-11-28T17:45:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.470+0000 7fa5c781c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c015c050 con 0x7fa5c00ca840 2023-11-28T17:45:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.470+0000 7fa5c781c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c015c330 con 0x7fa5c015bae0 2023-11-28T17:45:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.471+0000 7fa5c4db7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c0153fb0 0x7fa5c00ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.471+0000 7fa5c4db7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c0153fb0 0x7fa5c00ce5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34050/0 (socket says 172.21.15.98:34050) 2023-11-28T17:45:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.471+0000 7fa5c4db7640 1 -- 172.21.15.98:0/3827150894 learned_addr learned my addr 172.21.15.98:0/3827150894 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:03.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.471+0000 7fa5c5db9640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c015bae0 0x7fa5c00ceb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.471+0000 7fa5c55b8640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00ca840 0x7fa5c015b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.471+0000 7fa5c4db7640 1 -- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c015bae0 msgr2=0x7fa5c00ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.471+0000 7fa5c4db7640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c015bae0 0x7fa5c00ceb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.471+0000 7fa5c4db7640 1 -- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00ca840 msgr2=0x7fa5c015b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.471+0000 7fa5c4db7640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00ca840 0x7fa5c015b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.471+0000 7fa5c4db7640 1 -- 172.21.15.98:0/3827150894 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ac0094d0 con 0x7fa5c0153fb0 2023-11-28T17:45:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.471+0000 7fa5c4db7640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c0153fb0 0x7fa5c00ce5e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac02b8c0 tx=0x7fa5ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.472+0000 7fa5bdffb640 1 -- 172.21.15.98:0/3827150894 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac032980 con 0x7fa5c0153fb0 2023-11-28T17:45:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.472+0000 7fa5bdffb640 1 -- 172.21.15.98:0/3827150894 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ac032b00 con 0x7fa5c0153fb0 2023-11-28T17:45:03.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.472+0000 7fa5bdffb640 1 -- 172.21.15.98:0/3827150894 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac0453f0 con 0x7fa5c0153fb0 2023-11-28T17:45:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.472+0000 7fa5c781c640 1 -- 172.21.15.98:0/3827150894 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c015c610 con 0x7fa5c0153fb0 2023-11-28T17:45:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.472+0000 7fa5c781c640 1 -- 172.21.15.98:0/3827150894 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c015cbd0 con 0x7fa5c0153fb0 2023-11-28T17:45:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.473+0000 7fa5c781c640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00b5270 0x7fa5c00cd560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.473+0000 7fa5c781c640 1 -- 172.21.15.98:0/3827150894 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c0074110 con 0x7fa5c00b5270 2023-11-28T17:45:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.473+0000 7fa5c55b8640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00b5270 0x7fa5c00cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.474+0000 7fa5c55b8640 1 -- 172.21.15.98:0/3827150894 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5c015dde0 con 0x7fa5c00b5270 2023-11-28T17:45:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.474+0000 7fa5bdffb640 1 -- 172.21.15.98:0/3827150894 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa5ac045550 con 0x7fa5c0153fb0 2023-11-28T17:45:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.474+0000 7fa5c55b8640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00b5270 0x7fa5c00cd560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4007f90 tx=0x7fa5b400c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:03.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.475+0000 7fa5bdffb640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa5a406b2d0 0x7fa5a406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.475+0000 7fa5bdffb640 1 -- 172.21.15.98:0/3827150894 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa5ac0ac250 con 0x7fa5c0153fb0 2023-11-28T17:45:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.475+0000 7fa5c5db9640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa5a406b2d0 0x7fa5a406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.475+0000 7fa5bdffb640 1 -- 172.21.15.98:0/3827150894 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4003b90 con 0x7fa5c00b5270 2023-11-28T17:45:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.476+0000 7fa5bdffb640 1 -- 172.21.15.98:0/3827150894 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fa5b4003cf0 con 0x7fa5c00b5270 2023-11-28T17:45:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.476+0000 7fa5bdffb640 1 -- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00b5270 msgr2=0x7fa5c00cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.476+0000 7fa5bdffb640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00b5270 0x7fa5c00cd560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4007f90 tx=0x7fa5b400c850 comp rx=0 tx=0).stop 2023-11-28T17:45:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.476+0000 7fa5bdffb640 1 -- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00b5270 msgr2=0x7fa5c00cd560 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.476+0000 7fa5bdffb640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00b5270 0x7fa5c00cd560 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4007f90 tx=0x7fa5b400c850 comp rx=0 tx=0).stop 2023-11-28T17:45:03.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.476+0000 7fa5c5db9640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa5a406b2d0 0x7fa5a406d790 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8006fd0 tx=0x7fa5b8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.483+0000 7fa5c781c640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00cdaa0 0x7fa5c01f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.483+0000 7fa5c781c640 1 -- 172.21.15.98:0/3827150894 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00cb1b0 con 0x7fa5c00cdaa0 2023-11-28T17:45:03.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.483+0000 7fa5c55b8640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00cdaa0 0x7fa5c01f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.484+0000 7fa5c55b8640 1 -- 172.21.15.98:0/3827150894 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa5c00cb450 con 0x7fa5c00cdaa0 2023-11-28T17:45:03.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.485+0000 7fa5c55b8640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00cdaa0 0x7fa5c01f2c30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4007f90 tx=0x7fa5b4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:03.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.485+0000 7fa5bdffb640 1 -- 172.21.15.98:0/3827150894 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b40048e0 con 0x7fa5c00cdaa0 2023-11-28T17:45:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.486+0000 7fa5bdffb640 1 -- 172.21.15.98:0/3827150894 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4466 (secure 0 0 0) 0x7fa5b4023070 con 0x7fa5c00cdaa0 2023-11-28T17:45:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.486+0000 7fa5bdffb640 1 -- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00cdaa0 msgr2=0x7fa5c01f2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.486+0000 7fa5bdffb640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00cdaa0 0x7fa5c01f2c30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4007f90 tx=0x7fa5b4004a40 comp rx=0 tx=0).stop 2023-11-28T17:45:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.486+0000 7fa5bdffb640 1 -- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00cdaa0 msgr2=0x7fa5c01f2c30 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:03.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.486+0000 7fa5bdffb640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c00cdaa0 0x7fa5c01f2c30 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4007f90 tx=0x7fa5b4004a40 comp rx=0 tx=0).stop 2023-11-28T17:45:03.487 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:45:03.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:45:03.488 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:45:03.488 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:45:03.488 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 36, 2023-11-28T17:45:03.488 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:45:03.488 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:45:03.488 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:45:03.488 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:45:03.488 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:03.488 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 12, 2023-11-28T17:45:03.488 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:03.489 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:45:03.489 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:45:03.489 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:03.489 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:03.489 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:03.489 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:03.489 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:03.489 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:03.489 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:03.489 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:03.489 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:03.489 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:03.490 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:03.490 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:45:03.490 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:45:03.490 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:03.490 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:03.490 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:03.490 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:03.490 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:03.490 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:03.490 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:03.491 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:03.491 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:03.491 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:03.491 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.491 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:45:03.491 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:45:03.491 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:45:03.491 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:45:03.491 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:45:03.491 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:45:03.492 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:03.492 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:03.492 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:45:03.492 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:45:03.492 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:45:03.492 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:45:03.492 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:45:03.492 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:45:03.492 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:45:03.492 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:03.492 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:45:03.493 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:03.493 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:03.493 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:03.493 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:03.493 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:03.493 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:03.493 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:03.493 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:03.493 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:03.493 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:03.493 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:03.493 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:45:03.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.494 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:45:03.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.494 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:45:03.494 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:45:03.494 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:03.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:03.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:03.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:45:03.494 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:03.495 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:45:03.495 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.495 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.495 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:45:03.495 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:45:03.495 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:03.496 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:03.496 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:03.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.496 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:45:03.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:45:03.496 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:03.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:03.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:03.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:45:03.496 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:03.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:45:03.497 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.497 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.497 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:45:03.497 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:45:03.498 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:03.498 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:03.498 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:03.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.498 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:45:03.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:45:03.498 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:03.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:03.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:45:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:45:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:45:03.499 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:45:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:45:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:45:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:03.500 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:45:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:45:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:45:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.502 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:45:03.502 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:45:03.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.486+0000 7fa5c781c640 1 -- 172.21.15.98:0/3827150894 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa5a406b2d0 msgr2=0x7fa5a406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.486+0000 7fa5c781c640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa5a406b2d0 0x7fa5a406d790 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8006fd0 tx=0x7fa5b8008040 comp rx=0 tx=0).stop 2023-11-28T17:45:03.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.487+0000 7fa5c781c640 1 -- 172.21.15.98:0/3827150894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c0153fb0 msgr2=0x7fa5c00ce5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.487+0000 7fa5c781c640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c0153fb0 0x7fa5c00ce5e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac02b8c0 tx=0x7fa5ac0046c0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.487+0000 7fa5c5db9640 1 -- 172.21.15.98:0/3827150894 reap_dead start 2023-11-28T17:45:03.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.487+0000 7fa5c781c640 1 -- 172.21.15.98:0/3827150894 shutdown_connections 2023-11-28T17:45:03.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.487+0000 7fa5c781c640 1 --2- 172.21.15.98:0/3827150894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c0153fb0 0x7fa5c00ce5e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.487+0000 7fa5c781c640 1 -- 172.21.15.98:0/3827150894 >> 172.21.15.98:0/3827150894 conn(0x7fa5c00cf4a0 msgr2=0x7fa5c00cbbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:03.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.487+0000 7fa5c781c640 1 -- 172.21.15.98:0/3827150894 shutdown_connections 2023-11-28T17:45:03.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.487+0000 7fa5c781c640 1 -- 172.21.15.98:0/3827150894 wait complete. 2023-11-28T17:45:03.507 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:45:03.507 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:45:03.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:45:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.712+0000 7fa07bc09640 1 Processor -- start 2023-11-28T17:45:03.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.713+0000 7fa07bc09640 1 -- start start 2023-11-28T17:45:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.714+0000 7fa07bc09640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa0740c5a60 0x7fa0740c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.714+0000 7fa07bc09640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0740c8150 con 0x7fa0741584d0 2023-11-28T17:45:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.714+0000 7fa07bc09640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0740c8850 con 0x7fa074151be0 2023-11-28T17:45:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.714+0000 7fa07bc09640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0740c5f50 con 0x7fa0740c5a60 2023-11-28T17:45:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.714+0000 7fa0791a4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa0740c5a60 0x7fa0740c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.714+0000 7fa07a1a6640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa0741584d0 0x7fa07406dd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49710/0 (socket says 172.21.15.98:49710) 2023-11-28T17:45:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.714+0000 7fa07a1a6640 1 -- 172.21.15.98:0/2224247182 learned_addr learned my addr 172.21.15.98:0/2224247182 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:45:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.715+0000 7fa0791a4640 1 -- 172.21.15.98:0/2224247182 >> v1:172.21.15.113:6789/0 conn(0x7fa074151be0 legacy=0x7fa0740c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.715+0000 7fa0791a4640 1 -- 172.21.15.98:0/2224247182 >> v1:172.21.15.98:6789/0 conn(0x7fa0741584d0 legacy=0x7fa07406dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.715+0000 7fa0791a4640 1 -- 172.21.15.98:0/2224247182 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0740c7b90 con 0x7fa0740c5a60 2023-11-28T17:45:03.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.715+0000 7fa0789a3640 1 -- 172.21.15.98:0/2224247182 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2664376386 0 0) 0x7fa0740c8150 con 0x7fa0741584d0 2023-11-28T17:45:03.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.716+0000 7fa0791a4640 1 --2- 172.21.15.98:0/2224247182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0740c5a60 0x7fa0740c75a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa0680097a0 tx=0x7fa06802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c2493f5da60d86 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:03.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.716+0000 7fa0789a3640 1 -- 172.21.15.98:0/2224247182 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa068031830 con 0x7fa0740c5a60 2023-11-28T17:45:03.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.716+0000 7fa0789a3640 1 -- 172.21.15.98:0/2224247182 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa068031990 con 0x7fa0740c5a60 2023-11-28T17:45:03.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.716+0000 7fa0789a3640 1 -- 172.21.15.98:0/2224247182 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa068031d20 con 0x7fa0740c5a60 2023-11-28T17:45:03.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.717+0000 7fa07bc09640 1 -- 172.21.15.98:0/2224247182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0740c5a60 msgr2=0x7fa0740c75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.717+0000 7fa07bc09640 1 --2- 172.21.15.98:0/2224247182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0740c5a60 0x7fa0740c75a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa0680097a0 tx=0x7fa06802b8d0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.717+0000 7fa07bc09640 1 -- 172.21.15.98:0/2224247182 shutdown_connections 2023-11-28T17:45:03.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.717+0000 7fa07bc09640 1 --2- 172.21.15.98:0/2224247182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0740c5a60 0x7fa0740c75a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.717+0000 7fa07bc09640 1 -- 172.21.15.98:0/2224247182 >> 172.21.15.98:0/2224247182 conn(0x7fa07406c0f0 msgr2=0x7fa07406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:03.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.717+0000 7fa07bc09640 1 -- 172.21.15.98:0/2224247182 shutdown_connections 2023-11-28T17:45:03.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.717+0000 7fa07bc09640 1 -- 172.21.15.98:0/2224247182 wait complete. 2023-11-28T17:45:03.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.718+0000 7fa07bc09640 1 Processor -- start 2023-11-28T17:45:03.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.719+0000 7fa07bc09640 1 -- start start 2023-11-28T17:45:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.719+0000 7fa07bc09640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0740c5a60 0x7fa0741ed4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.720+0000 7fa07bc09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa074151be0 0x7fa07406a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.720+0000 7fa07bc09640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0741584d0 0x7fa0741f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.720+0000 7fa07bc09640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0741efd60 con 0x7fa074151be0 2023-11-28T17:45:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.720+0000 7fa07bc09640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07415bc60 con 0x7fa0740c5a60 2023-11-28T17:45:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.720+0000 7fa07bc09640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0741ef480 con 0x7fa0741584d0 2023-11-28T17:45:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.720+0000 7fa0791a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa074151be0 0x7fa07406a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.720+0000 7fa0791a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa074151be0 0x7fa07406a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34072/0 (socket says 172.21.15.98:34072) 2023-11-28T17:45:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.720+0000 7fa0791a4640 1 -- 172.21.15.98:0/2330651269 learned_addr learned my addr 172.21.15.98:0/2330651269 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.720+0000 7fa0799a5640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0740c5a60 0x7fa0741ed4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.720+0000 7fa07a1a6640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0741584d0 0x7fa0741f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.720+0000 7fa0791a4640 1 -- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0741584d0 msgr2=0x7fa0741f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.720+0000 7fa0791a4640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0741584d0 0x7fa0741f2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.720+0000 7fa0791a4640 1 -- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0740c5a60 msgr2=0x7fa0741ed4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.720+0000 7fa0791a4640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0740c5a60 0x7fa0741ed4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.720+0000 7fa0791a4640 1 -- 172.21.15.98:0/2330651269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa068009400 con 0x7fa074151be0 2023-11-28T17:45:03.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.721+0000 7fa0791a4640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa074151be0 0x7fa07406a740 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa0680098c0 tx=0x7fa0680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:03.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.721+0000 7fa0727fc640 1 -- 172.21.15.98:0/2330651269 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06803aa60 con 0x7fa074151be0 2023-11-28T17:45:03.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.721+0000 7fa0727fc640 1 -- 172.21.15.98:0/2330651269 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06803abc0 con 0x7fa074151be0 2023-11-28T17:45:03.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.721+0000 7fa0727fc640 1 -- 172.21.15.98:0/2330651269 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0680443f0 con 0x7fa074151be0 2023-11-28T17:45:03.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.721+0000 7fa07bc09640 1 -- 172.21.15.98:0/2330651269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0741eda20 con 0x7fa074151be0 2023-11-28T17:45:03.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.721+0000 7fa07bc09640 1 -- 172.21.15.98:0/2330651269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0741f3670 con 0x7fa074151be0 2023-11-28T17:45:03.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.722+0000 7fa07bc09640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0740bcc90 0x7fa0741ef600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.722+0000 7fa07bc09640 1 -- 172.21.15.98:0/2330651269 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740744d0 con 0x7fa0740bcc90 2023-11-28T17:45:03.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.722+0000 7fa0799a5640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0740bcc90 0x7fa0741ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.723+0000 7fa0799a5640 1 -- 172.21.15.98:0/2330651269 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa074158920 con 0x7fa0740bcc90 2023-11-28T17:45:03.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.723+0000 7fa0727fc640 1 -- 172.21.15.98:0/2330651269 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa068044550 con 0x7fa074151be0 2023-11-28T17:45:03.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.724+0000 7fa0799a5640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0740bcc90 0x7fa0741ef600 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa06400e7c0 tx=0x7fa06400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:03.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.724+0000 7fa0727fc640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa06006b2a0 0x7fa06006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.724+0000 7fa0727fc640 1 -- 172.21.15.98:0/2330651269 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa0680abfe0 con 0x7fa074151be0 2023-11-28T17:45:03.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.724+0000 7fa07a1a6640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa06006b2a0 0x7fa06006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.724+0000 7fa0727fc640 1 -- 172.21.15.98:0/2330651269 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa064019070 con 0x7fa0740bcc90 2023-11-28T17:45:03.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.725+0000 7fa0727fc640 1 -- 172.21.15.98:0/2330651269 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fa06400eed0 con 0x7fa0740bcc90 2023-11-28T17:45:03.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.725+0000 7fa0727fc640 1 -- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0740bcc90 msgr2=0x7fa0741ef600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.725+0000 7fa0727fc640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0740bcc90 0x7fa0741ef600 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa06400e7c0 tx=0x7fa06400ec80 comp rx=0 tx=0).stop 2023-11-28T17:45:03.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.725+0000 7fa0727fc640 1 -- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0740bcc90 msgr2=0x7fa0741ef600 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:03.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.725+0000 7fa0727fc640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa0740bcc90 0x7fa0741ef600 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa06400e7c0 tx=0x7fa06400ec80 comp rx=0 tx=0).stop 2023-11-28T17:45:03.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.725+0000 7fa07a1a6640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa06006b2a0 0x7fa06006d760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa06c000f60 tx=0x7fa06c007ad0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:03.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.732+0000 7fa07bc09640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa074071240 0x7fa07414e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:03.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.732+0000 7fa07bc09640 1 -- 172.21.15.98:0/2330651269 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740681b0 con 0x7fa074071240 2023-11-28T17:45:03.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.732+0000 7fa0799a5640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa074071240 0x7fa07414e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:03.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.733+0000 7fa0799a5640 1 -- 172.21.15.98:0/2330651269 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa0740c84e0 con 0x7fa074071240 2023-11-28T17:45:03.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.733+0000 7fa0799a5640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa074071240 0x7fa07414e340 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa064018000 tx=0x7fa064003ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:03.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.734+0000 7fa0727fc640 1 -- 172.21.15.98:0/2330651269 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06401ea40 con 0x7fa074071240 2023-11-28T17:45:03.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.734+0000 7fa0727fc640 1 -- 172.21.15.98:0/2330651269 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4976 (secure 0 0 0) 0x7fa064022070 con 0x7fa074071240 2023-11-28T17:45:03.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.734+0000 7fa0727fc640 1 -- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa074071240 msgr2=0x7fa07414e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.734+0000 7fa0727fc640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa074071240 0x7fa07414e340 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa064018000 tx=0x7fa064003ae0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.734+0000 7fa0727fc640 1 -- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa074071240 msgr2=0x7fa07414e340 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:03.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.734+0000 7fa0727fc640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa074071240 0x7fa07414e340 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.736 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:45:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:45:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:45:03.736 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:45:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 36, 2023-11-28T17:45:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:45:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:45:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:45:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:45:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 12, 2023-11-28T17:45:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:45:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:45:03.737 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:03.738 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:45:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:45:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:03.739 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:45:03.740 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:45:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:45:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:45:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:45:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:45:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:45:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:45:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:45:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:45:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:45:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:45:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:45:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:45:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:03.742 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:45:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.743 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:45:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:45:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:45:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:45:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:45:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:45:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:45:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:03.745 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:03.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.746 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:45:03.746 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:45:03.746 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:03.746 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:03.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.746 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:03.746 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:45:03.747 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.747 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:03.747 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:45:03.747 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.747 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.747 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.747 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:45:03.748 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:45:03.748 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:03.748 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:03.748 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:03.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.748 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:45:03.748 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:45:03.748 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:03.748 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:03.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:03.749 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:45:03.749 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.749 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:03.749 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:45:03.749 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:03.749 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.749 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.749 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:45:03.749 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:45:03.750 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:03.750 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:03.750 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:03.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.750 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.750 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:45:03.750 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:45:03.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.750 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:03.750 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:03.751 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:45:03.751 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.751 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:03.751 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:45:03.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.751 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:03.751 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:03.751 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:45:03.751 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.751 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:03.751 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:45:03.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.751 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:03.752 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:03.752 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 8 2023-11-28T17:45:03.752 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.752 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:03.752 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:45:03.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.752 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:03.752 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:03.752 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:45:03.752 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.752 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:03.753 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:45:03.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:03.753 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:03.753 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:03.753 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:45:03.753 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:03.753 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:03.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:03.753 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:45:03.753 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:45:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.735+0000 7fa07bc09640 1 -- 172.21.15.98:0/2330651269 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa06006b2a0 msgr2=0x7fa06006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.735+0000 7fa07bc09640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa06006b2a0 0x7fa06006d760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa06c000f60 tx=0x7fa06c007ad0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.735+0000 7fa07bc09640 1 -- 172.21.15.98:0/2330651269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa074151be0 msgr2=0x7fa07406a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.735+0000 7fa07bc09640 1 --2- 172.21.15.98:0/2330651269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa074151be0 0x7fa07406a740 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa0680098c0 tx=0x7fa0680046c0 comp rx=0 tx=0).stop 2023-11-28T17:45:03.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.735+0000 7fa07a1a6640 1 -- 172.21.15.98:0/2330651269 reap_dead start 2023-11-28T17:45:03.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.735+0000 7fa07a1a6640 1 -- 172.21.15.98:0/2330651269 reap_dead start 2023-11-28T17:45:03.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.735+0000 7fa07bc09640 1 -- 172.21.15.98:0/2330651269 shutdown_connections 2023-11-28T17:45:03.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.735+0000 7fa07bc09640 1 -- 172.21.15.98:0/2330651269 >> 172.21.15.98:0/2330651269 conn(0x7fa07406c0f0 msgr2=0x7fa074156270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:03.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.735+0000 7fa07bc09640 1 -- 172.21.15.98:0/2330651269 shutdown_connections 2023-11-28T17:45:03.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:03.735+0000 7fa07bc09640 1 -- 172.21.15.98:0/2330651269 wait complete. 2023-11-28T17:45:03.756 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:45:03.757 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-11-28T17:45:03.757 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-11-28T17:45:03.757 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-11-28T17:45:03.757 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-11-28T17:45:03.757 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-28T17:45:03.757 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:45:03.799 INFO:tasks.ceph.mon.b:Stopped 2023-11-28T17:45:03.799 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-11-28T17:45:03.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:45:04.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.013+0000 7f43d4589640 1 Processor -- start 2023-11-28T17:45:04.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.014+0000 7f43d4589640 1 -- start start 2023-11-28T17:45:04.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.015+0000 7f43d4589640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43cc157b60 0x7f43cc15bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:04.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.015+0000 7f43d4589640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43cc15b150 con 0x7f43cc0c4910 2023-11-28T17:45:04.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.015+0000 7f43d4589640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43cc06e1f0 con 0x7f43cc151d80 2023-11-28T17:45:04.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.015+0000 7f43d4589640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43cc157f90 con 0x7f43cc157b60 2023-11-28T17:45:04.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.015+0000 7f43d2325640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f43cc151d80 legacy=0x7f43cc0c5c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:45:04.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.016+0000 7f43d1b24640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43cc157b60 0x7f43cc15bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:04.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.016+0000 7f43d2b26640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f43cc0c4910 0x7f43cc1624e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49712/0 (socket says 172.21.15.98:49712) 2023-11-28T17:45:04.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.016+0000 7f43d2b26640 1 -- 172.21.15.98:0/1781032487 learned_addr learned my addr 172.21.15.98:0/1781032487 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:45:04.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.016+0000 7f43d1b24640 1 -- 172.21.15.98:0/1781032487 >> v1:172.21.15.113:6789/0 conn(0x7f43cc151d80 legacy=0x7f43cc0c5c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:04.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.016+0000 7f43d1b24640 1 -- 172.21.15.98:0/1781032487 >> v1:172.21.15.98:6789/0 conn(0x7f43cc0c4910 legacy=0x7f43cc1624e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:04.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.016+0000 7f43d1b24640 1 -- 172.21.15.98:0/1781032487 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43cc06e4d0 con 0x7f43cc157b60 2023-11-28T17:45:04.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.016+0000 7f43d1323640 1 -- 172.21.15.98:0/1781032487 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476164157 0 0) 0x7f43cc15b150 con 0x7f43cc0c4910 2023-11-28T17:45:04.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.017+0000 7f43d1b24640 1 --2- 172.21.15.98:0/1781032487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc157b60 0x7f43cc15bc10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f43bc0097a0 tx=0x7f43bc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=949c1b75d72fb617 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:04.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.017+0000 7f43d1323640 1 -- 172.21.15.98:0/1781032487 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc030610 con 0x7f43cc157b60 2023-11-28T17:45:04.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.018+0000 7f43d1323640 1 -- 172.21.15.98:0/1781032487 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43bc030770 con 0x7f43cc157b60 2023-11-28T17:45:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.018+0000 7f43d1323640 1 -- 172.21.15.98:0/1781032487 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc030b00 con 0x7f43cc157b60 2023-11-28T17:45:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.018+0000 7f43d4589640 1 -- 172.21.15.98:0/1781032487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc157b60 msgr2=0x7f43cc15bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.018+0000 7f43d4589640 1 --2- 172.21.15.98:0/1781032487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc157b60 0x7f43cc15bc10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f43bc0097a0 tx=0x7f43bc02b960 comp rx=0 tx=0).stop 2023-11-28T17:45:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.018+0000 7f43d4589640 1 -- 172.21.15.98:0/1781032487 shutdown_connections 2023-11-28T17:45:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.018+0000 7f43d4589640 1 --2- 172.21.15.98:0/1781032487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc157b60 0x7f43cc15bc10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.018+0000 7f43d4589640 1 -- 172.21.15.98:0/1781032487 >> 172.21.15.98:0/1781032487 conn(0x7f43cc06a4d0 msgr2=0x7f43cc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.018+0000 7f43d4589640 1 -- 172.21.15.98:0/1781032487 shutdown_connections 2023-11-28T17:45:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.018+0000 7f43d4589640 1 -- 172.21.15.98:0/1781032487 wait complete. 2023-11-28T17:45:04.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.020+0000 7f43d4589640 1 Processor -- start 2023-11-28T17:45:04.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.020+0000 7f43d4589640 1 -- start start 2023-11-28T17:45:04.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.021+0000 7f43d4589640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43cc0c4910 0x7f43cc1ef320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:04.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.021+0000 7f43d4589640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc151d80 0x7f43cc1f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:04.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.021+0000 7f43d4589640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43cc157b60 0x7f43cc154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:04.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.021+0000 7f43d4589640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc155440 con 0x7f43cc151d80 2023-11-28T17:45:04.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.021+0000 7f43d4589640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc1f2f60 con 0x7f43cc0c4910 2023-11-28T17:45:04.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.021+0000 7f43d4589640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc157220 con 0x7f43cc157b60 2023-11-28T17:45:04.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.021+0000 7f43d1b24640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc151d80 0x7f43cc1f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:04.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.021+0000 7f43d2325640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43cc0c4910 msgr2=0x7f43cc1ef320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:04.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.021+0000 7f43d2325640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43cc0c4910 0x7f43cc1ef320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:04.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.021+0000 7f43d1b24640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc151d80 0x7f43cc1f33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34102/0 (socket says 172.21.15.98:34102) 2023-11-28T17:45:04.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.021+0000 7f43d1b24640 1 -- 172.21.15.98:0/3248375724 learned_addr learned my addr 172.21.15.98:0/3248375724 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:04.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.022+0000 7f43d2b26640 1 --2- 172.21.15.98:0/3248375724 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43cc157b60 0x7f43cc154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:04.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.022+0000 7f43d1b24640 1 -- 172.21.15.98:0/3248375724 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43cc157b60 msgr2=0x7f43cc154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:04.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.022+0000 7f43d1b24640 1 --2- 172.21.15.98:0/3248375724 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43cc157b60 0x7f43cc154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:04.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.022+0000 7f43d1b24640 1 -- 172.21.15.98:0/3248375724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43cc0c4910 msgr2=0x7f43cc1ef320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:45:04.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.022+0000 7f43d1b24640 1 --2- 172.21.15.98:0/3248375724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43cc0c4910 0x7f43cc1ef320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:04.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.022+0000 7f43d1b24640 1 -- 172.21.15.98:0/3248375724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43bc009400 con 0x7f43cc151d80 2023-11-28T17:45:04.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.022+0000 7f43d2b26640 1 --2- 172.21.15.98:0/3248375724 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43cc157b60 0x7f43cc154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:45:04.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.022+0000 7f43d1b24640 1 --2- 172.21.15.98:0/3248375724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc151d80 0x7f43cc1f33f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f43bc02bf70 tx=0x7f43bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:04.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.022+0000 7f43c2ffd640 1 -- 172.21.15.98:0/3248375724 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc03e040 con 0x7f43cc151d80 2023-11-28T17:45:04.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.023+0000 7f43c2ffd640 1 -- 172.21.15.98:0/3248375724 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43bc039cb0 con 0x7f43cc151d80 2023-11-28T17:45:04.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.023+0000 7f43c2ffd640 1 -- 172.21.15.98:0/3248375724 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc043680 con 0x7f43cc151d80 2023-11-28T17:45:04.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.023+0000 7f43d4589640 1 -- 172.21.15.98:0/3248375724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43cc157490 con 0x7f43cc151d80 2023-11-28T17:45:04.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.023+0000 7f43d4589640 1 -- 172.21.15.98:0/3248375724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43cc1f2910 con 0x7f43cc151d80 2023-11-28T17:45:04.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.024+0000 7f43d4589640 1 -- 172.21.15.98:0/3248375724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43cc067bf0 con 0x7f43cc151d80 2023-11-28T17:45:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.025+0000 7f43c2ffd640 1 -- 172.21.15.98:0/3248375724 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f43bc0423e0 con 0x7f43cc151d80 2023-11-28T17:45:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.026+0000 7f43c2ffd640 1 --2- 172.21.15.98:0/3248375724 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f43b006b280 0x7f43b006d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.026+0000 7f43c2ffd640 1 -- 172.21.15.98:0/3248375724 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f43bc0ac200 con 0x7f43cc151d80 2023-11-28T17:45:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.026+0000 7f43d2325640 1 --2- 172.21.15.98:0/3248375724 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f43b006b280 0x7f43b006d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:04.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.027+0000 7f43d2325640 1 --2- 172.21.15.98:0/3248375724 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f43b006b280 0x7f43b006d740 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f43c8003930 tx=0x7f43c8006d30 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:04.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.030+0000 7f43c2ffd640 1 -- 172.21.15.98:0/3248375724 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f43bc076ec0 con 0x7f43cc151d80 2023-11-28T17:45:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.291+0000 7f43d4589640 1 -- 172.21.15.98:0/3248375724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f43cc074f60 con 0x7f43cc151d80 2023-11-28T17:45:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.291+0000 7f43c2ffd640 1 -- 172.21.15.98:0/3248375724 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f43bc07a4e0 con 0x7f43cc151d80 2023-11-28T17:45:04.292 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:45:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.294+0000 7f43d4589640 1 -- 172.21.15.98:0/3248375724 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f43b006b280 msgr2=0x7f43b006d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.294+0000 7f43d4589640 1 --2- 172.21.15.98:0/3248375724 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f43b006b280 0x7f43b006d740 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f43c8003930 tx=0x7f43c8006d30 comp rx=0 tx=0).stop 2023-11-28T17:45:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.294+0000 7f43d4589640 1 -- 172.21.15.98:0/3248375724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc151d80 msgr2=0x7f43cc1f33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.294+0000 7f43d4589640 1 --2- 172.21.15.98:0/3248375724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc151d80 0x7f43cc1f33f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f43bc02bf70 tx=0x7f43bc004730 comp rx=0 tx=0).stop 2023-11-28T17:45:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.294+0000 7f43d4589640 1 -- 172.21.15.98:0/3248375724 shutdown_connections 2023-11-28T17:45:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.294+0000 7f43d4589640 1 --2- 172.21.15.98:0/3248375724 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43cc157b60 0x7f43cc154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.294+0000 7f43d4589640 1 --2- 172.21.15.98:0/3248375724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43cc151d80 0x7f43cc1f33f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.294+0000 7f43d4589640 1 --2- 172.21.15.98:0/3248375724 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f43b006b280 0x7f43b006d740 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.294+0000 7f43d4589640 1 --2- 172.21.15.98:0/3248375724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43cc0c4910 0x7f43cc1ef320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.294+0000 7f43d4589640 1 -- 172.21.15.98:0/3248375724 >> 172.21.15.98:0/3248375724 conn(0x7f43cc06a4d0 msgr2=0x7f43cc14f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.294+0000 7f43d4589640 1 -- 172.21.15.98:0/3248375724 shutdown_connections 2023-11-28T17:45:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:04.294+0000 7f43d4589640 1 -- 172.21.15.98:0/3248375724 wait complete. 2023-11-28T17:45:05.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~14s 2023-11-28T17:45:07.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:45:07.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.533+0000 7f1a11c46640 1 Processor -- start 2023-11-28T17:45:07.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.533+0000 7f1a11c46640 1 -- start start 2023-11-28T17:45:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.535+0000 7f1a11c46640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1a0c0c7e40 0x7f1a0c15a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.535+0000 7f1a11c46640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a0c0c93e0 con 0x7f1a0c15a8f0 2023-11-28T17:45:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.535+0000 7f1a11c46640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a0c06d1f0 con 0x7f1a0c1574b0 2023-11-28T17:45:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.535+0000 7f1a11c46640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a0c15ad40 con 0x7f1a0c0c7e40 2023-11-28T17:45:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.535+0000 7f1a0b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f1a0c1574b0 legacy=0x7f1a0c0cd3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:45:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.535+0000 7f1a03fff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f1a0c0c7e40 msgr2=0x7f1a0c15a3b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.535+0000 7f1a03fff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1a0c0c7e40 0x7f1a0c15a3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.535+0000 7f1a0bfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1a0c15a8f0 0x7f1a0c164870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49716/0 (socket says 172.21.15.98:49716) 2023-11-28T17:45:07.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.535+0000 7f1a0bfff640 1 -- 172.21.15.98:0/3324873584 learned_addr learned my addr 172.21.15.98:0/3324873584 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:45:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.535+0000 7f1a0affd640 1 -- 172.21.15.98:0/3324873584 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221895547 0 0) 0x7f1a0c0c93e0 con 0x7f1a0c15a8f0 2023-11-28T17:45:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.535+0000 7f1a0affd640 1 -- 172.21.15.98:0/3324873584 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f8003520 con 0x7f1a0c15a8f0 2023-11-28T17:45:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a0affd640 1 -- 172.21.15.98:0/3324873584 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4120437419 0 0) 0x7f19f8003520 con 0x7f1a0c15a8f0 2023-11-28T17:45:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a0affd640 1 -- 172.21.15.98:0/3324873584 >> v2:172.21.15.113:3300/0 conn(0x7f1a0c0c7e40 msgr2=0x7f1a0c15a3b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a0affd640 1 --2- 172.21.15.98:0/3324873584 >> v2:172.21.15.113:3300/0 conn(0x7f1a0c0c7e40 0x7f1a0c15a3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a0affd640 1 -- 172.21.15.98:0/3324873584 >> v1:172.21.15.113:6789/0 conn(0x7f1a0c1574b0 legacy=0x7f1a0c0cd3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a0affd640 1 -- 172.21.15.98:0/3324873584 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0c06e750 con 0x7f1a0c15a8f0 2023-11-28T17:45:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a0affd640 1 -- 172.21.15.98:0/3324873584 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f1a040019f0 con 0x7f1a0c15a8f0 2023-11-28T17:45:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a0affd640 1 -- 172.21.15.98:0/3324873584 >> v1:172.21.15.98:6789/0 conn(0x7f1a0c15a8f0 legacy=0x7f1a0c164870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a0affd640 1 --2- 172.21.15.98:0/3324873584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19f8003970 0x7f19f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a0affd640 1 --2- 172.21.15.98:0/3324873584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f80047c0 0x7f19f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:07.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a0affd640 1 --2- 172.21.15.98:0/3324873584 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19f80052a0 0x7f19f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:07.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a0affd640 1 -- 172.21.15.98:0/3324873584 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c15ad40 con 0x7f19f80047c0 2023-11-28T17:45:07.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a0affd640 1 -- 172.21.15.98:0/3324873584 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f800c1a0 con 0x7f19f8003970 2023-11-28T17:45:07.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a0affd640 1 -- 172.21.15.98:0/3324873584 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f800c360 con 0x7f19f80052a0 2023-11-28T17:45:07.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a0b7fe640 1 -- 172.21.15.98:0/3324873584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19f8003970 msgr2=0x7f19f8003d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:07.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a0b7fe640 1 --2- 172.21.15.98:0/3324873584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19f8003970 0x7f19f8003d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:07.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.536+0000 7f1a03fff640 1 --2- 172.21.15.98:0/3324873584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f80047c0 0x7f19f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:07.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.537+0000 7f1a0bfff640 1 --2- 172.21.15.98:0/3324873584 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19f80052a0 0x7f19f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:07.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.537+0000 7f1a03fff640 1 -- 172.21.15.98:0/3324873584 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19f80052a0 msgr2=0x7f19f800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:07.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.537+0000 7f1a03fff640 1 --2- 172.21.15.98:0/3324873584 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19f80052a0 0x7f19f800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:07.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.537+0000 7f1a03fff640 1 -- 172.21.15.98:0/3324873584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19f8003970 msgr2=0x7f19f8003d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:07.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.537+0000 7f1a03fff640 1 --2- 172.21.15.98:0/3324873584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19f8003970 0x7f19f8003d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:07.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.537+0000 7f1a03fff640 1 -- 172.21.15.98:0/3324873584 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f8003520 con 0x7f19f80047c0 2023-11-28T17:45:07.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.537+0000 7f1a0bfff640 1 -- 172.21.15.98:0/3324873584 reap_dead start 2023-11-28T17:45:07.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.538+0000 7f1a03fff640 1 --2- 172.21.15.98:0/3324873584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f80047c0 0x7f19f8004c40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009830 tx=0x7f19f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f42687604a944a9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:07.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.538+0000 7f1a0affd640 1 -- 172.21.15.98:0/3324873584 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f4003ae0 con 0x7f19f80047c0 2023-11-28T17:45:07.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.538+0000 7f1a0affd640 1 -- 172.21.15.98:0/3324873584 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f402e070 con 0x7f19f80047c0 2023-11-28T17:45:07.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.538+0000 7f1a0affd640 1 -- 172.21.15.98:0/3324873584 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f4003e80 con 0x7f19f80047c0 2023-11-28T17:45:07.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.539+0000 7f1a11c46640 1 -- 172.21.15.98:0/3324873584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f80047c0 msgr2=0x7f19f8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:07.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.539+0000 7f1a11c46640 1 --2- 172.21.15.98:0/3324873584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f80047c0 0x7f19f8004c40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009830 tx=0x7f19f80057f0 comp rx=0 tx=0).stop 2023-11-28T17:45:07.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.539+0000 7f1a11c46640 1 -- 172.21.15.98:0/3324873584 shutdown_connections 2023-11-28T17:45:07.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.539+0000 7f1a11c46640 1 --2- 172.21.15.98:0/3324873584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f80047c0 0x7f19f8004c40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:07.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.539+0000 7f1a11c46640 1 -- 172.21.15.98:0/3324873584 >> 172.21.15.98:0/3324873584 conn(0x7f1a0c06ab00 msgr2=0x7f1a0c066230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:07.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.539+0000 7f1a11c46640 1 -- 172.21.15.98:0/3324873584 shutdown_connections 2023-11-28T17:45:07.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.539+0000 7f1a11c46640 1 -- 172.21.15.98:0/3324873584 wait complete. 2023-11-28T17:45:07.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.540+0000 7f1a11c46640 1 Processor -- start 2023-11-28T17:45:07.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.541+0000 7f1a11c46640 1 -- start start 2023-11-28T17:45:07.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.541+0000 7f1a11c46640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f80047c0 0x7f1a0c0af840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:07.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.541+0000 7f1a11c46640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a0c0c69c0 0x7f1a0c0c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:07.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.541+0000 7f1a11c46640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a0c0cc730 0x7f1a0c151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:07.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.541+0000 7f1a11c46640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c0c5b80 con 0x7f19f80047c0 2023-11-28T17:45:07.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.542+0000 7f1a11c46640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c0aecf0 con 0x7f1a0c0c69c0 2023-11-28T17:45:07.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.542+0000 7f1a11c46640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c0ae1d0 con 0x7f1a0c0cc730 2023-11-28T17:45:07.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.542+0000 7f1a0b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f80047c0 0x7f1a0c0af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:07.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.542+0000 7f1a03fff640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a0c0c69c0 msgr2=0x7f1a0c0c5fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:07.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.542+0000 7f1a03fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a0c0c69c0 0x7f1a0c0c5fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:07.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.542+0000 7f1a0b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f80047c0 0x7f1a0c0af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34112/0 (socket says 172.21.15.98:34112) 2023-11-28T17:45:07.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.542+0000 7f1a0b7fe640 1 -- 172.21.15.98:0/3208504324 learned_addr learned my addr 172.21.15.98:0/3208504324 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:07.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.542+0000 7f1a0bfff640 1 --2- 172.21.15.98:0/3208504324 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a0c0cc730 0x7f1a0c151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:07.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.542+0000 7f1a0b7fe640 1 -- 172.21.15.98:0/3208504324 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a0c0cc730 msgr2=0x7f1a0c151270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:07.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.542+0000 7f1a0b7fe640 1 --2- 172.21.15.98:0/3208504324 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a0c0cc730 0x7f1a0c151270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:07.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.542+0000 7f1a0b7fe640 1 -- 172.21.15.98:0/3208504324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a0c0c69c0 msgr2=0x7f1a0c0c5fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:45:07.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.542+0000 7f1a0b7fe640 1 --2- 172.21.15.98:0/3208504324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a0c0c69c0 0x7f1a0c0c5fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:07.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.542+0000 7f1a0b7fe640 1 -- 172.21.15.98:0/3208504324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0c0ae450 con 0x7f19f80047c0 2023-11-28T17:45:07.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.542+0000 7f1a0bfff640 1 --2- 172.21.15.98:0/3208504324 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a0c0cc730 0x7f1a0c151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:45:07.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.542+0000 7f1a0b7fe640 1 --2- 172.21.15.98:0/3208504324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f80047c0 0x7f1a0c0af840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00b570 tx=0x7f19fc00ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:07.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.543+0000 7f1a08ff9640 1 -- 172.21.15.98:0/3208504324 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19fc003e50 con 0x7f19f80047c0 2023-11-28T17:45:07.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.543+0000 7f1a08ff9640 1 -- 172.21.15.98:0/3208504324 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19fc005070 con 0x7f19f80047c0 2023-11-28T17:45:07.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.543+0000 7f1a08ff9640 1 -- 172.21.15.98:0/3208504324 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19fc005440 con 0x7f19f80047c0 2023-11-28T17:45:07.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.543+0000 7f1a11c46640 1 -- 172.21.15.98:0/3208504324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0c0ae730 con 0x7f19f80047c0 2023-11-28T17:45:07.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.543+0000 7f1a11c46640 1 -- 172.21.15.98:0/3208504324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0c0afd80 con 0x7f19f80047c0 2023-11-28T17:45:07.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.544+0000 7f1a11c46640 1 -- 172.21.15.98:0/3208504324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0c0c5df0 con 0x7f19f80047c0 2023-11-28T17:45:07.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.545+0000 7f1a08ff9640 1 -- 172.21.15.98:0/3208504324 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f19fc0055a0 con 0x7f19f80047c0 2023-11-28T17:45:07.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.546+0000 7f1a08ff9640 1 --2- 172.21.15.98:0/3208504324 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19ec06b250 0x7f19ec06d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:07.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.546+0000 7f1a08ff9640 1 -- 172.21.15.98:0/3208504324 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f19fc089920 con 0x7f19f80047c0 2023-11-28T17:45:07.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.546+0000 7f1a03fff640 1 --2- 172.21.15.98:0/3208504324 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19ec06b250 0x7f19ec06d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:07.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.547+0000 7f1a03fff640 1 --2- 172.21.15.98:0/3208504324 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19ec06b250 0x7f19ec06d710 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f19f4033000 tx=0x7f19f4008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:07.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.551+0000 7f1a08ff9640 1 -- 172.21.15.98:0/3208504324 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f19fc054700 con 0x7f19f80047c0 2023-11-28T17:45:07.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.809+0000 7f1a11c46640 1 -- 172.21.15.98:0/3208504324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1a0c0c58e0 con 0x7f19f80047c0 2023-11-28T17:45:07.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.810+0000 7f1a08ff9640 1 -- 172.21.15.98:0/3208504324 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f19fc013020 con 0x7f19f80047c0 2023-11-28T17:45:07.812 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:45:07.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.813+0000 7f1a11c46640 1 -- 172.21.15.98:0/3208504324 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19ec06b250 msgr2=0x7f19ec06d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:07.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.813+0000 7f1a11c46640 1 --2- 172.21.15.98:0/3208504324 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19ec06b250 0x7f19ec06d710 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f19f4033000 tx=0x7f19f4008030 comp rx=0 tx=0).stop 2023-11-28T17:45:07.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.813+0000 7f1a11c46640 1 -- 172.21.15.98:0/3208504324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f80047c0 msgr2=0x7f1a0c0af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:07.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.813+0000 7f1a11c46640 1 --2- 172.21.15.98:0/3208504324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f80047c0 0x7f1a0c0af840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00b570 tx=0x7f19fc00ba30 comp rx=0 tx=0).stop 2023-11-28T17:45:07.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.813+0000 7f1a11c46640 1 -- 172.21.15.98:0/3208504324 shutdown_connections 2023-11-28T17:45:07.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.813+0000 7f1a11c46640 1 --2- 172.21.15.98:0/3208504324 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a0c0cc730 0x7f1a0c151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:07.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.813+0000 7f1a11c46640 1 --2- 172.21.15.98:0/3208504324 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19ec06b250 0x7f19ec06d710 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:07.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.813+0000 7f1a11c46640 1 --2- 172.21.15.98:0/3208504324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a0c0c69c0 0x7f1a0c0c5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:07.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.813+0000 7f1a11c46640 1 --2- 172.21.15.98:0/3208504324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f80047c0 0x7f1a0c0af840 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:07.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.813+0000 7f1a11c46640 1 -- 172.21.15.98:0/3208504324 >> 172.21.15.98:0/3208504324 conn(0x7f1a0c06ab00 msgr2=0x7f1a0c066230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.813+0000 7f1a11c46640 1 -- 172.21.15.98:0/3208504324 shutdown_connections 2023-11-28T17:45:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:07.813+0000 7f1a11c46640 1 -- 172.21.15.98:0/3208504324 wait complete. 2023-11-28T17:45:08.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:45:08.694 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:45:10.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:45:11.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.053+0000 7fdb2f59e640 1 Processor -- start 2023-11-28T17:45:11.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.053+0000 7fdb2f59e640 1 -- start start 2023-11-28T17:45:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.055+0000 7fdb2f59e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdb30154060 0x7fdb300bff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.055+0000 7fdb2f59e640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdb300c0f40 0x7fdb3006dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.055+0000 7fdb2f59e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb30154440 con 0x7fdb3006afe0 2023-11-28T17:45:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.055+0000 7fdb2f59e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb30159c60 con 0x7fdb30154060 2023-11-28T17:45:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.055+0000 7fdb2f59e640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb30159e80 con 0x7fdb300c0f40 2023-11-28T17:45:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.055+0000 7fdb2ed9d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7fdb300c0f40 msgr2=0x7fdb3006dd30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.055+0000 7fdb2ed9d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdb300c0f40 0x7fdb3006dd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.055+0000 7fdb25d9b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdb3006afe0 0x7fdb300c0830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40418/0 (socket says 172.21.15.98:40418) 2023-11-28T17:45:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.055+0000 7fdb25d9b640 1 -- 172.21.15.98:0/761538315 learned_addr learned my addr 172.21.15.98:0/761538315 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:45:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.055+0000 7fdb2e59c640 1 --2- 172.21.15.98:0/761538315 >> v2:172.21.15.98:3300/0 conn(0x7fdb30154060 0x7fdb300bff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.056+0000 7fdb2e59c640 1 -- 172.21.15.98:0/761538315 >> v2:172.21.15.113:3300/0 conn(0x7fdb300c0f40 msgr2=0x7fdb3006dd30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.056+0000 7fdb2e59c640 1 --2- 172.21.15.98:0/761538315 >> v2:172.21.15.113:3300/0 conn(0x7fdb300c0f40 0x7fdb3006dd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.056+0000 7fdb2e59c640 1 -- 172.21.15.98:0/761538315 >> v1:172.21.15.98:6789/0 conn(0x7fdb3006afe0 legacy=0x7fdb300c0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.056+0000 7fdb2e59c640 1 -- 172.21.15.98:0/761538315 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3015a160 con 0x7fdb30154060 2023-11-28T17:45:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.057+0000 7fdb2e59c640 1 --2- 172.21.15.98:0/761538315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb30154060 0x7fdb300bff20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdb3006ea10 tx=0x7fdb2002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4792376d82a25e2a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.057+0000 7fdb2dd9b640 1 -- 172.21.15.98:0/761538315 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb20031640 con 0x7fdb30154060 2023-11-28T17:45:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.057+0000 7fdb2dd9b640 1 -- 172.21.15.98:0/761538315 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb200317a0 con 0x7fdb30154060 2023-11-28T17:45:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.057+0000 7fdb2dd9b640 1 -- 172.21.15.98:0/761538315 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb20031b30 con 0x7fdb30154060 2023-11-28T17:45:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.058+0000 7fdb2f59e640 1 -- 172.21.15.98:0/761538315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb30154060 msgr2=0x7fdb300bff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.058+0000 7fdb2f59e640 1 --2- 172.21.15.98:0/761538315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb30154060 0x7fdb300bff20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdb3006ea10 tx=0x7fdb2002b8f0 comp rx=0 tx=0).stop 2023-11-28T17:45:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.058+0000 7fdb2f59e640 1 -- 172.21.15.98:0/761538315 shutdown_connections 2023-11-28T17:45:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.058+0000 7fdb2f59e640 1 --2- 172.21.15.98:0/761538315 >> v2:172.21.15.113:3300/0 conn(0x7fdb300c0f40 0x7fdb3006dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.058+0000 7fdb2f59e640 1 --2- 172.21.15.98:0/761538315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb30154060 0x7fdb300bff20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.058+0000 7fdb2f59e640 1 -- 172.21.15.98:0/761538315 >> 172.21.15.98:0/761538315 conn(0x7fdb30066230 msgr2=0x7fdb3006a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.058+0000 7fdb2f59e640 1 -- 172.21.15.98:0/761538315 shutdown_connections 2023-11-28T17:45:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.058+0000 7fdb2f59e640 1 -- 172.21.15.98:0/761538315 wait complete. 2023-11-28T17:45:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.060+0000 7fdb2f59e640 1 Processor -- start 2023-11-28T17:45:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.060+0000 7fdb2f59e640 1 -- start start 2023-11-28T17:45:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.061+0000 7fdb2f59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb3006afe0 0x7fdb301f0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.061+0000 7fdb2f59e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb30154060 0x7fdb30150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.061+0000 7fdb2f59e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb300c51a0 0x7fdb30070f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.061+0000 7fdb2f59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb30070b10 con 0x7fdb3006afe0 2023-11-28T17:45:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.061+0000 7fdb2f59e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb301f0940 con 0x7fdb300c51a0 2023-11-28T17:45:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.061+0000 7fdb2f59e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb30071fc0 con 0x7fdb30154060 2023-11-28T17:45:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.061+0000 7fdb2e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb3006afe0 0x7fdb301f0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.061+0000 7fdb2ed9d640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb300c51a0 msgr2=0x7fdb30070f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.061+0000 7fdb2e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb3006afe0 0x7fdb301f0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50752/0 (socket says 172.21.15.98:50752) 2023-11-28T17:45:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.061+0000 7fdb2e59c640 1 -- 172.21.15.98:0/2820471703 learned_addr learned my addr 172.21.15.98:0/2820471703 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.061+0000 7fdb2ed9d640 1 --2- 172.21.15.98:0/2820471703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb300c51a0 0x7fdb30070f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.062+0000 7fdb2e59c640 1 -- 172.21.15.98:0/2820471703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb30154060 msgr2=0x7fdb30150720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.062+0000 7fdb2e59c640 1 --2- 172.21.15.98:0/2820471703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb30154060 0x7fdb30150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.062+0000 7fdb2e59c640 1 -- 172.21.15.98:0/2820471703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb300c51a0 msgr2=0x7fdb30070f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:45:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.062+0000 7fdb2e59c640 1 --2- 172.21.15.98:0/2820471703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb300c51a0 0x7fdb30070f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.062+0000 7fdb2e59c640 1 -- 172.21.15.98:0/2820471703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb20009400 con 0x7fdb3006afe0 2023-11-28T17:45:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.062+0000 7fdb2e59c640 1 --2- 172.21.15.98:0/2820471703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb3006afe0 0x7fdb301f0400 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdb2002bf00 tx=0x7fdb200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.062+0000 7fdb277fe640 1 -- 172.21.15.98:0/2820471703 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2003bd30 con 0x7fdb3006afe0 2023-11-28T17:45:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.062+0000 7fdb277fe640 1 -- 172.21.15.98:0/2820471703 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb200433f0 con 0x7fdb3006afe0 2023-11-28T17:45:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.062+0000 7fdb2f59e640 1 -- 172.21.15.98:0/2820471703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb30072290 con 0x7fdb3006afe0 2023-11-28T17:45:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.062+0000 7fdb277fe640 1 -- 172.21.15.98:0/2820471703 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb200437a0 con 0x7fdb3006afe0 2023-11-28T17:45:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.062+0000 7fdb2f59e640 1 -- 172.21.15.98:0/2820471703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb30072850 con 0x7fdb3006afe0 2023-11-28T17:45:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.064+0000 7fdb2f59e640 1 -- 172.21.15.98:0/2820471703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb30070c50 con 0x7fdb3006afe0 2023-11-28T17:45:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.064+0000 7fdb277fe640 1 -- 172.21.15.98:0/2820471703 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fdb2002f050 con 0x7fdb3006afe0 2023-11-28T17:45:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.065+0000 7fdb277fe640 1 --2- 172.21.15.98:0/2820471703 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdb1006b280 0x7fdb1006d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.065+0000 7fdb277fe640 1 -- 172.21.15.98:0/2820471703 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdb200abb60 con 0x7fdb3006afe0 2023-11-28T17:45:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.065+0000 7fdb25d9b640 1 --2- 172.21.15.98:0/2820471703 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdb1006b280 0x7fdb1006d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.066+0000 7fdb25d9b640 1 --2- 172.21.15.98:0/2820471703 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdb1006b280 0x7fdb1006d740 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdb30152e40 tx=0x7fdb18007a40 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.070+0000 7fdb277fe640 1 -- 172.21.15.98:0/2820471703 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdb30070c50 con 0x7fdb3006afe0 2023-11-28T17:45:11.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.328+0000 7fdb2f59e640 1 -- 172.21.15.98:0/2820471703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdb30070c50 con 0x7fdb3006afe0 2023-11-28T17:45:11.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.329+0000 7fdb277fe640 1 -- 172.21.15.98:0/2820471703 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7fdb20048620 con 0x7fdb3006afe0 2023-11-28T17:45:11.331 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:45:11.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.331+0000 7fdb2f59e640 1 -- 172.21.15.98:0/2820471703 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdb1006b280 msgr2=0x7fdb1006d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:11.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.331+0000 7fdb2f59e640 1 --2- 172.21.15.98:0/2820471703 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdb1006b280 0x7fdb1006d740 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdb30152e40 tx=0x7fdb18007a40 comp rx=0 tx=0).stop 2023-11-28T17:45:11.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.331+0000 7fdb2f59e640 1 -- 172.21.15.98:0/2820471703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb3006afe0 msgr2=0x7fdb301f0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:11.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.331+0000 7fdb2f59e640 1 --2- 172.21.15.98:0/2820471703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb3006afe0 0x7fdb301f0400 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdb2002bf00 tx=0x7fdb200046c0 comp rx=0 tx=0).stop 2023-11-28T17:45:11.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.331+0000 7fdb2f59e640 1 -- 172.21.15.98:0/2820471703 shutdown_connections 2023-11-28T17:45:11.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.331+0000 7fdb2f59e640 1 --2- 172.21.15.98:0/2820471703 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdb1006b280 0x7fdb1006d740 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:11.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.331+0000 7fdb2f59e640 1 --2- 172.21.15.98:0/2820471703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb300c51a0 0x7fdb30070f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:11.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.331+0000 7fdb2f59e640 1 --2- 172.21.15.98:0/2820471703 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdb30154060 0x7fdb30150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:11.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.331+0000 7fdb2f59e640 1 --2- 172.21.15.98:0/2820471703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb3006afe0 0x7fdb301f0400 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:11.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.332+0000 7fdb2f59e640 1 -- 172.21.15.98:0/2820471703 >> 172.21.15.98:0/2820471703 conn(0x7fdb30066230 msgr2=0x7fdb3014f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:11.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.332+0000 7fdb2f59e640 1 -- 172.21.15.98:0/2820471703 shutdown_connections 2023-11-28T17:45:11.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:11.332+0000 7fdb2f59e640 1 -- 172.21.15.98:0/2820471703 wait complete. 2023-11-28T17:45:11.929 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~20s 2023-11-28T17:45:14.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:45:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.570+0000 7f4f1d55a640 1 Processor -- start 2023-11-28T17:45:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.571+0000 7f4f1d55a640 1 -- start start 2023-11-28T17:45:14.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.572+0000 7f4f1d55a640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4f18157b60 0x7f4f1815bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:14.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.572+0000 7f4f1d55a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 0x7f4f180c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:14.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.572+0000 7f4f1d55a640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1815b150 con 0x7f4f18151d80 2023-11-28T17:45:14.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.572+0000 7f4f1d55a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f180c5260 con 0x7f4f180c4910 2023-11-28T17:45:14.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.573+0000 7f4f1d55a640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f18157f90 con 0x7f4f18157b60 2023-11-28T17:45:14.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.573+0000 7f4f0e7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4f18157b60 msgr2=0x7f4f1815bc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:14.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.573+0000 7f4f16ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4f18151d80 legacy=0x7f4f180c5c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:45:14.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.573+0000 7f4f0e7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4f18157b60 0x7f4f1815bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:14.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.573+0000 7f4f177fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 0x7f4f180c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:14.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.573+0000 7f4f177fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 0x7f4f180c4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50762/0 (socket says 172.21.15.98:50762) 2023-11-28T17:45:14.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.573+0000 7f4f177fe640 1 -- 172.21.15.98:0/1605611929 learned_addr learned my addr 172.21.15.98:0/1605611929 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:14.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.573+0000 7f4f177fe640 1 -- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 msgr2=0x7f4f180c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:45:14.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.573+0000 7f4f177fe640 1 -- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 msgr2=0x7f4f180c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:14.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.573+0000 7f4f177fe640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 0x7f4f180c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:14.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.573+0000 7f4f177fe640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 0x7f4f180c4d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:14.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.774+0000 7f4f16ffd640 1 -- 172.21.15.98:0/1605611929 >> v1:172.21.15.113:6789/0 conn(0x7f4f18151d80 legacy=0x7f4f180c5c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:45:14.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.774+0000 7f4f0e7fc640 1 -- 172.21.15.98:0/1605611929 >> v2:172.21.15.113:3300/0 conn(0x7f4f18157b60 msgr2=0x7f4f1815bc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:14.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.774+0000 7f4f0e7fc640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.113:3300/0 conn(0x7f4f18157b60 0x7f4f1815bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:45:14.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.774+0000 7f4f177fe640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 0x7f4f180c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:14.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.774+0000 7f4f177fe640 1 -- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 msgr2=0x7f4f180c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:45:14.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.774+0000 7f4f177fe640 1 -- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 msgr2=0x7f4f180c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:14.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.774+0000 7f4f177fe640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 0x7f4f180c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:14.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:14.774+0000 7f4f177fe640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 0x7f4f180c4d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:45:15.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.175+0000 7f4f16ffd640 1 -- 172.21.15.98:0/1605611929 >> v1:172.21.15.113:6789/0 conn(0x7f4f18151d80 legacy=0x7f4f180c5c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:45:15.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.175+0000 7f4f0e7fc640 1 -- 172.21.15.98:0/1605611929 >> v2:172.21.15.113:3300/0 conn(0x7f4f18157b60 msgr2=0x7f4f1815bc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:15.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.175+0000 7f4f0e7fc640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.113:3300/0 conn(0x7f4f18157b60 0x7f4f1815bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:45:15.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.175+0000 7f4f177fe640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 0x7f4f180c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:15.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.176+0000 7f4f177fe640 1 -- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 msgr2=0x7f4f180c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:45:15.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.176+0000 7f4f177fe640 1 -- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 msgr2=0x7f4f180c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:15.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.176+0000 7f4f177fe640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 0x7f4f180c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:15.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.176+0000 7f4f177fe640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 0x7f4f180c4d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:45:15.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.976+0000 7f4f16ffd640 1 -- 172.21.15.98:0/1605611929 >> v1:172.21.15.113:6789/0 conn(0x7f4f18151d80 legacy=0x7f4f180c5c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:45:15.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.976+0000 7f4f0e7fc640 1 -- 172.21.15.98:0/1605611929 >> v2:172.21.15.113:3300/0 conn(0x7f4f18157b60 msgr2=0x7f4f1815bc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:15.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.976+0000 7f4f0e7fc640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.113:3300/0 conn(0x7f4f18157b60 0x7f4f1815bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:45:15.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.977+0000 7f4f177fe640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 0x7f4f180c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:15.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.978+0000 7f4f177fe640 1 -- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 msgr2=0x7f4f180c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:45:15.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.978+0000 7f4f177fe640 1 -- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 msgr2=0x7f4f180c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:15.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.978+0000 7f4f177fe640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 0x7f4f180c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:15.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:15.978+0000 7f4f177fe640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 0x7f4f180c4d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:45:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.573+0000 7f4f157fa640 1 -- 172.21.15.98:0/1605611929 >> v2:172.21.15.113:3300/0 conn(0x7f4f18157b60 msgr2=0x7f4f1815bc10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.573+0000 7f4f157fa640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.113:3300/0 conn(0x7f4f18157b60 0x7f4f1815bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.573+0000 7f4f157fa640 1 -- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 msgr2=0x7f4f180c4d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.573+0000 7f4f157fa640 1 --2- 172.21.15.98:0/1605611929 >> v2:172.21.15.98:3300/0 conn(0x7f4f180c4910 0x7f4f180c4d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.573+0000 7f4f157fa640 1 -- 172.21.15.98:0/1605611929 >> v1:172.21.15.113:6789/0 conn(0x7f4f18151d80 legacy=0x7f4f180c5c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.573+0000 7f4f157fa640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 0x7f4ef8009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.573+0000 7f4f157fa640 1 -- 172.21.15.98:0/1605611929 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1806e1f0 con 0x7f4ef8003690 2023-11-28T17:45:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.573+0000 7f4f157fa640 1 -- 172.21.15.98:0/1605611929 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f180c5260 con 0x7f4ef8007350 2023-11-28T17:45:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.574+0000 7f4f0e7fc640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 0x7f4ef8009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.574+0000 7f4f0e7fc640 1 -- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 msgr2=0x7f4ef8009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:45:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.574+0000 7f4f0e7fc640 1 -- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 msgr2=0x7f4ef8009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.574+0000 7f4f0e7fc640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 0x7f4ef8009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:17.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.574+0000 7f4f0e7fc640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 0x7f4ef8009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:17.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.776+0000 7f4f0e7fc640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 0x7f4ef8009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:17.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.776+0000 7f4f0e7fc640 1 -- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 msgr2=0x7f4ef8009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:45:17.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.776+0000 7f4f0e7fc640 1 -- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 msgr2=0x7f4ef8009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:17.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.776+0000 7f4f0e7fc640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 0x7f4ef8009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:17.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:17.776+0000 7f4f0e7fc640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 0x7f4ef8009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:45:18.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.177+0000 7f4f0e7fc640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 0x7f4ef8009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:18.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.178+0000 7f4f0e7fc640 1 -- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 msgr2=0x7f4ef8009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:45:18.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.178+0000 7f4f0e7fc640 1 -- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 msgr2=0x7f4ef8009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:18.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.178+0000 7f4f0e7fc640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 0x7f4ef8009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:18.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.178+0000 7f4f0e7fc640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 0x7f4ef8009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:45:18.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.412+0000 7f4f167fc640 1 -- 172.21.15.98:0/1605611929 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356520961 0 0) 0x7f4f1806e1f0 con 0x7f4ef8003690 2023-11-28T17:45:18.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.412+0000 7f4f167fc640 1 -- 172.21.15.98:0/1605611929 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f04003520 con 0x7f4ef8003690 2023-11-28T17:45:18.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.413+0000 7f4f167fc640 1 -- 172.21.15.98:0/1605611929 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1336958366 0 0) 0x7f4f04003520 con 0x7f4ef8003690 2023-11-28T17:45:18.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.413+0000 7f4f167fc640 1 -- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 msgr2=0x7f4ef8009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:18.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.413+0000 7f4f167fc640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4ef8007350 0x7f4ef8009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.413+0000 7f4f167fc640 1 -- 172.21.15.98:0/1605611929 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1815b150 con 0x7f4ef8003690 2023-11-28T17:45:18.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.413+0000 7f4f167fc640 1 -- 172.21.15.98:0/1605611929 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f4f080042c0 con 0x7f4ef8003690 2023-11-28T17:45:18.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.413+0000 7f4f167fc640 1 -- 172.21.15.98:0/1605611929 >> v1:172.21.15.98:6789/0 conn(0x7f4ef8003690 legacy=0x7f4ef8005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:18.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.413+0000 7f4f177fe640 1 -- 172.21.15.98:0/1605611929 reap_dead start 2023-11-28T17:45:18.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.414+0000 7f4f167fc640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f04003970 0x7f4f04003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.414+0000 7f4f167fc640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f040047c0 0x7f4f04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.414+0000 7f4f167fc640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f040051d0 0x7f4f0400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.414+0000 7f4f167fc640 1 -- 172.21.15.98:0/1605611929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f180c5260 con 0x7f4f040051d0 2023-11-28T17:45:18.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.414+0000 7f4f167fc640 1 -- 172.21.15.98:0/1605611929 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0400c100 con 0x7f4f04003970 2023-11-28T17:45:18.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.414+0000 7f4f167fc640 1 -- 172.21.15.98:0/1605611929 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0400c2c0 con 0x7f4f040047c0 2023-11-28T17:45:18.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.414+0000 7f4f16ffd640 1 -- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f04003970 msgr2=0x7f4f04003d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:18.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.414+0000 7f4f16ffd640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f04003970 0x7f4f04003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:18.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.415+0000 7f4f177fe640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f040051d0 0x7f4f0400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:18.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.415+0000 7f4f0e7fc640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f040047c0 0x7f4f04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:18.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.416+0000 7f4f177fe640 1 -- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f040047c0 msgr2=0x7f4f04004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:18.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.416+0000 7f4f177fe640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f040047c0 0x7f4f04004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.416+0000 7f4f177fe640 1 -- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f04003970 msgr2=0x7f4f04003d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:18.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.416+0000 7f4f177fe640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f04003970 0x7f4f04003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.416+0000 7f4f177fe640 1 -- 172.21.15.98:0/1605611929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f04003520 con 0x7f4f040051d0 2023-11-28T17:45:18.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.419+0000 7f4f177fe640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f040051d0 0x7f4f0400b980 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4f1806bb80 tx=0x7f4f040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88ce0396f79079d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:18.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.420+0000 7f4f167fc640 1 -- 172.21.15.98:0/1605611929 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f10035410 con 0x7f4f040051d0 2023-11-28T17:45:18.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.420+0000 7f4f167fc640 1 -- 172.21.15.98:0/1605611929 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1002f030 con 0x7f4f040051d0 2023-11-28T17:45:18.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.421+0000 7f4f167fc640 1 -- 172.21.15.98:0/1605611929 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f100355b0 con 0x7f4f040051d0 2023-11-28T17:45:18.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.422+0000 7f4f1d55a640 1 -- 172.21.15.98:0/1605611929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f040051d0 msgr2=0x7f4f0400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:18.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.422+0000 7f4f1d55a640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f040051d0 0x7f4f0400b980 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4f1806bb80 tx=0x7f4f040057f0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.422+0000 7f4f1d55a640 1 -- 172.21.15.98:0/1605611929 shutdown_connections 2023-11-28T17:45:18.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.422+0000 7f4f1d55a640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f040051d0 0x7f4f0400b980 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.422+0000 7f4f1d55a640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f040047c0 0x7f4f04004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.422+0000 7f4f1d55a640 1 --2- 172.21.15.98:0/1605611929 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f04003970 0x7f4f04003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.422+0000 7f4f1d55a640 1 -- 172.21.15.98:0/1605611929 >> 172.21.15.98:0/1605611929 conn(0x7f4f1806a4d0 msgr2=0x7f4f1806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:18.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.422+0000 7f4f1d55a640 1 -- 172.21.15.98:0/1605611929 shutdown_connections 2023-11-28T17:45:18.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.422+0000 7f4f1d55a640 1 -- 172.21.15.98:0/1605611929 wait complete. 2023-11-28T17:45:18.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.425+0000 7f4f1d55a640 1 Processor -- start 2023-11-28T17:45:18.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.425+0000 7f4f1d55a640 1 -- start start 2023-11-28T17:45:18.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.426+0000 7f4f1d55a640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f04003970 0x7f4f18150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.426+0000 7f4f1d55a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f040047c0 0x7f4f181efa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.426+0000 7f4f1d55a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f040051d0 0x7f4f181eff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.426+0000 7f4f1d55a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f180c8950 con 0x7f4f040051d0 2023-11-28T17:45:18.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.426+0000 7f4f1d55a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f18152450 con 0x7f4f040047c0 2023-11-28T17:45:18.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.426+0000 7f4f1d55a640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f181ef520 con 0x7f4f04003970 2023-11-28T17:45:18.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.427+0000 7f4f177fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f040051d0 0x7f4f181eff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:18.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.427+0000 7f4f177fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f040051d0 0x7f4f181eff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50790/0 (socket says 172.21.15.98:50790) 2023-11-28T17:45:18.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.427+0000 7f4f177fe640 1 -- 172.21.15.98:0/759692320 learned_addr learned my addr 172.21.15.98:0/759692320 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:18.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.427+0000 7f4f0e7fc640 1 -- 172.21.15.98:0/759692320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f040047c0 msgr2=0x7f4f181efa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:18.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.427+0000 7f4f0e7fc640 1 --2- 172.21.15.98:0/759692320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f040047c0 0x7f4f181efa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:18.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.427+0000 7f4f16ffd640 1 --2- 172.21.15.98:0/759692320 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f04003970 0x7f4f18150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:18.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.428+0000 7f4f177fe640 1 -- 172.21.15.98:0/759692320 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f04003970 msgr2=0x7f4f18150ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:18.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.428+0000 7f4f177fe640 1 --2- 172.21.15.98:0/759692320 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f04003970 0x7f4f18150ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.428+0000 7f4f177fe640 1 -- 172.21.15.98:0/759692320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f040047c0 msgr2=0x7f4f181efa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:45:18.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.428+0000 7f4f177fe640 1 --2- 172.21.15.98:0/759692320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f040047c0 0x7f4f181efa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.428+0000 7f4f177fe640 1 -- 172.21.15.98:0/759692320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f181526a0 con 0x7f4f040051d0 2023-11-28T17:45:18.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.428+0000 7f4f177fe640 1 --2- 172.21.15.98:0/759692320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f040051d0 0x7f4f181eff60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4f10002230 tx=0x7f4f10004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:18.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.432+0000 7f4f0ffff640 1 -- 172.21.15.98:0/759692320 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f10040830 con 0x7f4f040051d0 2023-11-28T17:45:18.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.432+0000 7f4f0ffff640 1 -- 172.21.15.98:0/759692320 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1000f040 con 0x7f4f040051d0 2023-11-28T17:45:18.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.432+0000 7f4f0ffff640 1 -- 172.21.15.98:0/759692320 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f10040bd0 con 0x7f4f040051d0 2023-11-28T17:45:18.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.432+0000 7f4f1d55a640 1 -- 172.21.15.98:0/759692320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f18152980 con 0x7f4f040051d0 2023-11-28T17:45:18.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.432+0000 7f4f1d55a640 1 -- 172.21.15.98:0/759692320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f18152f40 con 0x7f4f040051d0 2023-11-28T17:45:18.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.434+0000 7f4f1d55a640 1 -- 172.21.15.98:0/759692320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f18067bf0 con 0x7f4f040051d0 2023-11-28T17:45:18.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.435+0000 7f4f0ffff640 1 -- 172.21.15.98:0/759692320 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4f1003c040 con 0x7f4f040051d0 2023-11-28T17:45:18.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.436+0000 7f4f0ffff640 1 --2- 172.21.15.98:0/759692320 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4ef806ba50 0x7f4ef806df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.436+0000 7f4f0ffff640 1 -- 172.21.15.98:0/759692320 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4f100abff0 con 0x7f4f040051d0 2023-11-28T17:45:18.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.436+0000 7f4f16ffd640 1 --2- 172.21.15.98:0/759692320 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4ef806ba50 0x7f4ef806df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:18.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.437+0000 7f4f16ffd640 1 --2- 172.21.15.98:0/759692320 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4ef806ba50 0x7f4ef806df10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4f08003840 tx=0x7f4f08009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:18.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.442+0000 7f4f0ffff640 1 -- 172.21.15.98:0/759692320 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f10076ec0 con 0x7f4f040051d0 2023-11-28T17:45:18.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~27s 2023-11-28T17:45:18.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.728+0000 7f4f1d55a640 1 -- 172.21.15.98:0/759692320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4f180c8950 con 0x7f4f040051d0 2023-11-28T17:45:18.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.729+0000 7f4f0ffff640 1 -- 172.21.15.98:0/759692320 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f4f1004e620 con 0x7f4f040051d0 2023-11-28T17:45:18.731 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":40,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:45:18.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.731+0000 7f4f1d55a640 1 -- 172.21.15.98:0/759692320 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4ef806ba50 msgr2=0x7f4ef806df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:18.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.731+0000 7f4f1d55a640 1 --2- 172.21.15.98:0/759692320 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4ef806ba50 0x7f4ef806df10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4f08003840 tx=0x7f4f08009040 comp rx=0 tx=0).stop 2023-11-28T17:45:18.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.731+0000 7f4f1d55a640 1 -- 172.21.15.98:0/759692320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f040051d0 msgr2=0x7f4f181eff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:18.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.731+0000 7f4f1d55a640 1 --2- 172.21.15.98:0/759692320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f040051d0 0x7f4f181eff60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4f10002230 tx=0x7f4f10004ab0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.731+0000 7f4f1d55a640 1 -- 172.21.15.98:0/759692320 shutdown_connections 2023-11-28T17:45:18.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.731+0000 7f4f1d55a640 1 --2- 172.21.15.98:0/759692320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f040051d0 0x7f4f181eff60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.731+0000 7f4f1d55a640 1 --2- 172.21.15.98:0/759692320 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4ef806ba50 0x7f4ef806df10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.731+0000 7f4f1d55a640 1 --2- 172.21.15.98:0/759692320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f040047c0 0x7f4f181efa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.731+0000 7f4f1d55a640 1 --2- 172.21.15.98:0/759692320 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f04003970 0x7f4f18150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.731+0000 7f4f1d55a640 1 -- 172.21.15.98:0/759692320 >> 172.21.15.98:0/759692320 conn(0x7f4f1806a4d0 msgr2=0x7f4f180c58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:18.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.732+0000 7f4f1d55a640 1 -- 172.21.15.98:0/759692320 shutdown_connections 2023-11-28T17:45:18.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.732+0000 7f4f1d55a640 1 -- 172.21.15.98:0/759692320 wait complete. 2023-11-28T17:45:18.753 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-11-28T17:45:18.753 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-11-28T17:45:18.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:45:18.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.965+0000 7f4b99173640 1 Processor -- start 2023-11-28T17:45:18.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.965+0000 7f4b99173640 1 -- start start 2023-11-28T17:45:18.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.967+0000 7f4b99173640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4b94151be0 0x7f4b940c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.967+0000 7f4b99173640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b940c8150 con 0x7f4b941584d0 2023-11-28T17:45:18.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.967+0000 7f4b99173640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b940c8850 con 0x7f4b940c5a60 2023-11-28T17:45:18.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.967+0000 7f4b99173640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b940c5f50 con 0x7f4b94151be0 2023-11-28T17:45:18.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.967+0000 7f4b92d9d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4b94151be0 msgr2=0x7f4b940c75a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:18.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.967+0000 7f4b92d9d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4b94151be0 0x7f4b940c75a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:18.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.968+0000 7f4b9359e640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4b941584d0 0x7f4b9406dd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40432/0 (socket says 172.21.15.98:40432) 2023-11-28T17:45:18.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.967+0000 7f4b8bfff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4b940c5a60 legacy=0x7f4b940c4730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:45:18.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.968+0000 7f4b9359e640 1 -- 172.21.15.98:0/2159436742 learned_addr learned my addr 172.21.15.98:0/2159436742 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:45:18.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.968+0000 7f4b9259c640 1 -- 172.21.15.98:0/2159436742 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677022478 0 0) 0x7f4b940c8150 con 0x7f4b941584d0 2023-11-28T17:45:18.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.968+0000 7f4b9259c640 1 -- 172.21.15.98:0/2159436742 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b80003520 con 0x7f4b941584d0 2023-11-28T17:45:18.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.968+0000 7f4b9259c640 1 -- 172.21.15.98:0/2159436742 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2475772320 0 0) 0x7f4b80003520 con 0x7f4b941584d0 2023-11-28T17:45:18.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.968+0000 7f4b9259c640 1 -- 172.21.15.98:0/2159436742 >> v2:172.21.15.113:3300/0 conn(0x7f4b94151be0 msgr2=0x7f4b940c75a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:18.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.968+0000 7f4b9259c640 1 --2- 172.21.15.98:0/2159436742 >> v2:172.21.15.113:3300/0 conn(0x7f4b94151be0 0x7f4b940c75a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.968+0000 7f4b9259c640 1 -- 172.21.15.98:0/2159436742 >> v1:172.21.15.113:6789/0 conn(0x7f4b940c5a60 legacy=0x7f4b940c4730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:18.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.968+0000 7f4b9259c640 1 -- 172.21.15.98:0/2159436742 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b940c7b90 con 0x7f4b941584d0 2023-11-28T17:45:18.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.968+0000 7f4b9259c640 1 -- 172.21.15.98:0/2159436742 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f4b8c0019f0 con 0x7f4b941584d0 2023-11-28T17:45:18.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.968+0000 7f4b9259c640 1 -- 172.21.15.98:0/2159436742 >> v1:172.21.15.98:6789/0 conn(0x7f4b941584d0 legacy=0x7f4b9406dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:18.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.968+0000 7f4b9259c640 1 --2- 172.21.15.98:0/2159436742 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b80003970 0x7f4b80003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.968+0000 7f4b9259c640 1 --2- 172.21.15.98:0/2159436742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b800047c0 0x7f4b80004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.969+0000 7f4b9259c640 1 --2- 172.21.15.98:0/2159436742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b800051f0 0x7f4b8000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.969+0000 7f4b9259c640 1 -- 172.21.15.98:0/2159436742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b940c5f50 con 0x7f4b800051f0 2023-11-28T17:45:18.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.969+0000 7f4b9259c640 1 -- 172.21.15.98:0/2159436742 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8000c150 con 0x7f4b800047c0 2023-11-28T17:45:18.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.969+0000 7f4b9259c640 1 -- 172.21.15.98:0/2159436742 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8000c310 con 0x7f4b80003970 2023-11-28T17:45:18.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.969+0000 7f4b8bfff640 1 -- 172.21.15.98:0/2159436742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b800047c0 msgr2=0x7f4b80004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:18.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.969+0000 7f4b8bfff640 1 --2- 172.21.15.98:0/2159436742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b800047c0 0x7f4b80004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:18.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.969+0000 7f4b9359e640 1 --2- 172.21.15.98:0/2159436742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b800051f0 0x7f4b8000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:18.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.969+0000 7f4b92d9d640 1 --2- 172.21.15.98:0/2159436742 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b80003970 0x7f4b80003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:18.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.969+0000 7f4b9359e640 1 -- 172.21.15.98:0/2159436742 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b80003970 msgr2=0x7f4b80003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:18.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.969+0000 7f4b9359e640 1 --2- 172.21.15.98:0/2159436742 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b80003970 0x7f4b80003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.969+0000 7f4b9359e640 1 -- 172.21.15.98:0/2159436742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b800047c0 msgr2=0x7f4b80004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:18.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.969+0000 7f4b9359e640 1 --2- 172.21.15.98:0/2159436742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b800047c0 0x7f4b80004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.969+0000 7f4b9359e640 1 -- 172.21.15.98:0/2159436742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b80003520 con 0x7f4b800051f0 2023-11-28T17:45:18.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.970+0000 7f4b9359e640 1 -- 172.21.15.98:0/2159436742 reap_dead start 2023-11-28T17:45:18.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.970+0000 7f4b9359e640 1 --2- 172.21.15.98:0/2159436742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b800051f0 0x7f4b8000b980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c00a750 tx=0x7f4b800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caaf6c75db391c81 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:18.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.971+0000 7f4b9259c640 1 -- 172.21.15.98:0/2159436742 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c034630 con 0x7f4b800051f0 2023-11-28T17:45:18.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.971+0000 7f4b9259c640 1 -- 172.21.15.98:0/2159436742 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8c02f070 con 0x7f4b800051f0 2023-11-28T17:45:18.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.971+0000 7f4b9259c640 1 -- 172.21.15.98:0/2159436742 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c0349f0 con 0x7f4b800051f0 2023-11-28T17:45:18.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.971+0000 7f4b99173640 1 -- 172.21.15.98:0/2159436742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b800051f0 msgr2=0x7f4b8000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:18.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.971+0000 7f4b99173640 1 --2- 172.21.15.98:0/2159436742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b800051f0 0x7f4b8000b980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c00a750 tx=0x7f4b800057f0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.971+0000 7f4b99173640 1 -- 172.21.15.98:0/2159436742 shutdown_connections 2023-11-28T17:45:18.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.971+0000 7f4b99173640 1 --2- 172.21.15.98:0/2159436742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b800051f0 0x7f4b8000b980 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.971+0000 7f4b99173640 1 -- 172.21.15.98:0/2159436742 >> 172.21.15.98:0/2159436742 conn(0x7f4b9406c0f0 msgr2=0x7f4b9406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:18.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.971+0000 7f4b99173640 1 -- 172.21.15.98:0/2159436742 shutdown_connections 2023-11-28T17:45:18.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.972+0000 7f4b99173640 1 -- 172.21.15.98:0/2159436742 wait complete. 2023-11-28T17:45:18.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.973+0000 7f4b99173640 1 Processor -- start 2023-11-28T17:45:18.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.973+0000 7f4b99173640 1 -- start start 2023-11-28T17:45:18.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.974+0000 7f4b99173640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b800051f0 0x7f4b94154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.974+0000 7f4b99173640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b9414f890 0x7f4b941efa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.974+0000 7f4b99173640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b941effa0 0x7f4b9414ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.974+0000 7f4b99173640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b940bdd40 con 0x7f4b9414f890 2023-11-28T17:45:18.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.974+0000 7f4b99173640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b940c1560 con 0x7f4b800051f0 2023-11-28T17:45:18.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.974+0000 7f4b99173640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9415ac10 con 0x7f4b941effa0 2023-11-28T17:45:18.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.975+0000 7f4b92d9d640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b800051f0 msgr2=0x7f4b94154190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:18.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.975+0000 7f4b92d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b800051f0 0x7f4b94154190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:18.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.975+0000 7f4b8bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b9414f890 0x7f4b941efa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:18.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.975+0000 7f4b8bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b9414f890 0x7f4b941efa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50802/0 (socket says 172.21.15.98:50802) 2023-11-28T17:45:18.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.975+0000 7f4b8bfff640 1 -- 172.21.15.98:0/1904913136 learned_addr learned my addr 172.21.15.98:0/1904913136 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.975+0000 7f4b9359e640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b941effa0 0x7f4b9414ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.975+0000 7f4b8bfff640 1 -- 172.21.15.98:0/1904913136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b941effa0 msgr2=0x7f4b9414ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.975+0000 7f4b8bfff640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b941effa0 0x7f4b9414ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.975+0000 7f4b8bfff640 1 -- 172.21.15.98:0/1904913136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b800051f0 msgr2=0x7f4b94154190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:45:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.975+0000 7f4b8bfff640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b800051f0 0x7f4b94154190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.975+0000 7f4b8bfff640 1 -- 172.21.15.98:0/1904913136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b941ef280 con 0x7f4b9414f890 2023-11-28T17:45:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.975+0000 7f4b9359e640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4b941effa0 0x7f4b9414ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:45:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.975+0000 7f4b8bfff640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b9414f890 0x7f4b941efa60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00b570 tx=0x7f4b7c00ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.976+0000 7f4b8b7fe640 1 -- 172.21.15.98:0/1904913136 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c003e50 con 0x7f4b9414f890 2023-11-28T17:45:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.976+0000 7f4b8b7fe640 1 -- 172.21.15.98:0/1904913136 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7c005070 con 0x7f4b9414f890 2023-11-28T17:45:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.976+0000 7f4b99173640 1 -- 172.21.15.98:0/1904913136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b94154f10 con 0x7f4b9414f890 2023-11-28T17:45:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.976+0000 7f4b8b7fe640 1 -- 172.21.15.98:0/1904913136 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c005400 con 0x7f4b9414f890 2023-11-28T17:45:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.976+0000 7f4b99173640 1 -- 172.21.15.98:0/1904913136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b941ef540 con 0x7f4b9414f890 2023-11-28T17:45:18.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.977+0000 7f4b99173640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b9414f1b0 0x7f4b941f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.977+0000 7f4b99173640 1 -- 172.21.15.98:0/1904913136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b940744d0 con 0x7f4b9414f1b0 2023-11-28T17:45:18.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.977+0000 7f4b92d9d640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b9414f1b0 0x7f4b941f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:18.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.977+0000 7f4b92d9d640 1 -- 172.21.15.98:0/1904913136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b940c6240 con 0x7f4b9414f1b0 2023-11-28T17:45:18.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.978+0000 7f4b92d9d640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b9414f1b0 0x7f4b941f27f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400d4d0 tx=0x7f4b8400d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:18.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.978+0000 7f4b8b7fe640 1 -- 172.21.15.98:0/1904913136 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b84003bc0 con 0x7f4b9414f1b0 2023-11-28T17:45:18.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.978+0000 7f4b8b7fe640 1 -- 172.21.15.98:0/1904913136 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4b7c005560 con 0x7f4b9414f890 2023-11-28T17:45:18.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.979+0000 7f4b8b7fe640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4b7406b340 0x7f4b7406d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.979+0000 7f4b8b7fe640 1 -- 172.21.15.98:0/1904913136 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4b7c089cc0 con 0x7f4b9414f890 2023-11-28T17:45:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.979+0000 7f4b9359e640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4b7406b340 0x7f4b7406d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.979+0000 7f4b8b7fe640 1 -- 172.21.15.98:0/1904913136 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f4b84003d20 con 0x7f4b9414f1b0 2023-11-28T17:45:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.979+0000 7f4b8b7fe640 1 -- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b9414f1b0 msgr2=0x7f4b941f27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.979+0000 7f4b8b7fe640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b9414f1b0 0x7f4b941f27f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400d4d0 tx=0x7f4b8400d990 comp rx=0 tx=0).stop 2023-11-28T17:45:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.979+0000 7f4b8b7fe640 1 -- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b9414f1b0 msgr2=0x7f4b941f27f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.979+0000 7f4b8b7fe640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b9414f1b0 0x7f4b941f27f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.980+0000 7f4b9359e640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4b7406b340 0x7f4b7406d800 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000c120 tx=0x7f4b8c02f000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:18.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.987+0000 7f4b99173640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b941f2d30 0x7f4b941f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:18.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.987+0000 7f4b99173640 1 -- 172.21.15.98:0/1904913136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b941509d0 con 0x7f4b941f2d30 2023-11-28T17:45:18.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.987+0000 7f4b92d9d640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b941f2d30 0x7f4b941f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:18.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.987+0000 7f4b92d9d640 1 -- 172.21.15.98:0/1904913136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4b941575c0 con 0x7f4b941f2d30 2023-11-28T17:45:18.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.987+0000 7f4b92d9d640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b941f2d30 0x7f4b941f3140 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400d4d0 tx=0x7f4b84005a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:18.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.988+0000 7f4b8b7fe640 1 -- 172.21.15.98:0/1904913136 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8401d220 con 0x7f4b941f2d30 2023-11-28T17:45:18.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.988+0000 7f4b8b7fe640 1 -- 172.21.15.98:0/1904913136 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4456 (secure 0 0 0) 0x7f4b84005870 con 0x7f4b941f2d30 2023-11-28T17:45:18.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.988+0000 7f4b8b7fe640 1 -- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b941f2d30 msgr2=0x7f4b941f3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:18.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.988+0000 7f4b8b7fe640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b941f2d30 0x7f4b941f3140 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400d4d0 tx=0x7f4b84005a80 comp rx=0 tx=0).stop 2023-11-28T17:45:18.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.988+0000 7f4b8b7fe640 1 -- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b941f2d30 msgr2=0x7f4b941f3140 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:18.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.988+0000 7f4b8b7fe640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b941f2d30 0x7f4b941f3140 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400d4d0 tx=0x7f4b84005a80 comp rx=0 tx=0).stop 2023-11-28T17:45:18.990 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:45:18.990 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:45:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:45:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:45:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 40, 2023-11-28T17:45:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:45:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:45:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:45:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:45:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:45:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:45:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:45:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:45:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:45:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:45:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:45:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:45:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:45:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:45:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:45:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:45:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:45:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:45:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:45:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:45:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:45:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:45:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:45:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:45:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:45:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:45:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:45:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:45:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:45:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:45:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:45:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:45:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:45:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:45:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:45:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:45:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:45:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:45:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:45:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:45:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:45:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:45:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:45:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:45:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:45:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:45:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:45:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.003 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:45:19.003 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:45:19.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.988+0000 7f4b99173640 1 -- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4b7406b340 msgr2=0x7f4b7406d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:19.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.988+0000 7f4b99173640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4b7406b340 0x7f4b7406d800 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000c120 tx=0x7f4b8c02f000 comp rx=0 tx=0).stop 2023-11-28T17:45:19.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.988+0000 7f4b99173640 1 -- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b9414f890 msgr2=0x7f4b941efa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:19.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.988+0000 7f4b99173640 1 --2- 172.21.15.98:0/1904913136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b9414f890 0x7f4b941efa60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00b570 tx=0x7f4b7c00ba30 comp rx=0 tx=0).stop 2023-11-28T17:45:19.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.988+0000 7f4b9359e640 1 -- 172.21.15.98:0/1904913136 reap_dead start 2023-11-28T17:45:19.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.989+0000 7f4b9359e640 1 -- 172.21.15.98:0/1904913136 reap_dead start 2023-11-28T17:45:19.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.989+0000 7f4b99173640 1 -- 172.21.15.98:0/1904913136 shutdown_connections 2023-11-28T17:45:19.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.989+0000 7f4b99173640 1 -- 172.21.15.98:0/1904913136 >> 172.21.15.98:0/1904913136 conn(0x7f4b9406c0f0 msgr2=0x7f4b94156970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:19.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.989+0000 7f4b99173640 1 -- 172.21.15.98:0/1904913136 shutdown_connections 2023-11-28T17:45:19.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:18.989+0000 7f4b99173640 1 -- 172.21.15.98:0/1904913136 wait complete. 2023-11-28T17:45:19.006 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:45:19.006 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-11-28T17:45:19.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:45:19.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.214+0000 7f3c74b78640 1 Processor -- start 2023-11-28T17:45:19.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.214+0000 7f3c74b78640 1 -- start start 2023-11-28T17:45:19.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.216+0000 7f3c74b78640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3c70154030 0x7f3c70070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:19.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.216+0000 7f3c74b78640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c700c8600 0x7f3c70070e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:19.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.216+0000 7f3c74b78640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c70154410 con 0x7f3c7006dcf0 2023-11-28T17:45:19.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.216+0000 7f3c74b78640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c700c5da0 con 0x7f3c70154030 2023-11-28T17:45:19.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.216+0000 7f3c74b78640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7015d370 con 0x7f3c700c8600 2023-11-28T17:45:19.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.216+0000 7f3c6e59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3c70154030 0x7f3c70070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:19.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.216+0000 7f3c6ed9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3c7006dcf0 0x7f3c700caff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40444/0 (socket says 172.21.15.98:40444) 2023-11-28T17:45:19.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.216+0000 7f3c6ed9d640 1 -- 172.21.15.98:0/637647680 learned_addr learned my addr 172.21.15.98:0/637647680 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:45:19.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.217+0000 7f3c6dd9b640 1 --2- 172.21.15.98:0/637647680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c700c8600 0x7f3c70070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:19.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.217+0000 7f3c6e59c640 1 -- 172.21.15.98:0/637647680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c700c8600 msgr2=0x7f3c70070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:19.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.217+0000 7f3c6e59c640 1 --2- 172.21.15.98:0/637647680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c700c8600 0x7f3c70070e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:19.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.217+0000 7f3c6e59c640 1 -- 172.21.15.98:0/637647680 >> v1:172.21.15.98:6789/0 conn(0x7f3c7006dcf0 legacy=0x7f3c700caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:19.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.217+0000 7f3c6e59c640 1 -- 172.21.15.98:0/637647680 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7015d650 con 0x7f3c70154030 2023-11-28T17:45:19.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.217+0000 7f3c6d59a640 1 -- 172.21.15.98:0/637647680 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498716743 0 0) 0x7f3c70154410 con 0x7f3c7006dcf0 2023-11-28T17:45:19.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.218+0000 7f3c6e59c640 1 --2- 172.21.15.98:0/637647680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c70154030 0x7f3c70070660 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3c600097a0 tx=0x7f3c6002b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a91751efcfc0b0c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:19.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.218+0000 7f3c6d59a640 1 -- 172.21.15.98:0/637647680 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c60030800 con 0x7f3c70154030 2023-11-28T17:45:19.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.218+0000 7f3c6d59a640 1 -- 172.21.15.98:0/637647680 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c60030960 con 0x7f3c70154030 2023-11-28T17:45:19.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.218+0000 7f3c6d59a640 1 -- 172.21.15.98:0/637647680 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c60030cf0 con 0x7f3c70154030 2023-11-28T17:45:19.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.219+0000 7f3c74b78640 1 -- 172.21.15.98:0/637647680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c70154030 msgr2=0x7f3c70070660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:19.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.219+0000 7f3c74b78640 1 --2- 172.21.15.98:0/637647680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c70154030 0x7f3c70070660 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3c600097a0 tx=0x7f3c6002b940 comp rx=0 tx=0).stop 2023-11-28T17:45:19.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.219+0000 7f3c74b78640 1 -- 172.21.15.98:0/637647680 shutdown_connections 2023-11-28T17:45:19.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.219+0000 7f3c74b78640 1 --2- 172.21.15.98:0/637647680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c700c8600 0x7f3c70070e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:19.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.219+0000 7f3c74b78640 1 --2- 172.21.15.98:0/637647680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c70154030 0x7f3c70070660 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:19.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.219+0000 7f3c74b78640 1 -- 172.21.15.98:0/637647680 >> 172.21.15.98:0/637647680 conn(0x7f3c7006c0f0 msgr2=0x7f3c7006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:19.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.219+0000 7f3c74b78640 1 -- 172.21.15.98:0/637647680 shutdown_connections 2023-11-28T17:45:19.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.219+0000 7f3c74b78640 1 -- 172.21.15.98:0/637647680 wait complete. 2023-11-28T17:45:19.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.221+0000 7f3c74b78640 1 Processor -- start 2023-11-28T17:45:19.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.221+0000 7f3c74b78640 1 -- start start 2023-11-28T17:45:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.222+0000 7f3c74b78640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c7006dcf0 0x7f3c7015a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.222+0000 7f3c74b78640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700c8600 0x7f3c70151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.222+0000 7f3c74b78640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c70154030 0x7f3c701f3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.222+0000 7f3c74b78640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c70159870 con 0x7f3c70154030 2023-11-28T17:45:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.222+0000 7f3c74b78640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c701490a0 con 0x7f3c700c8600 2023-11-28T17:45:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.222+0000 7f3c74b78640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7015a980 con 0x7f3c7006dcf0 2023-11-28T17:45:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.222+0000 7f3c6dd9b640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700c8600 msgr2=0x7f3c70151750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.222+0000 7f3c6dd9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700c8600 0x7f3c70151750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.222+0000 7f3c6ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c70154030 0x7f3c701f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:19.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.222+0000 7f3c6ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c70154030 0x7f3c701f3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50832/0 (socket says 172.21.15.98:50832) 2023-11-28T17:45:19.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.222+0000 7f3c6ed9d640 1 -- 172.21.15.98:0/771452604 learned_addr learned my addr 172.21.15.98:0/771452604 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:19.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.222+0000 7f3c6e59c640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c7006dcf0 0x7f3c7015a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:19.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.223+0000 7f3c6ed9d640 1 -- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c7006dcf0 msgr2=0x7f3c7015a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:19.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.223+0000 7f3c6ed9d640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c7006dcf0 0x7f3c7015a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:19.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.223+0000 7f3c6ed9d640 1 -- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700c8600 msgr2=0x7f3c70151750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:45:19.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.223+0000 7f3c6ed9d640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c700c8600 0x7f3c70151750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:19.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.223+0000 7f3c6ed9d640 1 -- 172.21.15.98:0/771452604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c60009400 con 0x7f3c70154030 2023-11-28T17:45:19.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.223+0000 7f3c6ed9d640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c70154030 0x7f3c701f3050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800cb30 tx=0x7f3c6800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:19.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.223+0000 7f3c5effd640 1 -- 172.21.15.98:0/771452604 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6800ac40 con 0x7f3c70154030 2023-11-28T17:45:19.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.223+0000 7f3c5effd640 1 -- 172.21.15.98:0/771452604 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6800adc0 con 0x7f3c70154030 2023-11-28T17:45:19.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.223+0000 7f3c74b78640 1 -- 172.21.15.98:0/771452604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c701f28e0 con 0x7f3c70154030 2023-11-28T17:45:19.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.223+0000 7f3c5effd640 1 -- 172.21.15.98:0/771452604 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68012640 con 0x7f3c70154030 2023-11-28T17:45:19.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.223+0000 7f3c74b78640 1 -- 172.21.15.98:0/771452604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c70149590 con 0x7f3c70154030 2023-11-28T17:45:19.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.224+0000 7f3c74b78640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c701f3590 0x7f3c701430e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:19.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.224+0000 7f3c74b78640 1 -- 172.21.15.98:0/771452604 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c700744d0 con 0x7f3c701f3590 2023-11-28T17:45:19.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.225+0000 7f3c6e59c640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c701f3590 0x7f3c701430e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:19.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.225+0000 7f3c5effd640 1 -- 172.21.15.98:0/771452604 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3c68007070 con 0x7f3c70154030 2023-11-28T17:45:19.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.226+0000 7f3c6e59c640 1 -- 172.21.15.98:0/771452604 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c70070ba0 con 0x7f3c701f3590 2023-11-28T17:45:19.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.226+0000 7f3c6e59c640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c701f3590 0x7f3c701430e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3c700c8f90 tx=0x7f3c60004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:19.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.226+0000 7f3c5effd640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3c5006b280 0x7f3c5006d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:19.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.227+0000 7f3c5effd640 1 -- 172.21.15.98:0/771452604 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3c68089be0 con 0x7f3c70154030 2023-11-28T17:45:19.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.227+0000 7f3c6dd9b640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3c5006b280 0x7f3c5006d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:19.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.227+0000 7f3c5effd640 1 -- 172.21.15.98:0/771452604 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c60048070 con 0x7f3c701f3590 2023-11-28T17:45:19.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.227+0000 7f3c6dd9b640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3c5006b280 0x7f3c5006d740 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3c64006e80 tx=0x7f3c64006e10 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:19.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.227+0000 7f3c5effd640 1 -- 172.21.15.98:0/771452604 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f3c60039aa0 con 0x7f3c701f3590 2023-11-28T17:45:19.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.227+0000 7f3c5effd640 1 -- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c701f3590 msgr2=0x7f3c701430e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:19.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.227+0000 7f3c5effd640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c701f3590 0x7f3c701430e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3c700c8f90 tx=0x7f3c60004730 comp rx=0 tx=0).stop 2023-11-28T17:45:19.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.227+0000 7f3c5effd640 1 -- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c701f3590 msgr2=0x7f3c701430e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:19.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.227+0000 7f3c5effd640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c701f3590 0x7f3c701430e0 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3c700c8f90 tx=0x7f3c60004730 comp rx=0 tx=0).stop 2023-11-28T17:45:19.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.235+0000 7f3c74b78640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c701f2a70 0x7f3c70143620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:19.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.235+0000 7f3c74b78640 1 -- 172.21.15.98:0/771452604 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c701526f0 con 0x7f3c701f2a70 2023-11-28T17:45:19.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.235+0000 7f3c6e59c640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c701f2a70 0x7f3c70143620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:19.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.236+0000 7f3c6e59c640 1 -- 172.21.15.98:0/771452604 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3c7015dc10 con 0x7f3c701f2a70 2023-11-28T17:45:19.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.237+0000 7f3c6e59c640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c701f2a70 0x7f3c70143620 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3c6002b980 tx=0x7f3c600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:19.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.237+0000 7f3c5effd640 1 -- 172.21.15.98:0/771452604 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6003e040 con 0x7f3c701f2a70 2023-11-28T17:45:19.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.238+0000 7f3c5effd640 1 -- 172.21.15.98:0/771452604 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4964 (secure 0 0 0) 0x7f3c6004ce90 con 0x7f3c701f2a70 2023-11-28T17:45:19.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.238+0000 7f3c5effd640 1 -- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c701f2a70 msgr2=0x7f3c70143620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:19.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.238+0000 7f3c5effd640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c701f2a70 0x7f3c70143620 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3c6002b980 tx=0x7f3c600046c0 comp rx=0 tx=0).stop 2023-11-28T17:45:19.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.238+0000 7f3c5effd640 1 -- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c701f2a70 msgr2=0x7f3c70143620 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:19.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.238+0000 7f3c5effd640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3c701f2a70 0x7f3c70143620 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3c6002b980 tx=0x7f3c600046c0 comp rx=0 tx=0).stop 2023-11-28T17:45:19.240 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:45:19.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:45:19.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:45:19.240 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:45:19.240 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 40, 2023-11-28T17:45:19.241 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:45:19.241 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:45:19.241 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:45:19.241 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:19.241 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:45:19.241 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:19.241 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:45:19.241 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:45:19.241 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:19.241 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:19.241 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:19.241 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:19.242 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:19.242 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:19.242 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:19.242 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:19.242 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:19.242 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:19.242 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:19.242 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:45:19.242 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:45:19.242 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:19.242 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:19.243 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:19.243 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:19.243 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:19.243 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:19.243 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:19.243 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:19.243 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:19.243 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:19.243 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:19.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.243 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:45:19.243 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:45:19.244 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:45:19.244 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:45:19.244 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:45:19.244 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:45:19.244 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:19.244 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:19.244 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:45:19.244 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:45:19.244 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:45:19.244 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:45:19.244 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:45:19.244 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:45:19.244 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:45:19.245 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:19.245 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:45:19.245 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:19.245 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:19.245 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:19.245 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:19.245 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:19.245 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:19.245 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:19.245 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:19.245 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:19.245 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:19.246 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:19.246 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:45:19.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.246 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:45:19.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:45:19.246 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:45:19.246 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:19.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:19.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:19.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:45:19.247 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:19.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:19.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:45:19.247 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:19.247 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:19.247 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:19.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:45:19.247 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:45:19.247 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:45:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:45:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:45:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:19.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:19.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:45:19.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:19.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:19.249 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:19.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:45:19.249 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:45:19.249 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:19.249 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:19.249 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:19.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:45:19.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:45:19.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:19.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:19.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:19.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:45:19.250 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:19.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:19.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:45:19.251 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:19.251 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:19.251 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:19.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:45:19.251 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:45:19.251 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:19.251 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:19.251 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:19.251 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:19.252 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:19.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.252 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:45:19.252 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:45:19.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.252 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:19.252 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:19.252 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:45:19.252 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:19.252 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:19.252 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:45:19.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.252 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:19.253 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:19.253 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:45:19.253 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:19.253 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:19.253 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:45:19.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.253 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:19.253 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:19.253 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 8 2023-11-28T17:45:19.253 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:19.253 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:19.253 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:45:19.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.254 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:19.254 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:19.254 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:45:19.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:19.254 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:19.254 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:45:19.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:19.254 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:19.254 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:19.254 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:45:19.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:19.254 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:19.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:19.255 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:45:19.255 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:45:19.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.238+0000 7f3c74b78640 1 -- 172.21.15.98:0/771452604 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3c5006b280 msgr2=0x7f3c5006d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:19.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.238+0000 7f3c74b78640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3c5006b280 0x7f3c5006d740 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3c64006e80 tx=0x7f3c64006e10 comp rx=0 tx=0).stop 2023-11-28T17:45:19.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.238+0000 7f3c74b78640 1 -- 172.21.15.98:0/771452604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c70154030 msgr2=0x7f3c701f3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:19.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.238+0000 7f3c74b78640 1 --2- 172.21.15.98:0/771452604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c70154030 0x7f3c701f3050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800cb30 tx=0x7f3c6800a060 comp rx=0 tx=0).stop 2023-11-28T17:45:19.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.238+0000 7f3c6ed9d640 1 -- 172.21.15.98:0/771452604 reap_dead start 2023-11-28T17:45:19.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.238+0000 7f3c6ed9d640 1 -- 172.21.15.98:0/771452604 reap_dead start 2023-11-28T17:45:19.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.239+0000 7f3c74b78640 1 -- 172.21.15.98:0/771452604 shutdown_connections 2023-11-28T17:45:19.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.239+0000 7f3c74b78640 1 -- 172.21.15.98:0/771452604 >> 172.21.15.98:0/771452604 conn(0x7f3c7006c0f0 msgr2=0x7f3c70070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:19.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.239+0000 7f3c74b78640 1 -- 172.21.15.98:0/771452604 shutdown_connections 2023-11-28T17:45:19.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:19.239+0000 7f3c74b78640 1 -- 172.21.15.98:0/771452604 wait complete. 2023-11-28T17:45:19.259 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:45:19.259 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-11-28T17:45:25.356 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~34s 2023-11-28T17:45:32.072 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~40s 2023-11-28T17:45:38.785 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~47s 2023-11-28T17:45:38.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:45:38.840 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:45:39.260 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-11-28T17:45:39.260 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-11-28T17:45:39.260 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-28T17:45:39.261 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-28T17:45:39.270 INFO:tasks.ceph.mon.b:Started 2023-11-28T17:45:39.270 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:45:39.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:45:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.490+0000 7faa8161a640 1 Processor -- start 2023-11-28T17:45:39.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.491+0000 7faa8161a640 1 -- start start 2023-11-28T17:45:39.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.492+0000 7faa8161a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faa7c0b0f20 0x7faa7c06a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:39.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.492+0000 7faa8161a640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7faa7c15a150 0x7faa7c071fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:39.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.492+0000 7faa8161a640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7c15e100 con 0x7faa7c159d30 2023-11-28T17:45:39.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.492+0000 7faa8161a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa7c06ab60 con 0x7faa7c0b0f20 2023-11-28T17:45:39.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.492+0000 7faa8161a640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa7c15d340 con 0x7faa7c15a150 2023-11-28T17:45:39.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.492+0000 7faa7bfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faa7c0b0f20 0x7faa7c06a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:39.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.492+0000 7faa7bfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faa7c0b0f20 0x7faa7c06a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41374/0 (socket says 172.21.15.98:41374) 2023-11-28T17:45:39.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.493+0000 7faa7bfff640 1 -- 172.21.15.98:0/4146257574 learned_addr learned my addr 172.21.15.98:0/4146257574 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:39.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.493+0000 7faa80e19640 1 --2- 172.21.15.98:0/4146257574 >> v2:172.21.15.113:3300/0 conn(0x7faa7c15a150 0x7faa7c071fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:39.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.493+0000 7faa80e19640 1 -- 172.21.15.98:0/4146257574 >> v2:172.21.15.113:3300/0 conn(0x7faa7c15a150 msgr2=0x7faa7c071fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:45:39.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.493+0000 7faa80e19640 1 -- 172.21.15.98:0/4146257574 >> v2:172.21.15.113:3300/0 conn(0x7faa7c15a150 msgr2=0x7faa7c071fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:39.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.493+0000 7faa80e19640 1 --2- 172.21.15.98:0/4146257574 >> v2:172.21.15.113:3300/0 conn(0x7faa7c15a150 0x7faa7c071fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:39.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.493+0000 7faa80e19640 1 --2- 172.21.15.98:0/4146257574 >> v2:172.21.15.113:3300/0 conn(0x7faa7c15a150 0x7faa7c071fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:39.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.493+0000 7faa7bfff640 1 -- 172.21.15.98:0/4146257574 >> v2:172.21.15.113:3300/0 conn(0x7faa7c15a150 msgr2=0x7faa7c071fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:39.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.493+0000 7faa7bfff640 1 --2- 172.21.15.98:0/4146257574 >> v2:172.21.15.113:3300/0 conn(0x7faa7c15a150 0x7faa7c071fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:39.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.493+0000 7faa7bfff640 1 -- 172.21.15.98:0/4146257574 >> v1:172.21.15.113:6789/0 conn(0x7faa7c159d30 legacy=0x7faa7c0c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:39.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.493+0000 7faa7bfff640 1 -- 172.21.15.98:0/4146257574 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c15d620 con 0x7faa7c0b0f20 2023-11-28T17:45:39.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.494+0000 7faa7bfff640 1 --2- 172.21.15.98:0/4146257574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c0b0f20 0x7faa7c06a4d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0cd0b0 tx=0x7faa6c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39088c735c5871b0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:39.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.494+0000 7faa7b7fe640 1 -- 172.21.15.98:0/4146257574 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c030860 con 0x7faa7c0b0f20 2023-11-28T17:45:39.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.495+0000 7faa7b7fe640 1 -- 172.21.15.98:0/4146257574 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6c0309c0 con 0x7faa7c0b0f20 2023-11-28T17:45:39.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.495+0000 7faa7b7fe640 1 -- 172.21.15.98:0/4146257574 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c030d50 con 0x7faa7c0b0f20 2023-11-28T17:45:39.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.495+0000 7faa8161a640 1 -- 172.21.15.98:0/4146257574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c0b0f20 msgr2=0x7faa7c06a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:39.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.495+0000 7faa8161a640 1 --2- 172.21.15.98:0/4146257574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c0b0f20 0x7faa7c06a4d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0cd0b0 tx=0x7faa6c02b9a0 comp rx=0 tx=0).stop 2023-11-28T17:45:39.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.495+0000 7faa8161a640 1 -- 172.21.15.98:0/4146257574 shutdown_connections 2023-11-28T17:45:39.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.495+0000 7faa8161a640 1 --2- 172.21.15.98:0/4146257574 >> v2:172.21.15.113:3300/0 conn(0x7faa7c15a150 0x7faa7c071fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:39.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.495+0000 7faa8161a640 1 --2- 172.21.15.98:0/4146257574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c0b0f20 0x7faa7c06a4d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:39.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.495+0000 7faa8161a640 1 -- 172.21.15.98:0/4146257574 >> 172.21.15.98:0/4146257574 conn(0x7faa7c066230 msgr2=0x7faa7c066610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:39.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.495+0000 7faa8161a640 1 -- 172.21.15.98:0/4146257574 shutdown_connections 2023-11-28T17:45:39.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.495+0000 7faa8161a640 1 -- 172.21.15.98:0/4146257574 wait complete. 2023-11-28T17:45:39.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.497+0000 7faa8161a640 1 Processor -- start 2023-11-28T17:45:39.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.497+0000 7faa8161a640 1 -- start start 2023-11-28T17:45:39.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.498+0000 7faa8161a640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faa7c0b0f20 0x7faa7c1f3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:39.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.498+0000 7faa8161a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c159d30 0x7faa7c0ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:39.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.498+0000 7faa8161a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa7c15a150 0x7faa7c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:39.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.498+0000 7faa8161a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c0ad9f0 con 0x7faa7c159d30 2023-11-28T17:45:39.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.498+0000 7faa8161a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c1e3040 con 0x7faa7c15a150 2023-11-28T17:45:39.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.498+0000 7faa8161a640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c1f38d0 con 0x7faa7c0b0f20 2023-11-28T17:45:39.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa73fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c159d30 0x7faa7c0ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:39.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa73fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c159d30 0x7faa7c0ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41390/0 (socket says 172.21.15.98:41390) 2023-11-28T17:45:39.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa73fff640 1 -- 172.21.15.98:0/2326681574 learned_addr learned my addr 172.21.15.98:0/2326681574 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:39.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa7bfff640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faa7c0b0f20 0x7faa7c1f3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:39.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa80e19640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa7c15a150 0x7faa7c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:39.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa80e19640 1 -- 172.21.15.98:0/2326681574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa7c15a150 msgr2=0x7faa7c0cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-11-28T17:45:39.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa80e19640 1 -- 172.21.15.98:0/2326681574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa7c15a150 msgr2=0x7faa7c0cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:45:39.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa80e19640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa7c15a150 0x7faa7c0cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:39.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa80e19640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa7c15a150 0x7faa7c0cae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:39.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa73fff640 1 -- 172.21.15.98:0/2326681574 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faa7c0b0f20 msgr2=0x7faa7c1f3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:39.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa73fff640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faa7c0b0f20 0x7faa7c1f3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:39.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa73fff640 1 -- 172.21.15.98:0/2326681574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa7c15a150 msgr2=0x7faa7c0cae00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:45:39.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa73fff640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa7c15a150 0x7faa7c0cae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:39.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa73fff640 1 -- 172.21.15.98:0/2326681574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6c0094d0 con 0x7faa7c159d30 2023-11-28T17:45:39.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa73fff640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c159d30 0x7faa7c0ca8c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa6400c400 tx=0x7faa6400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:39.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.499+0000 7faa797fa640 1 -- 172.21.15.98:0/2326681574 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa640052d0 con 0x7faa7c159d30 2023-11-28T17:45:39.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.500+0000 7faa797fa640 1 -- 172.21.15.98:0/2326681574 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa64005450 con 0x7faa7c159d30 2023-11-28T17:45:39.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.500+0000 7faa797fa640 1 -- 172.21.15.98:0/2326681574 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64007260 con 0x7faa7c159d30 2023-11-28T17:45:39.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.500+0000 7faa8161a640 1 -- 172.21.15.98:0/2326681574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7c151ef0 con 0x7faa7c159d30 2023-11-28T17:45:39.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.500+0000 7faa8161a640 1 -- 172.21.15.98:0/2326681574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7c151650 con 0x7faa7c159d30 2023-11-28T17:45:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.501+0000 7faa8161a640 1 -- 172.21.15.98:0/2326681574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7c0adb60 con 0x7faa7c159d30 2023-11-28T17:45:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.502+0000 7faa797fa640 1 -- 172.21.15.98:0/2326681574 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7faa640073c0 con 0x7faa7c159d30 2023-11-28T17:45:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.502+0000 7faa797fa640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7faa5c06b2d0 0x7faa5c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.503+0000 7faa797fa640 1 -- 172.21.15.98:0/2326681574 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7faa64088f40 con 0x7faa7c159d30 2023-11-28T17:45:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.503+0000 7faa7bfff640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7faa5c06b2d0 0x7faa5c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:39.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.504+0000 7faa7bfff640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7faa5c06b2d0 0x7faa5c06d790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faa6c036000 tx=0x7faa6c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:39.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.508+0000 7faa797fa640 1 -- 172.21.15.98:0/2326681574 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa7c0adb60 con 0x7faa7c159d30 2023-11-28T17:45:39.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.772+0000 7faa8161a640 1 -- 172.21.15.98:0/2326681574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7faa7c0adb30 con 0x7faa7c159d30 2023-11-28T17:45:39.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.773+0000 7faa797fa640 1 -- 172.21.15.98:0/2326681574 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7faa64053de0 con 0x7faa7c159d30 2023-11-28T17:45:39.773 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":40,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:45:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.775+0000 7faa8161a640 1 -- 172.21.15.98:0/2326681574 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7faa5c06b2d0 msgr2=0x7faa5c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.775+0000 7faa8161a640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7faa5c06b2d0 0x7faa5c06d790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faa6c036000 tx=0x7faa6c008030 comp rx=0 tx=0).stop 2023-11-28T17:45:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.775+0000 7faa8161a640 1 -- 172.21.15.98:0/2326681574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c159d30 msgr2=0x7faa7c0ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:39.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.775+0000 7faa8161a640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c159d30 0x7faa7c0ca8c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa6400c400 tx=0x7faa6400c8c0 comp rx=0 tx=0).stop 2023-11-28T17:45:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.776+0000 7faa8161a640 1 -- 172.21.15.98:0/2326681574 shutdown_connections 2023-11-28T17:45:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.776+0000 7faa8161a640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7faa5c06b2d0 0x7faa5c06d790 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.776+0000 7faa8161a640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa7c15a150 0x7faa7c0cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.776+0000 7faa8161a640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c159d30 0x7faa7c0ca8c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.776+0000 7faa8161a640 1 --2- 172.21.15.98:0/2326681574 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faa7c0b0f20 0x7faa7c1f3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.776+0000 7faa8161a640 1 -- 172.21.15.98:0/2326681574 >> 172.21.15.98:0/2326681574 conn(0x7faa7c066230 msgr2=0x7faa7c0c1080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.776+0000 7faa8161a640 1 -- 172.21.15.98:0/2326681574 shutdown_connections 2023-11-28T17:45:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:39.776+0000 7faa8161a640 1 -- 172.21.15.98:0/2326681574 wait complete. 2023-11-28T17:45:42.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:45:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.016+0000 7fdcc5fce640 1 Processor -- start 2023-11-28T17:45:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.016+0000 7fdcc5fce640 1 -- start start 2023-11-28T17:45:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.017+0000 7fdcc5fce640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 0x7fdcc015bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.017+0000 7fdcc5fce640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 0x7fdcc00c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.017+0000 7fdcc5fce640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc015b150 con 0x7fdcc0151d80 2023-11-28T17:45:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.017+0000 7fdcc5fce640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc00c5260 con 0x7fdcc00c4910 2023-11-28T17:45:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.017+0000 7fdcc5fce640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc0157f90 con 0x7fdcc0157b60 2023-11-28T17:45:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.018+0000 7fdcbffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 0x7fdcc00c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.018+0000 7fdcbffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 0x7fdcc00c4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56076/0 (socket says 172.21.15.98:56076) 2023-11-28T17:45:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.018+0000 7fdcbffff640 1 -- 172.21.15.98:0/2450393567 learned_addr learned my addr 172.21.15.98:0/2450393567 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.018+0000 7fdcbffff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 msgr2=0x7fdcc00c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:45:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.018+0000 7fdcbffff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 msgr2=0x7fdcc00c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.018+0000 7fdcbffff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 0x7fdcc00c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.018+0000 7fdcbffff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 0x7fdcc00c4d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.018+0000 7fdcb7fff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 0x7fdcc015bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.018+0000 7fdcb7fff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 msgr2=0x7fdcc015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:45:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.018+0000 7fdcb7fff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 msgr2=0x7fdcc015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.018+0000 7fdcb7fff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 0x7fdcc015bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.018+0000 7fdcb7fff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 0x7fdcc015bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:43.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.219+0000 7fdcbffff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 0x7fdcc00c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.220+0000 7fdcb7fff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 0x7fdcc015bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.220+0000 7fdcbffff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 msgr2=0x7fdcc00c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:45:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.220+0000 7fdcbffff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 msgr2=0x7fdcc00c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.220+0000 7fdcbffff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 0x7fdcc00c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.220+0000 7fdcbffff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 0x7fdcc00c4d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:45:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.220+0000 7fdcb7fff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 msgr2=0x7fdcc015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:45:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.220+0000 7fdcb7fff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 msgr2=0x7fdcc015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.220+0000 7fdcb7fff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 0x7fdcc015bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.220+0000 7fdcb7fff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 0x7fdcc015bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:45:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.621+0000 7fdcbffff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 0x7fdcc00c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.621+0000 7fdcb7fff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 0x7fdcc015bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.622+0000 7fdcbffff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 msgr2=0x7fdcc00c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:45:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.622+0000 7fdcbffff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 msgr2=0x7fdcc00c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.622+0000 7fdcbffff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 0x7fdcc00c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.622+0000 7fdcbffff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 0x7fdcc00c4d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:45:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.622+0000 7fdcb7fff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 msgr2=0x7fdcc015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:45:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.622+0000 7fdcb7fff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 msgr2=0x7fdcc015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.622+0000 7fdcb7fff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 0x7fdcc015bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:43.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:43.622+0000 7fdcb7fff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 0x7fdcc015bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:45:44.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:44.423+0000 7fdcbffff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 0x7fdcc00c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:44.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:44.424+0000 7fdcb7fff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 0x7fdcc015bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:44.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:44.424+0000 7fdcbffff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 msgr2=0x7fdcc00c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:45:44.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:44.424+0000 7fdcbffff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 msgr2=0x7fdcc00c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:44.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:44.424+0000 7fdcbffff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 0x7fdcc00c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:44.424+0000 7fdcbffff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 0x7fdcc00c4d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:45:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:44.424+0000 7fdcb7fff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 msgr2=0x7fdcc015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:45:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:44.424+0000 7fdcb7fff640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 msgr2=0x7fdcc015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:45:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:44.424+0000 7fdcb7fff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 0x7fdcc015bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:45:44.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:44.424+0000 7fdcb7fff640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 0x7fdcc015bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:45:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.138+0000 7fdcbeffd640 1 -- 172.21.15.98:0/2450393567 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283255376 0 0) 0x7fdcc015b150 con 0x7fdcc0151d80 2023-11-28T17:45:45.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.139+0000 7fdcbeffd640 1 -- 172.21.15.98:0/2450393567 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcac003520 con 0x7fdcc0151d80 2023-11-28T17:45:45.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.145+0000 7fdcbeffd640 1 -- 172.21.15.98:0/2450393567 <== mon.1 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1922301763 0 0) 0x7fdcac003520 con 0x7fdcc0151d80 2023-11-28T17:45:45.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.145+0000 7fdcbeffd640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 msgr2=0x7fdcc015bc10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:45.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.145+0000 7fdcbeffd640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.113:3300/0 conn(0x7fdcc0157b60 0x7fdcc015bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.145+0000 7fdcbeffd640 1 -- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 msgr2=0x7fdcc00c4d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:45.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.145+0000 7fdcbeffd640 1 --2- 172.21.15.98:0/2450393567 >> v2:172.21.15.98:3300/0 conn(0x7fdcc00c4910 0x7fdcc00c4d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.145+0000 7fdcbeffd640 1 -- 172.21.15.98:0/2450393567 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc006e1f0 con 0x7fdcc0151d80 2023-11-28T17:45:45.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.145+0000 7fdcbeffd640 1 -- 172.21.15.98:0/2450393567 <== mon.1 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7fdcb0001670 con 0x7fdcc0151d80 2023-11-28T17:45:45.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.145+0000 7fdcbeffd640 1 -- 172.21.15.98:0/2450393567 >> v1:172.21.15.113:6789/0 conn(0x7fdcc0151d80 legacy=0x7fdcc00c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.145+0000 7fdcbeffd640 1 --2- 172.21.15.98:0/2450393567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcac0039c0 0x7fdcac003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.145+0000 7fdcbeffd640 1 --2- 172.21.15.98:0/2450393567 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcac0047c0 0x7fdcac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.145+0000 7fdcbeffd640 1 --2- 172.21.15.98:0/2450393567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcac005180 0x7fdcac00fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.145+0000 7fdcbeffd640 1 -- 172.21.15.98:0/2450393567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00c5260 con 0x7fdcac005180 2023-11-28T17:45:45.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.145+0000 7fdcbeffd640 1 -- 172.21.15.98:0/2450393567 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac010440 con 0x7fdcac0039c0 2023-11-28T17:45:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.145+0000 7fdcbeffd640 1 -- 172.21.15.98:0/2450393567 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac010600 con 0x7fdcac0047c0 2023-11-28T17:45:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.146+0000 7fdcbf7fe640 1 --2- 172.21.15.98:0/2450393567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcac0039c0 0x7fdcac003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.146+0000 7fdcbffff640 1 --2- 172.21.15.98:0/2450393567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcac005180 0x7fdcac00fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.146+0000 7fdcb7fff640 1 --2- 172.21.15.98:0/2450393567 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcac0047c0 0x7fdcac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.147+0000 7fdcbf7fe640 1 -- 172.21.15.98:0/2450393567 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcac0047c0 msgr2=0x7fdcac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.147+0000 7fdcbf7fe640 1 --2- 172.21.15.98:0/2450393567 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcac0047c0 0x7fdcac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.147+0000 7fdcbf7fe640 1 -- 172.21.15.98:0/2450393567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcac005180 msgr2=0x7fdcac00fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.147+0000 7fdcbf7fe640 1 --2- 172.21.15.98:0/2450393567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcac005180 0x7fdcac00fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.147+0000 7fdcbf7fe640 1 -- 172.21.15.98:0/2450393567 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac003520 con 0x7fdcac0039c0 2023-11-28T17:45:45.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.147+0000 7fdcbffff640 1 -- 172.21.15.98:0/2450393567 reap_dead start 2023-11-28T17:45:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.150+0000 7fdcbf7fe640 1 --2- 172.21.15.98:0/2450393567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcac0039c0 0x7fdcac003de0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0007f20 tx=0x7fdcac0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=386dc71cfca02290 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.151+0000 7fdcbeffd640 1 -- 172.21.15.98:0/2450393567 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb002c8b0 con 0x7fdcac0039c0 2023-11-28T17:45:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.151+0000 7fdcbeffd640 1 -- 172.21.15.98:0/2450393567 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb002ca10 con 0x7fdcac0039c0 2023-11-28T17:45:45.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.151+0000 7fdcbeffd640 1 -- 172.21.15.98:0/2450393567 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb002cdc0 con 0x7fdcac0039c0 2023-11-28T17:45:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.152+0000 7fdcc5fce640 1 -- 172.21.15.98:0/2450393567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcac0039c0 msgr2=0x7fdcac003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.152+0000 7fdcc5fce640 1 --2- 172.21.15.98:0/2450393567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcac0039c0 0x7fdcac003de0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0007f20 tx=0x7fdcac0057f0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.153+0000 7fdcc5fce640 1 -- 172.21.15.98:0/2450393567 shutdown_connections 2023-11-28T17:45:45.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.153+0000 7fdcc5fce640 1 --2- 172.21.15.98:0/2450393567 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcac0039c0 0x7fdcac003de0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.153+0000 7fdcc5fce640 1 -- 172.21.15.98:0/2450393567 >> 172.21.15.98:0/2450393567 conn(0x7fdcc006a4d0 msgr2=0x7fdcc006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:45.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.153+0000 7fdcc5fce640 1 -- 172.21.15.98:0/2450393567 shutdown_connections 2023-11-28T17:45:45.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.153+0000 7fdcc5fce640 1 -- 172.21.15.98:0/2450393567 wait complete. 2023-11-28T17:45:45.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.158+0000 7fdcc5fce640 1 Processor -- start 2023-11-28T17:45:45.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.158+0000 7fdcc5fce640 1 -- start start 2023-11-28T17:45:45.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.161+0000 7fdcc5fce640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcac0039c0 0x7fdcc0159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.161+0000 7fdcc5fce640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcc0158d20 0x7fdcc00c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.161+0000 7fdcc5fce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc01eb300 0x7fdcc01eb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.161+0000 7fdcc5fce640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc01ef2d0 con 0x7fdcc01eb300 2023-11-28T17:45:45.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.161+0000 7fdcc5fce640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc01ef760 con 0x7fdcc0158d20 2023-11-28T17:45:45.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.161+0000 7fdcc5fce640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc01597d0 con 0x7fdcac0039c0 2023-11-28T17:45:45.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.161+0000 7fdcbffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc01eb300 0x7fdcc01eb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.161+0000 7fdcbffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc01eb300 0x7fdcc01eb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56124/0 (socket says 172.21.15.98:56124) 2023-11-28T17:45:45.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.161+0000 7fdcbffff640 1 -- 172.21.15.98:0/304123460 learned_addr learned my addr 172.21.15.98:0/304123460 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:45.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.161+0000 7fdcb7fff640 1 --2- 172.21.15.98:0/304123460 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcc0158d20 0x7fdcc00c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.162+0000 7fdcbf7fe640 1 --2- 172.21.15.98:0/304123460 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcac0039c0 0x7fdcc0159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.162+0000 7fdcbffff640 1 -- 172.21.15.98:0/304123460 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcac0039c0 msgr2=0x7fdcc0159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.162+0000 7fdcbffff640 1 --2- 172.21.15.98:0/304123460 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcac0039c0 0x7fdcc0159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.162+0000 7fdcbffff640 1 -- 172.21.15.98:0/304123460 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcc0158d20 msgr2=0x7fdcc00c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.162+0000 7fdcbffff640 1 --2- 172.21.15.98:0/304123460 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcc0158d20 0x7fdcc00c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.162+0000 7fdcbffff640 1 -- 172.21.15.98:0/304123460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc0159a90 con 0x7fdcc01eb300 2023-11-28T17:45:45.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.162+0000 7fdcbffff640 1 --2- 172.21.15.98:0/304123460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc01eb300 0x7fdcc01eb780 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00c1c70 tx=0x7fdcb8007e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:45.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.163+0000 7fdcbcff9640 1 -- 172.21.15.98:0/304123460 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8003c60 con 0x7fdcc01eb300 2023-11-28T17:45:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.163+0000 7fdcbcff9640 1 -- 172.21.15.98:0/304123460 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb800f040 con 0x7fdcc01eb300 2023-11-28T17:45:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.163+0000 7fdcbcff9640 1 -- 172.21.15.98:0/304123460 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8015860 con 0x7fdcc01eb300 2023-11-28T17:45:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.163+0000 7fdcc5fce640 1 -- 172.21.15.98:0/304123460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc01ee9a0 con 0x7fdcc01eb300 2023-11-28T17:45:45.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.163+0000 7fdcc5fce640 1 -- 172.21.15.98:0/304123460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc01eef60 con 0x7fdcc01eb300 2023-11-28T17:45:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.165+0000 7fdcc5fce640 1 -- 172.21.15.98:0/304123460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc01ef120 con 0x7fdcc01eb300 2023-11-28T17:45:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.166+0000 7fdcbcff9640 1 -- 172.21.15.98:0/304123460 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fdcb80159c0 con 0x7fdcc01eb300 2023-11-28T17:45:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.167+0000 7fdcbcff9640 1 --2- 172.21.15.98:0/304123460 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdca006b2a0 0x7fdca006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.167+0000 7fdcbcff9640 1 -- 172.21.15.98:0/304123460 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdcb808a550 con 0x7fdcc01eb300 2023-11-28T17:45:45.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.167+0000 7fdcbf7fe640 1 --2- 172.21.15.98:0/304123460 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdca006b2a0 0x7fdca006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.168+0000 7fdcbf7fe640 1 --2- 172.21.15.98:0/304123460 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdca006b2a0 0x7fdca006d760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0036000 tx=0x7fdcb0030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:45.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.174+0000 7fdcbcff9640 1 -- 172.21.15.98:0/304123460 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdcb8055480 con 0x7fdcc01eb300 2023-11-28T17:45:45.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.434+0000 7fdcc5fce640 1 -- 172.21.15.98:0/304123460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdcc0067bf0 con 0x7fdcc01eb300 2023-11-28T17:45:45.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.435+0000 7fdcbcff9640 1 -- 172.21.15.98:0/304123460 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7fdcb8058aa0 con 0x7fdcc01eb300 2023-11-28T17:45:45.436 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:45:45.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.438+0000 7fdcc5fce640 1 -- 172.21.15.98:0/304123460 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdca006b2a0 msgr2=0x7fdca006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.438+0000 7fdcc5fce640 1 --2- 172.21.15.98:0/304123460 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdca006b2a0 0x7fdca006d760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0036000 tx=0x7fdcb0030000 comp rx=0 tx=0).stop 2023-11-28T17:45:45.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.438+0000 7fdcc5fce640 1 -- 172.21.15.98:0/304123460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc01eb300 msgr2=0x7fdcc01eb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.438+0000 7fdcc5fce640 1 --2- 172.21.15.98:0/304123460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc01eb300 0x7fdcc01eb780 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00c1c70 tx=0x7fdcb8007e20 comp rx=0 tx=0).stop 2023-11-28T17:45:45.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.438+0000 7fdcc5fce640 1 -- 172.21.15.98:0/304123460 shutdown_connections 2023-11-28T17:45:45.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.438+0000 7fdcc5fce640 1 --2- 172.21.15.98:0/304123460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc01eb300 0x7fdcc01eb780 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.438+0000 7fdcc5fce640 1 --2- 172.21.15.98:0/304123460 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdca006b2a0 0x7fdca006d760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.438+0000 7fdcc5fce640 1 --2- 172.21.15.98:0/304123460 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdcc0158d20 0x7fdcc00c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.438+0000 7fdcc5fce640 1 --2- 172.21.15.98:0/304123460 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdcac0039c0 0x7fdcc0159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.438+0000 7fdcc5fce640 1 -- 172.21.15.98:0/304123460 >> 172.21.15.98:0/304123460 conn(0x7fdcc006a4d0 msgr2=0x7fdcc00c7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:45.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.438+0000 7fdcc5fce640 1 -- 172.21.15.98:0/304123460 shutdown_connections 2023-11-28T17:45:45.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.438+0000 7fdcc5fce640 1 -- 172.21.15.98:0/304123460 wait complete. 2023-11-28T17:45:45.457 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:45:45.457 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:45:45.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:45:45.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~54s 2023-11-28T17:45:45.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.661+0000 7f2fb3c07640 1 Processor -- start 2023-11-28T17:45:45.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.661+0000 7f2fb3c07640 1 -- start start 2023-11-28T17:45:45.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.663+0000 7f2fb3c07640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2fac154030 0x7f2fac070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.663+0000 7f2fb3c07640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2fac0c8600 0x7f2fac070e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.663+0000 7f2fb3c07640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2fac06dcf0 0x7f2fac06e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.663+0000 7f2fb3c07640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fac154410 con 0x7f2fac0c8600 2023-11-28T17:45:45.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.663+0000 7f2fb3c07640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fac0c5da0 con 0x7f2fac154030 2023-11-28T17:45:45.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.663+0000 7f2fb3c07640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac15d370 con 0x7f2fac06dcf0 2023-11-28T17:45:45.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.663+0000 7f2fb11a2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2fac0c8600 0x7f2fac070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.663+0000 7f2fb11a2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2fac0c8600 0x7f2fac070e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56128/0 (socket says 172.21.15.98:56128) 2023-11-28T17:45:45.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.663+0000 7f2fb11a2640 1 -- 172.21.15.98:0/1894652613 learned_addr learned my addr 172.21.15.98:0/1894652613 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:45.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.664+0000 7f2fb11a2640 1 -- 172.21.15.98:0/1894652613 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2fac06dcf0 msgr2=0x7f2fac06e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:45.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.664+0000 7f2fb21a4640 1 --2- 172.21.15.98:0/1894652613 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2fac06dcf0 0x7f2fac06e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.664+0000 7f2fb19a3640 1 --2- 172.21.15.98:0/1894652613 >> v2:172.21.15.113:3300/0 conn(0x7f2fac154030 0x7f2fac070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.664+0000 7f2fb11a2640 1 --2- 172.21.15.98:0/1894652613 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2fac06dcf0 0x7f2fac06e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.664+0000 7f2fb11a2640 1 -- 172.21.15.98:0/1894652613 >> v2:172.21.15.113:3300/0 conn(0x7f2fac154030 msgr2=0x7f2fac070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:45.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.664+0000 7f2fb11a2640 1 --2- 172.21.15.98:0/1894652613 >> v2:172.21.15.113:3300/0 conn(0x7f2fac154030 0x7f2fac070660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.664+0000 7f2fb11a2640 1 -- 172.21.15.98:0/1894652613 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac15d650 con 0x7f2fac0c8600 2023-11-28T17:45:45.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.664+0000 7f2fb21a4640 1 --2- 172.21.15.98:0/1894652613 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2fac06dcf0 0x7f2fac06e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:45:45.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.665+0000 7f2fb11a2640 1 --2- 172.21.15.98:0/1894652613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac0c8600 0x7f2fac070e90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2fac06eba0 tx=0x7f2f9c02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6c9801d53035f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:45.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.665+0000 7f2fb09a1640 1 -- 172.21.15.98:0/1894652613 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c030850 con 0x7f2fac0c8600 2023-11-28T17:45:45.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.665+0000 7f2fb09a1640 1 -- 172.21.15.98:0/1894652613 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c0309b0 con 0x7f2fac0c8600 2023-11-28T17:45:45.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.665+0000 7f2fb09a1640 1 -- 172.21.15.98:0/1894652613 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c030d40 con 0x7f2fac0c8600 2023-11-28T17:45:45.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.666+0000 7f2fb3c07640 1 -- 172.21.15.98:0/1894652613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac0c8600 msgr2=0x7f2fac070e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.666+0000 7f2fb3c07640 1 --2- 172.21.15.98:0/1894652613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac0c8600 0x7f2fac070e90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2fac06eba0 tx=0x7f2f9c02b990 comp rx=0 tx=0).stop 2023-11-28T17:45:45.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.666+0000 7f2fb3c07640 1 -- 172.21.15.98:0/1894652613 shutdown_connections 2023-11-28T17:45:45.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.666+0000 7f2fb3c07640 1 --2- 172.21.15.98:0/1894652613 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2fac06dcf0 0x7f2fac06e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.666+0000 7f2fb3c07640 1 --2- 172.21.15.98:0/1894652613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac0c8600 0x7f2fac070e90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.666+0000 7f2fb3c07640 1 --2- 172.21.15.98:0/1894652613 >> v2:172.21.15.113:3300/0 conn(0x7f2fac154030 0x7f2fac070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.666+0000 7f2fb3c07640 1 -- 172.21.15.98:0/1894652613 >> 172.21.15.98:0/1894652613 conn(0x7f2fac06c0f0 msgr2=0x7f2fac06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:45.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.666+0000 7f2fb3c07640 1 -- 172.21.15.98:0/1894652613 shutdown_connections 2023-11-28T17:45:45.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.666+0000 7f2fb3c07640 1 -- 172.21.15.98:0/1894652613 wait complete. 2023-11-28T17:45:45.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.668+0000 7f2fb3c07640 1 Processor -- start 2023-11-28T17:45:45.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.668+0000 7f2fb3c07640 1 -- start start 2023-11-28T17:45:45.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.669+0000 7f2fb3c07640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2fac06dcf0 0x7f2fac152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.669+0000 7f2fb3c07640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fac0c8600 0x7f2fac14c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.669+0000 7f2fb3c07640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac154030 0x7f2fac14c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.669+0000 7f2fb3c07640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac1f03b0 con 0x7f2fac154030 2023-11-28T17:45:45.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.669+0000 7f2fb3c07640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac157e60 con 0x7f2fac0c8600 2023-11-28T17:45:45.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.669+0000 7f2fb3c07640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac1efe50 con 0x7f2fac06dcf0 2023-11-28T17:45:45.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.669+0000 7f2fb21a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac154030 0x7f2fac14c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.669+0000 7f2fb21a4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac154030 0x7f2fac14c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56134/0 (socket says 172.21.15.98:56134) 2023-11-28T17:45:45.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.669+0000 7f2fb21a4640 1 -- 172.21.15.98:0/3301698938 learned_addr learned my addr 172.21.15.98:0/3301698938 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:45.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.670+0000 7f2fb19a3640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2fac06dcf0 0x7f2fac152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.670+0000 7f2fb11a2640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fac0c8600 0x7f2fac14c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.670+0000 7f2fb21a4640 1 -- 172.21.15.98:0/3301698938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2fac06dcf0 msgr2=0x7f2fac152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.670+0000 7f2fb21a4640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2fac06dcf0 0x7f2fac152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.670+0000 7f2fb21a4640 1 -- 172.21.15.98:0/3301698938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fac0c8600 msgr2=0x7f2fac14c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.670+0000 7f2fb21a4640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fac0c8600 0x7f2fac14c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.670+0000 7f2fb21a4640 1 -- 172.21.15.98:0/3301698938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c0094d0 con 0x7f2fac154030 2023-11-28T17:45:45.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.670+0000 7f2fb21a4640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac154030 0x7f2fac14c8d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400cf20 tx=0x7f2fa400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:45.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.670+0000 7f2fa27fc640 1 -- 172.21.15.98:0/3301698938 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa400d840 con 0x7f2fac154030 2023-11-28T17:45:45.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.670+0000 7f2fa27fc640 1 -- 172.21.15.98:0/3301698938 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa400d9a0 con 0x7f2fac154030 2023-11-28T17:45:45.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.670+0000 7f2fa27fc640 1 -- 172.21.15.98:0/3301698938 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa400dd50 con 0x7f2fac154030 2023-11-28T17:45:45.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.670+0000 7f2fb3c07640 1 -- 172.21.15.98:0/3301698938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fac157620 con 0x7f2fac154030 2023-11-28T17:45:45.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.670+0000 7f2fb3c07640 1 -- 172.21.15.98:0/3301698938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fac157af0 con 0x7f2fac154030 2023-11-28T17:45:45.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.671+0000 7f2fb3c07640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac0c4280 0x7f2fac0705e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.672+0000 7f2fb3c07640 1 -- 172.21.15.98:0/3301698938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0744d0 con 0x7f2fac0c4280 2023-11-28T17:45:45.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.672+0000 7f2fb19a3640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac0c4280 0x7f2fac0705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.672+0000 7f2fb19a3640 1 -- 172.21.15.98:0/3301698938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fac070ba0 con 0x7f2fac0c4280 2023-11-28T17:45:45.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.672+0000 7f2fb19a3640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac0c4280 0x7f2fac0705e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800e7a0 tx=0x7f2fa800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:45.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.673+0000 7f2fa27fc640 1 -- 172.21.15.98:0/3301698938 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8004030 con 0x7f2fac0c4280 2023-11-28T17:45:45.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.673+0000 7f2fa27fc640 1 -- 172.21.15.98:0/3301698938 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2fa401e3b0 con 0x7f2fac154030 2023-11-28T17:45:45.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.673+0000 7f2fa27fc640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2f9006b340 0x7f2f9006d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.674+0000 7f2fa27fc640 1 -- 172.21.15.98:0/3301698938 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2fa408a5e0 con 0x7f2fac154030 2023-11-28T17:45:45.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.674+0000 7f2fb11a2640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2f9006b340 0x7f2f9006d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.674+0000 7f2fa27fc640 1 -- 172.21.15.98:0/3301698938 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f2fa8004190 con 0x7f2fac0c4280 2023-11-28T17:45:45.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.674+0000 7f2fa27fc640 1 -- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac0c4280 msgr2=0x7f2fac0705e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.674+0000 7f2fa27fc640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac0c4280 0x7f2fac0705e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800e7a0 tx=0x7f2fa800ec60 comp rx=0 tx=0).stop 2023-11-28T17:45:45.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.674+0000 7f2fa27fc640 1 -- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac0c4280 msgr2=0x7f2fac0705e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:45.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.674+0000 7f2fa27fc640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac0c4280 0x7f2fac0705e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.674+0000 7f2fb11a2640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2f9006b340 0x7f2f9006d800 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c036000 tx=0x7f2f9c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:45.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.681+0000 7f2fb3c07640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac159bf0 0x7f2fac15a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.681+0000 7f2fb3c07640 1 -- 172.21.15.98:0/3301698938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac157ce0 con 0x7f2fac159bf0 2023-11-28T17:45:45.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.681+0000 7f2fb19a3640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac159bf0 0x7f2fac15a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.681+0000 7f2fb19a3640 1 -- 172.21.15.98:0/3301698938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2fac15dc10 con 0x7f2fac159bf0 2023-11-28T17:45:45.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.682+0000 7f2fb19a3640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac159bf0 0x7f2fac15a000 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800e7a0 tx=0x7f2fa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:45.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.682+0000 7f2fa27fc640 1 -- 172.21.15.98:0/3301698938 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa801f230 con 0x7f2fac159bf0 2023-11-28T17:45:45.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.682+0000 7f2fa27fc640 1 -- 172.21.15.98:0/3301698938 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4467 (secure 0 0 0) 0x7f2fa800a780 con 0x7f2fac159bf0 2023-11-28T17:45:45.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.682+0000 7f2fa27fc640 1 -- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac159bf0 msgr2=0x7f2fac15a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.682+0000 7f2fa27fc640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac159bf0 0x7f2fac15a000 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800e7a0 tx=0x7f2fa80046c0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.682+0000 7f2fa27fc640 1 -- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac159bf0 msgr2=0x7f2fac15a000 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:45.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.682+0000 7f2fa27fc640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac159bf0 0x7f2fac15a000 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800e7a0 tx=0x7f2fa80046c0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.684 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:45:45.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:45:45.684 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:45:45.685 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:45:45.685 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 48, 2023-11-28T17:45:45.685 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:45:45.685 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:45:45.685 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:45:45.685 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:45:45.685 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:45.685 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:45:45.685 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:45.685 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:45:45.686 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:45:45.686 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:45.686 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:45.686 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:45.686 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:45.686 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:45.686 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:45.686 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:45.686 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:45.686 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:45.686 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:45.687 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:45.687 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:45:45.687 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:45:45.687 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:45.687 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:45.687 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:45.687 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:45.687 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:45.687 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:45.687 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:45.688 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:45.688 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:45.688 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:45.688 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:45.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.688 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:45:45.688 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:45:45.688 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:45:45.688 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:45:45.688 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:45:45.688 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:45:45.689 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:45.689 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:45.689 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:45:45.689 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:45:45.689 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:45:45.689 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:45:45.689 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:45:45.689 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:45:45.689 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:45:45.689 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:45.690 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:45:45.690 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:45.690 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:45.690 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:45.690 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:45.690 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:45.690 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:45.690 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:45.690 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:45.690 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:45.691 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:45.691 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:45.691 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:45:45.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.691 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:45:45.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.691 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:45:45.691 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:45:45.691 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:45.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:45.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:45.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:45:45.692 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:45.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:45.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:45:45.692 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:45.692 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:45.692 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:45.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:45:45.693 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:45:45.693 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:45.693 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:45.693 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:45.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.693 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:45:45.693 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:45:45.694 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:45.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:45.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:45.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:45:45.694 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:45.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:45.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:45:45.695 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:45.695 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:45.695 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:45.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.695 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:45:45.695 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:45:45.695 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:45.695 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:45.695 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:45.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.696 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:45:45.696 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:45:45.696 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:45.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:45.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:45.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:45:45.696 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:45.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:45.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:45:45.697 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:45.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:45.697 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:45.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:45:45.697 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:45:45.697 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:45.697 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:45.698 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:45.698 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:45.698 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:45.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.698 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:45:45.698 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:45:45.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.698 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:45.698 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:45.698 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:45:45.698 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:45.699 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:45.699 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:45:45.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.699 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:45.699 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:45.699 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:45:45.699 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:45.699 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:45.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.699 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:45:45.699 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:45:45.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.683+0000 7f2fb3c07640 1 -- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2f9006b340 msgr2=0x7f2f9006d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.683+0000 7f2fb3c07640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2f9006b340 0x7f2f9006d800 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c036000 tx=0x7f2f9c008030 comp rx=0 tx=0).stop 2023-11-28T17:45:45.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.683+0000 7f2fb3c07640 1 -- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac154030 msgr2=0x7f2fac14c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.683+0000 7f2fb21a4640 1 -- 172.21.15.98:0/3301698938 reap_dead start 2023-11-28T17:45:45.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.683+0000 7f2fb3c07640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac154030 0x7f2fac14c8d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400cf20 tx=0x7f2fa400a710 comp rx=0 tx=0).stop 2023-11-28T17:45:45.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.683+0000 7f2fb3c07640 1 -- 172.21.15.98:0/3301698938 shutdown_connections 2023-11-28T17:45:45.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.683+0000 7f2fb3c07640 1 --2- 172.21.15.98:0/3301698938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fac154030 0x7f2fac14c8d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.683+0000 7f2fb3c07640 1 -- 172.21.15.98:0/3301698938 >> 172.21.15.98:0/3301698938 conn(0x7f2fac06c0f0 msgr2=0x7f2fac14fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:45.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.683+0000 7f2fb3c07640 1 -- 172.21.15.98:0/3301698938 shutdown_connections 2023-11-28T17:45:45.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.683+0000 7f2fb3c07640 1 -- 172.21.15.98:0/3301698938 wait complete. 2023-11-28T17:45:45.702 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:45:45.702 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:45:45.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:45:45.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.902+0000 7f325f50f640 1 Processor -- start 2023-11-28T17:45:45.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.902+0000 7f325f50f640 1 -- start start 2023-11-28T17:45:45.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.904+0000 7f325f50f640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3258071b50 0x7f325815e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.904+0000 7f325f50f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f32580c5a60 0x7f32580c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.904+0000 7f325f50f640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32580c6630 con 0x7f3258071770 2023-11-28T17:45:45.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.904+0000 7f325f50f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32580c7000 con 0x7f32580c5a60 2023-11-28T17:45:45.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.904+0000 7f325f50f640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325806e710 con 0x7f3258071b50 2023-11-28T17:45:45.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.904+0000 7f325daac640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f32580c5a60 0x7f32580c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.904+0000 7f325daac640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f32580c5a60 0x7f32580c6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56168/0 (socket says 172.21.15.98:56168) 2023-11-28T17:45:45.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.904+0000 7f325daac640 1 -- 172.21.15.98:0/1974077631 learned_addr learned my addr 172.21.15.98:0/1974077631 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:45.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.905+0000 7f325caaa640 1 --2- 172.21.15.98:0/1974077631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3258071b50 0x7f325815e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.905+0000 7f325daac640 1 -- 172.21.15.98:0/1974077631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3258071b50 msgr2=0x7f325815e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:45.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.905+0000 7f325daac640 1 --2- 172.21.15.98:0/1974077631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3258071b50 0x7f325815e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.905+0000 7f325daac640 1 -- 172.21.15.98:0/1974077631 >> v1:172.21.15.113:6789/0 conn(0x7f3258071770 legacy=0x7f3258157fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:45.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.905+0000 7f325daac640 1 -- 172.21.15.98:0/1974077631 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325806de70 con 0x7f32580c5a60 2023-11-28T17:45:45.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.906+0000 7f325daac640 1 --2- 172.21.15.98:0/1974077631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32580c5a60 0x7f32580c6ac0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3250009e50 tx=0x7f325002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79cef68f1e00792c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.906+0000 7f324ffff640 1 -- 172.21.15.98:0/1974077631 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325002c9e0 con 0x7f32580c5a60 2023-11-28T17:45:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.906+0000 7f324ffff640 1 -- 172.21.15.98:0/1974077631 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325002cb40 con 0x7f32580c5a60 2023-11-28T17:45:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.906+0000 7f324ffff640 1 -- 172.21.15.98:0/1974077631 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32500343f0 con 0x7f32580c5a60 2023-11-28T17:45:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.907+0000 7f325f50f640 1 -- 172.21.15.98:0/1974077631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32580c5a60 msgr2=0x7f32580c6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.907+0000 7f325f50f640 1 --2- 172.21.15.98:0/1974077631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32580c5a60 0x7f32580c6ac0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3250009e50 tx=0x7f325002be40 comp rx=0 tx=0).stop 2023-11-28T17:45:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.907+0000 7f325f50f640 1 -- 172.21.15.98:0/1974077631 shutdown_connections 2023-11-28T17:45:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.907+0000 7f325f50f640 1 --2- 172.21.15.98:0/1974077631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32580c5a60 0x7f32580c6ac0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.907+0000 7f325f50f640 1 --2- 172.21.15.98:0/1974077631 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3258071b50 0x7f325815e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.907+0000 7f325f50f640 1 -- 172.21.15.98:0/1974077631 >> 172.21.15.98:0/1974077631 conn(0x7f325806c0f0 msgr2=0x7f325806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.907+0000 7f325f50f640 1 -- 172.21.15.98:0/1974077631 shutdown_connections 2023-11-28T17:45:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.907+0000 7f325f50f640 1 -- 172.21.15.98:0/1974077631 wait complete. 2023-11-28T17:45:45.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.908+0000 7f325f50f640 1 Processor -- start 2023-11-28T17:45:45.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.909+0000 7f325f50f640 1 -- start start 2023-11-28T17:45:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.909+0000 7f325f50f640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3258071770 0x7f32581f7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.909+0000 7f325f50f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3258071b50 0x7f325815a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.909+0000 7f325f50f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32580c5a60 0x7f3258150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.909+0000 7f325f50f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32581f43f0 con 0x7f32580c5a60 2023-11-28T17:45:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.909+0000 7f325f50f640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32581f7f50 con 0x7f3258071b50 2023-11-28T17:45:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.909+0000 7f325f50f640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325815b140 con 0x7f3258071770 2023-11-28T17:45:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.910+0000 7f325daac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32580c5a60 0x7f3258150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.910+0000 7f325daac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32580c5a60 0x7f3258150650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56180/0 (socket says 172.21.15.98:56180) 2023-11-28T17:45:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.910+0000 7f325daac640 1 -- 172.21.15.98:0/210677538 learned_addr learned my addr 172.21.15.98:0/210677538 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.910+0000 7f325d2ab640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3258071770 0x7f32581f7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.910+0000 7f325caaa640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3258071b50 0x7f325815a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.910+0000 7f325daac640 1 -- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3258071770 msgr2=0x7f32581f7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.910+0000 7f325daac640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3258071770 0x7f32581f7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.910+0000 7f325daac640 1 -- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3258071b50 msgr2=0x7f325815a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.910+0000 7f325daac640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3258071b50 0x7f325815a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.910+0000 7f325daac640 1 -- 172.21.15.98:0/210677538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3250009b30 con 0x7f32580c5a60 2023-11-28T17:45:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.911+0000 7f325daac640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32580c5a60 0x7f3258150650 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3250036040 tx=0x7f3250004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:45.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.911+0000 7f324dffb640 1 -- 172.21.15.98:0/210677538 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250009510 con 0x7f32580c5a60 2023-11-28T17:45:45.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.911+0000 7f324dffb640 1 -- 172.21.15.98:0/210677538 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325003e450 con 0x7f32580c5a60 2023-11-28T17:45:45.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.911+0000 7f325f50f640 1 -- 172.21.15.98:0/210677538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325815b420 con 0x7f32580c5a60 2023-11-28T17:45:45.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.911+0000 7f324dffb640 1 -- 172.21.15.98:0/210677538 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325003e820 con 0x7f32580c5a60 2023-11-28T17:45:45.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.911+0000 7f325f50f640 1 -- 172.21.15.98:0/210677538 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32581f4080 con 0x7f32580c5a60 2023-11-28T17:45:45.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.912+0000 7f325f50f640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32580c9d90 0x7f32580ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.912+0000 7f325f50f640 1 -- 172.21.15.98:0/210677538 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32580744d0 con 0x7f32580c9d90 2023-11-28T17:45:45.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.912+0000 7f325d2ab640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32580c9d90 0x7f32580ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.913+0000 7f324dffb640 1 -- 172.21.15.98:0/210677538 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f325003e980 con 0x7f32580c5a60 2023-11-28T17:45:45.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.913+0000 7f325d2ab640 1 -- 172.21.15.98:0/210677538 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32580711a0 con 0x7f32580c9d90 2023-11-28T17:45:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.913+0000 7f325d2ab640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32580c9d90 0x7f32580ca170 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f325400d500 tx=0x7f325400d9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.914+0000 7f324dffb640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f323c06b2a0 0x7f323c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.914+0000 7f324dffb640 1 -- 172.21.15.98:0/210677538 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f32500acd80 con 0x7f32580c5a60 2023-11-28T17:45:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.914+0000 7f325caaa640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f323c06b2a0 0x7f323c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.914+0000 7f324dffb640 1 -- 172.21.15.98:0/210677538 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3254014620 con 0x7f32580c9d90 2023-11-28T17:45:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.915+0000 7f325caaa640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f323c06b2a0 0x7f323c06d760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3248006fd0 tx=0x7f3248008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.915+0000 7f324dffb640 1 -- 172.21.15.98:0/210677538 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f3254014780 con 0x7f32580c9d90 2023-11-28T17:45:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.915+0000 7f324dffb640 1 -- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32580c9d90 msgr2=0x7f32580ca170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.915+0000 7f324dffb640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32580c9d90 0x7f32580ca170 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f325400d500 tx=0x7f325400d9c0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.915+0000 7f324dffb640 1 -- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32580c9d90 msgr2=0x7f32580ca170 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.915+0000 7f324dffb640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32580c9d90 0x7f32580ca170 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f325400d500 tx=0x7f325400d9c0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.922+0000 7f325f50f640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3258159010 0x7f3258159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:45.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.922+0000 7f325f50f640 1 -- 172.21.15.98:0/210677538 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325806e430 con 0x7f3258159010 2023-11-28T17:45:45.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.922+0000 7f325d2ab640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3258159010 0x7f3258159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:45.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.923+0000 7f325d2ab640 1 -- 172.21.15.98:0/210677538 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f32580c8600 con 0x7f3258159010 2023-11-28T17:45:45.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.924+0000 7f325d2ab640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3258159010 0x7f3258159420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f325400d500 tx=0x7f32540046f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:45.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.924+0000 7f324dffb640 1 -- 172.21.15.98:0/210677538 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325401fa60 con 0x7f3258159010 2023-11-28T17:45:45.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.925+0000 7f324dffb640 1 -- 172.21.15.98:0/210677538 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4465 (secure 0 0 0) 0x7f325401fbc0 con 0x7f3258159010 2023-11-28T17:45:45.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.925+0000 7f324dffb640 1 -- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3258159010 msgr2=0x7f3258159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.925+0000 7f324dffb640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3258159010 0x7f3258159420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f325400d500 tx=0x7f32540046f0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.925+0000 7f324dffb640 1 -- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3258159010 msgr2=0x7f3258159420 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:45.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.925+0000 7f324dffb640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3258159010 0x7f3258159420 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f325400d500 tx=0x7f32540046f0 comp rx=0 tx=0).stop 2023-11-28T17:45:45.926 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:45:45.927 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:45:45.927 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:45:45.927 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:45:45.927 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 48, 2023-11-28T17:45:45.927 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:45:45.927 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:45:45.927 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:45:45.927 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:45:45.927 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:45.928 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:45:45.928 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:45.928 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:45:45.929 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:45:45.929 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:45.929 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:45.929 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:45.929 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:45.929 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:45.929 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:45.929 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:45.929 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:45.929 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:45.929 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:45.930 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:45.930 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:45:45.930 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:45:45.930 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:45.930 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:45.930 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:45.930 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:45.930 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:45.930 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:45.930 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:45.930 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:45.930 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:45.931 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:45.931 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:45.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.931 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:45:45.931 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:45:45.931 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:45:45.931 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:45:45.931 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:45:45.931 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:45:45.931 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:45.931 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:45.931 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:45:45.932 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:45:45.932 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:45:45.932 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:45:45.932 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:45:45.932 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:45:45.932 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:45:45.932 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:45.932 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:45:45.932 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:45.932 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:45.932 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:45.932 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:45.932 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:45.933 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:45.933 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:45.933 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:45.933 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:45.933 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:45.933 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:45.933 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:45:45.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.933 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:45:45.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.933 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:45:45.933 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:45:45.934 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:45.934 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:45.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:45.934 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:45:45.934 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:45.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:45.934 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:45:45.934 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:45.935 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:45.935 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:45.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.935 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:45:45.935 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:45:45.935 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:45.935 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:45.935 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:45.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.937 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:45:45.937 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:45:45.937 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:45.937 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:45.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:45.937 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:45:45.937 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:45.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:45.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:45:45.938 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:45.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:45.938 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:45.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:45:45.938 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:45:45.938 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:45.938 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:45.938 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:45.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.939 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:45:45.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:45:45.939 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:45.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:45.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:45.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:45:45.939 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:45.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:45.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:45:45.940 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:45.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:45.940 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:45.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:45:45.940 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:45:45.940 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:45.940 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:45.940 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:45.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:45.941 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:45.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.941 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:45:45.941 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:45:45.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.941 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:45.941 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:45.941 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:45:45.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:45.941 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:45.941 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:45:45.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:45.942 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:45.942 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:45.942 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:45:45.942 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:45.942 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:45.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:45.942 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:45:45.943 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:45:45.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.925+0000 7f325f50f640 1 -- 172.21.15.98:0/210677538 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f323c06b2a0 msgr2=0x7f323c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.925+0000 7f325f50f640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f323c06b2a0 0x7f323c06d760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3248006fd0 tx=0x7f3248008040 comp rx=0 tx=0).stop 2023-11-28T17:45:45.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.925+0000 7f325f50f640 1 -- 172.21.15.98:0/210677538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32580c5a60 msgr2=0x7f3258150650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:45.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.925+0000 7f325f50f640 1 --2- 172.21.15.98:0/210677538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32580c5a60 0x7f3258150650 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3250036040 tx=0x7f3250004a40 comp rx=0 tx=0).stop 2023-11-28T17:45:45.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.925+0000 7f325daac640 1 -- 172.21.15.98:0/210677538 reap_dead start 2023-11-28T17:45:45.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.925+0000 7f325daac640 1 -- 172.21.15.98:0/210677538 reap_dead start 2023-11-28T17:45:45.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.925+0000 7f325f50f640 1 -- 172.21.15.98:0/210677538 shutdown_connections 2023-11-28T17:45:45.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.925+0000 7f325f50f640 1 -- 172.21.15.98:0/210677538 >> 172.21.15.98:0/210677538 conn(0x7f325806c0f0 msgr2=0x7f325815dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:45.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.925+0000 7f325f50f640 1 -- 172.21.15.98:0/210677538 shutdown_connections 2023-11-28T17:45:45.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:45.925+0000 7f325f50f640 1 -- 172.21.15.98:0/210677538 wait complete. 2023-11-28T17:45:45.945 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:45:45.945 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:45:45.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:45:46.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.153+0000 7f40232ff640 1 Processor -- start 2023-11-28T17:45:46.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.153+0000 7f40232ff640 1 -- start start 2023-11-28T17:45:46.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.154+0000 7f40232ff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f401c06b160 0x7f401c0c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:46.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.154+0000 7f40232ff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401c0c2d70 con 0x7f401c0c24e0 2023-11-28T17:45:46.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.154+0000 7f40232ff640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401c0c6300 con 0x7f401c0c28c0 2023-11-28T17:45:46.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.154+0000 7f40232ff640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f401c0c7430 con 0x7f401c06b160 2023-11-28T17:45:46.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.155+0000 7f40222fd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f401c0c24e0 0x7f401c157ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43288/0 (socket says 172.21.15.98:43288) 2023-11-28T17:45:46.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.155+0000 7f40222fd640 1 -- 172.21.15.98:0/2863500458 learned_addr learned my addr 172.21.15.98:0/2863500458 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:45:46.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.155+0000 7f4021afc640 1 --2- 172.21.15.98:0/2863500458 >> v2:172.21.15.98:3300/0 conn(0x7f401c06b160 0x7f401c0c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:46.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.155+0000 7f4021afc640 1 -- 172.21.15.98:0/2863500458 >> v1:172.21.15.113:6789/0 conn(0x7f401c0c28c0 legacy=0x7f401c162390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:46.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.155+0000 7f4021afc640 1 -- 172.21.15.98:0/2863500458 >> v1:172.21.15.98:6789/0 conn(0x7f401c0c24e0 legacy=0x7f401c157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:46.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.155+0000 7f4021afc640 1 -- 172.21.15.98:0/2863500458 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c0c7710 con 0x7f401c06b160 2023-11-28T17:45:46.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.155+0000 7f40212fb640 1 -- 172.21.15.98:0/2863500458 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871704961 0 0) 0x7f401c0c2d70 con 0x7f401c0c24e0 2023-11-28T17:45:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.156+0000 7f4021afc640 1 --2- 172.21.15.98:0/2863500458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f401c06b160 0x7f401c0c6dc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f400c0097a0 tx=0x7f400c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dddd2badcf4c1e5d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.157+0000 7f40212fb640 1 -- 172.21.15.98:0/2863500458 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c02e070 con 0x7f401c06b160 2023-11-28T17:45:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.157+0000 7f40212fb640 1 -- 172.21.15.98:0/2863500458 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400c0315a0 con 0x7f401c06b160 2023-11-28T17:45:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.157+0000 7f40212fb640 1 -- 172.21.15.98:0/2863500458 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c031930 con 0x7f401c06b160 2023-11-28T17:45:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.157+0000 7f40232ff640 1 -- 172.21.15.98:0/2863500458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f401c06b160 msgr2=0x7f401c0c6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.157+0000 7f40232ff640 1 --2- 172.21.15.98:0/2863500458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f401c06b160 0x7f401c0c6dc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f400c0097a0 tx=0x7f400c02b840 comp rx=0 tx=0).stop 2023-11-28T17:45:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.157+0000 7f40232ff640 1 -- 172.21.15.98:0/2863500458 shutdown_connections 2023-11-28T17:45:46.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.157+0000 7f40232ff640 1 --2- 172.21.15.98:0/2863500458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f401c06b160 0x7f401c0c6dc0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.157+0000 7f40232ff640 1 -- 172.21.15.98:0/2863500458 >> 172.21.15.98:0/2863500458 conn(0x7f401c0cafe0 msgr2=0x7f401c0cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:46.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.157+0000 7f40232ff640 1 -- 172.21.15.98:0/2863500458 shutdown_connections 2023-11-28T17:45:46.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.157+0000 7f40232ff640 1 -- 172.21.15.98:0/2863500458 wait complete. 2023-11-28T17:45:46.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.159+0000 7f40232ff640 1 Processor -- start 2023-11-28T17:45:46.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.159+0000 7f40232ff640 1 -- start start 2023-11-28T17:45:46.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.160+0000 7f40232ff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f401c06b160 0x7f401c0aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:46.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.160+0000 7f40232ff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f401c0c24e0 0x7f401c14c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:46.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.160+0000 7f40232ff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c0c28c0 0x7f401c14c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:46.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.160+0000 7f40232ff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c1f0010 con 0x7f401c0c24e0 2023-11-28T17:45:46.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.160+0000 7f40232ff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c1f2b10 con 0x7f401c06b160 2023-11-28T17:45:46.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.160+0000 7f40232ff640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401c1ef420 con 0x7f401c0c28c0 2023-11-28T17:45:46.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.160+0000 7f4021afc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f401c0c24e0 0x7f401c14c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:46.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.160+0000 7f4021afc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f401c0c24e0 0x7f401c14c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56188/0 (socket says 172.21.15.98:56188) 2023-11-28T17:45:46.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.160+0000 7f4021afc640 1 -- 172.21.15.98:0/1586645938 learned_addr learned my addr 172.21.15.98:0/1586645938 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:46.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.161+0000 7f4022afe640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c0c28c0 0x7f401c14c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.161+0000 7f40222fd640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f401c06b160 0x7f401c0aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.161+0000 7f4021afc640 1 -- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c0c28c0 msgr2=0x7f401c14c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.161+0000 7f4021afc640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c0c28c0 0x7f401c14c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.161+0000 7f4021afc640 1 -- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f401c06b160 msgr2=0x7f401c0aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.161+0000 7f4021afc640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f401c06b160 0x7f401c0aecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.161+0000 7f4021afc640 1 -- 172.21.15.98:0/1586645938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c009400 con 0x7f401c0c24e0 2023-11-28T17:45:46.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.161+0000 7f4021afc640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f401c0c24e0 0x7f401c14c390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f400c034040 tx=0x7f400c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:46.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.161+0000 7f4012ffd640 1 -- 172.21.15.98:0/1586645938 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c032900 con 0x7f401c0c24e0 2023-11-28T17:45:46.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.161+0000 7f4012ffd640 1 -- 172.21.15.98:0/1586645938 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400c032a60 con 0x7f401c0c24e0 2023-11-28T17:45:46.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.161+0000 7f4012ffd640 1 -- 172.21.15.98:0/1586645938 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c032e10 con 0x7f401c0c24e0 2023-11-28T17:45:46.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.161+0000 7f40232ff640 1 -- 172.21.15.98:0/1586645938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401c1ef6e0 con 0x7f401c0c24e0 2023-11-28T17:45:46.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.161+0000 7f40232ff640 1 -- 172.21.15.98:0/1586645938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401c1efca0 con 0x7f401c0c24e0 2023-11-28T17:45:46.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.162+0000 7f40232ff640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15ba20 0x7f401c15ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:46.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.162+0000 7f40232ff640 1 -- 172.21.15.98:0/1586645938 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401c074310 con 0x7f401c15ba20 2023-11-28T17:45:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.163+0000 7f40222fd640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15ba20 0x7f401c15ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.164+0000 7f40222fd640 1 -- 172.21.15.98:0/1586645938 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f401c0c68a0 con 0x7f401c15ba20 2023-11-28T17:45:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.164+0000 7f4012ffd640 1 -- 172.21.15.98:0/1586645938 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f400c03b3f0 con 0x7f401c0c24e0 2023-11-28T17:45:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.164+0000 7f40222fd640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15ba20 0x7f401c15ad30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f401800d410 tx=0x7f401800d8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.165+0000 7f4012ffd640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f400006b2d0 0x7f400006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.165+0000 7f4012ffd640 1 -- 172.21.15.98:0/1586645938 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f400c0abcb0 con 0x7f401c0c24e0 2023-11-28T17:45:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.165+0000 7f4022afe640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f400006b2d0 0x7f400006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.165+0000 7f4012ffd640 1 -- 172.21.15.98:0/1586645938 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40180075a0 con 0x7f401c15ba20 2023-11-28T17:45:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.165+0000 7f4012ffd640 1 -- 172.21.15.98:0/1586645938 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f40180047e0 con 0x7f401c15ba20 2023-11-28T17:45:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.165+0000 7f4012ffd640 1 -- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15ba20 msgr2=0x7f401c15ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.165+0000 7f4012ffd640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15ba20 0x7f401c15ad30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f401800d410 tx=0x7f401800d8d0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.165+0000 7f4012ffd640 1 -- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15ba20 msgr2=0x7f401c15ad30 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:46.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.165+0000 7f4012ffd640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15ba20 0x7f401c15ad30 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f401800d410 tx=0x7f401800d8d0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.166+0000 7f4022afe640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f400006b2d0 0x7f400006d790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f401c15ac50 tx=0x7f4014009410 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.172+0000 7f40232ff640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15b270 0x7f401c1470e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.172+0000 7f40232ff640 1 -- 172.21.15.98:0/1586645938 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401c1ef870 con 0x7f401c15b270 2023-11-28T17:45:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.173+0000 7f40222fd640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15b270 0x7f401c1470e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.174+0000 7f40222fd640 1 -- 172.21.15.98:0/1586645938 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f401c15b680 con 0x7f401c15b270 2023-11-28T17:45:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.174+0000 7f40222fd640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15b270 0x7f401c1470e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4018015000 tx=0x7f4018001080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.174+0000 7f4012ffd640 1 -- 172.21.15.98:0/1586645938 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4018021070 con 0x7f401c15b270 2023-11-28T17:45:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.175+0000 7f4012ffd640 1 -- 172.21.15.98:0/1586645938 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4975 (secure 0 0 0) 0x7f401801b310 con 0x7f401c15b270 2023-11-28T17:45:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.175+0000 7f4012ffd640 1 -- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15b270 msgr2=0x7f401c1470e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.175+0000 7f4012ffd640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15b270 0x7f401c1470e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4018015000 tx=0x7f4018001080 comp rx=0 tx=0).stop 2023-11-28T17:45:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.175+0000 7f4012ffd640 1 -- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15b270 msgr2=0x7f401c1470e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.175+0000 7f4012ffd640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f401c15b270 0x7f401c1470e0 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4018015000 tx=0x7f4018001080 comp rx=0 tx=0).stop 2023-11-28T17:45:46.176 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:45:46.177 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:45:46.177 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:45:46.177 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:45:46.177 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 48, 2023-11-28T17:45:46.177 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:45:46.177 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:45:46.177 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:45:46.177 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:45:46.177 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:46.177 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2023-11-28T17:45:46.177 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:46.177 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:45:46.178 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:45:46.178 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:46.178 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:46.178 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:46.178 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:46.178 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:46.178 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:46.178 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:46.178 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:46.178 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:46.178 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:46.179 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:46.179 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:45:46.179 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:45:46.179 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:46.179 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:46.179 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:46.179 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:46.179 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:46.179 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:46.179 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:46.179 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:46.179 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:46.179 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:46.180 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:46.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:46.180 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:45:46.180 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:45:46.180 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:45:46.180 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:45:46.180 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:45:46.180 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:45:46.180 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:46.180 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:46.180 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:45:46.180 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:45:46.180 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:45:46.180 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:45:46.181 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:45:46.181 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:45:46.181 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:45:46.181 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:46.181 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:45:46.181 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:46.181 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:46.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:46.181 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:46.181 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:46.181 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:46.181 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:46.182 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:46.182 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:46.182 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:46.182 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:46.182 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:45:46.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:46.182 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:45:46.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:46.182 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:45:46.182 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:45:46.182 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:46.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:46.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:46.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:46.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:45:46.183 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:46.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:46.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:46.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:46.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:45:46.183 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:46.183 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:46.184 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:46.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:46.184 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:45:46.184 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:45:46.184 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:46.184 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:46.184 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:46.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:46.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:46.184 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:45:46.184 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:45:46.184 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:46.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:46.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:46.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:46.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:45:46.185 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:46.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:46.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:46.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:46.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:45:46.185 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:46.185 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:46.185 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:46.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:46.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:45:46.186 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:45:46.186 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:46.186 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:46.186 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:46.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:46.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:46.186 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:45:46.186 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:45:46.186 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:46.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:46.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:46.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:46.187 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:45:46.187 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:46.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:46.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:46.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:46.187 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:45:46.187 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:46.187 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:46.187 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:46.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:46.187 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:45:46.187 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:45:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:45:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:45:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:45:46.188 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:45:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:45:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:45:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:46.189 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:46.190 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 8 2023-11-28T17:45:46.190 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:46.190 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:46.190 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:45:46.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:46.190 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:46.190 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:46.190 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:45:46.190 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:46.190 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:46.190 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:45:46.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:46.191 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:46.191 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:46.191 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:45:46.191 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:46.191 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:46.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:46.191 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:45:46.191 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:45:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.176+0000 7f40232ff640 1 -- 172.21.15.98:0/1586645938 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f400006b2d0 msgr2=0x7f400006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.176+0000 7f40232ff640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f400006b2d0 0x7f400006d790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f401c15ac50 tx=0x7f4014009410 comp rx=0 tx=0).stop 2023-11-28T17:45:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.176+0000 7f40232ff640 1 -- 172.21.15.98:0/1586645938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f401c0c24e0 msgr2=0x7f401c14c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.176+0000 7f40232ff640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f401c0c24e0 0x7f401c14c390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f400c034040 tx=0x7f400c0046c0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.176+0000 7f4022afe640 1 -- 172.21.15.98:0/1586645938 reap_dead start 2023-11-28T17:45:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.176+0000 7f40232ff640 1 -- 172.21.15.98:0/1586645938 shutdown_connections 2023-11-28T17:45:46.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.176+0000 7f40232ff640 1 --2- 172.21.15.98:0/1586645938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f401c0c24e0 0x7f401c14c390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.176+0000 7f40232ff640 1 -- 172.21.15.98:0/1586645938 >> 172.21.15.98:0/1586645938 conn(0x7f401c0cafe0 msgr2=0x7f401c151330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.176+0000 7f40232ff640 1 -- 172.21.15.98:0/1586645938 shutdown_connections 2023-11-28T17:45:46.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.176+0000 7f40232ff640 1 -- 172.21.15.98:0/1586645938 wait complete. 2023-11-28T17:45:46.194 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:45:46.194 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-11-28T17:45:46.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-11-28T17:45:46.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.401+0000 7f19f5151640 1 Processor -- start 2023-11-28T17:45:46.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.401+0000 7f19f5151640 1 -- start start 2023-11-28T17:45:46.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.402+0000 7f19f5151640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f19f00c5f10 0x7f19f00c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:46.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.402+0000 7f19f5151640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19f00ca250 0x7f19f0161680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:46.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.402+0000 7f19f5151640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f00cab90 con 0x7f19f01582e0 2023-11-28T17:45:46.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.402+0000 7f19f5151640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f00c7470 con 0x7f19f00ca250 2023-11-28T17:45:46.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.402+0000 7f19f5151640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f00c9710 con 0x7f19f00c5f10 2023-11-28T17:45:46.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.403+0000 7f19ef59e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19f00ca250 0x7f19f0161680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:46.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.403+0000 7f19ef59e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19f00ca250 0x7f19f0161680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56194/0 (socket says 172.21.15.98:56194) 2023-11-28T17:45:46.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.403+0000 7f19ef59e640 1 -- 172.21.15.98:0/378929935 learned_addr learned my addr 172.21.15.98:0/378929935 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:46.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.403+0000 7f19e7fff640 1 --2- 172.21.15.98:0/378929935 >> v2:172.21.15.113:3300/0 conn(0x7f19f00c5f10 0x7f19f00c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:46.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.403+0000 7f19ef59e640 1 -- 172.21.15.98:0/378929935 >> v2:172.21.15.113:3300/0 conn(0x7f19f00c5f10 msgr2=0x7f19f00c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:46.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.403+0000 7f19ef59e640 1 --2- 172.21.15.98:0/378929935 >> v2:172.21.15.113:3300/0 conn(0x7f19f00c5f10 0x7f19f00c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.403+0000 7f19ef59e640 1 -- 172.21.15.98:0/378929935 >> v1:172.21.15.113:6789/0 conn(0x7f19f01582e0 legacy=0x7f19f00c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:46.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.403+0000 7f19ef59e640 1 -- 172.21.15.98:0/378929935 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f00c66b0 con 0x7f19f00ca250 2023-11-28T17:45:46.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.403+0000 7f19e7fff640 1 --2- 172.21.15.98:0/378929935 >> v2:172.21.15.113:3300/0 conn(0x7f19f00c5f10 0x7f19f00c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:45:46.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.404+0000 7f19ef59e640 1 --2- 172.21.15.98:0/378929935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f00ca250 0x7f19f0161680 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f19f006b4b0 tx=0x7f19e802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc4fea2553eeb449 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:46.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.404+0000 7f19ee59c640 1 -- 172.21.15.98:0/378929935 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e802ca80 con 0x7f19f00ca250 2023-11-28T17:45:46.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.404+0000 7f19ee59c640 1 -- 172.21.15.98:0/378929935 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e802cbe0 con 0x7f19f00ca250 2023-11-28T17:45:46.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.404+0000 7f19ee59c640 1 -- 172.21.15.98:0/378929935 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e80343f0 con 0x7f19f00ca250 2023-11-28T17:45:46.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.405+0000 7f19f5151640 1 -- 172.21.15.98:0/378929935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f00ca250 msgr2=0x7f19f0161680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:46.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.405+0000 7f19f5151640 1 --2- 172.21.15.98:0/378929935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f00ca250 0x7f19f0161680 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f19f006b4b0 tx=0x7f19e802bee0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.405+0000 7f19f5151640 1 -- 172.21.15.98:0/378929935 shutdown_connections 2023-11-28T17:45:46.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.405+0000 7f19f5151640 1 --2- 172.21.15.98:0/378929935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f00ca250 0x7f19f0161680 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.405+0000 7f19f5151640 1 --2- 172.21.15.98:0/378929935 >> v2:172.21.15.113:3300/0 conn(0x7f19f00c5f10 0x7f19f00c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.405+0000 7f19f5151640 1 -- 172.21.15.98:0/378929935 >> 172.21.15.98:0/378929935 conn(0x7f19f006a4d0 msgr2=0x7f19f006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:46.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.405+0000 7f19f5151640 1 -- 172.21.15.98:0/378929935 shutdown_connections 2023-11-28T17:45:46.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.405+0000 7f19f5151640 1 -- 172.21.15.98:0/378929935 wait complete. 2023-11-28T17:45:46.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.407+0000 7f19f5151640 1 Processor -- start 2023-11-28T17:45:46.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.407+0000 7f19f5151640 1 -- start start 2023-11-28T17:45:46.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.408+0000 7f19f5151640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19f00c5f10 0x7f19f0156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:46.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.408+0000 7f19f5151640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f00ca250 0x7f19f014e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:46.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.408+0000 7f19f5151640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19f01582e0 0x7f19f01f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:46.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.408+0000 7f19f5151640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f0159a60 con 0x7f19f00ca250 2023-11-28T17:45:46.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.408+0000 7f19f5151640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f0157080 con 0x7f19f01582e0 2023-11-28T17:45:46.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.408+0000 7f19f5151640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f015f220 con 0x7f19f00c5f10 2023-11-28T17:45:46.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.408+0000 7f19e7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f00ca250 0x7f19f014e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:46.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.408+0000 7f19e7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f00ca250 0x7f19f014e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56198/0 (socket says 172.21.15.98:56198) 2023-11-28T17:45:46.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.408+0000 7f19e7fff640 1 -- 172.21.15.98:0/1239502605 learned_addr learned my addr 172.21.15.98:0/1239502605 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:46.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.409+0000 7f19e7fff640 1 -- 172.21.15.98:0/1239502605 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19f00c5f10 msgr2=0x7f19f0156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:46.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.409+0000 7f19eed9d640 1 --2- 172.21.15.98:0/1239502605 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19f00c5f10 0x7f19f0156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:46.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.409+0000 7f19ef59e640 1 --2- 172.21.15.98:0/1239502605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19f01582e0 0x7f19f01f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:46.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.409+0000 7f19e7fff640 1 --2- 172.21.15.98:0/1239502605 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19f00c5f10 0x7f19f0156800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.409+0000 7f19e7fff640 1 -- 172.21.15.98:0/1239502605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19f01582e0 msgr2=0x7f19f01f4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.409+0000 7f19e7fff640 1 --2- 172.21.15.98:0/1239502605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19f01582e0 0x7f19f01f4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.409+0000 7f19e7fff640 1 -- 172.21.15.98:0/1239502605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e8009b30 con 0x7f19f00ca250 2023-11-28T17:45:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.409+0000 7f19e7fff640 1 --2- 172.21.15.98:0/1239502605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f00ca250 0x7f19f014e620 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f19d800c7f0 tx=0x7f19d800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.409+0000 7f19e77fe640 1 -- 172.21.15.98:0/1239502605 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8007820 con 0x7f19f00ca250 2023-11-28T17:45:46.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.409+0000 7f19e77fe640 1 -- 172.21.15.98:0/1239502605 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d8007980 con 0x7f19f00ca250 2023-11-28T17:45:46.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.409+0000 7f19e77fe640 1 -- 172.21.15.98:0/1239502605 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8007cf0 con 0x7f19f00ca250 2023-11-28T17:45:46.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.409+0000 7f19f5151640 1 -- 172.21.15.98:0/1239502605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f01572b0 con 0x7f19f00ca250 2023-11-28T17:45:46.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.410+0000 7f19f5151640 1 -- 172.21.15.98:0/1239502605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f0154a10 con 0x7f19f00ca250 2023-11-28T17:45:46.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.411+0000 7f19f5151640 1 -- 172.21.15.98:0/1239502605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f0067bf0 con 0x7f19f00ca250 2023-11-28T17:45:46.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.412+0000 7f19e77fe640 1 -- 172.21.15.98:0/1239502605 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f19d80113f0 con 0x7f19f00ca250 2023-11-28T17:45:46.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.413+0000 7f19e77fe640 1 --2- 172.21.15.98:0/1239502605 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19d006b2a0 0x7f19d006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:46.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.413+0000 7f19e77fe640 1 -- 172.21.15.98:0/1239502605 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f19d8089ed0 con 0x7f19f00ca250 2023-11-28T17:45:46.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.413+0000 7f19eed9d640 1 --2- 172.21.15.98:0/1239502605 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19d006b2a0 0x7f19d006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:46.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.414+0000 7f19eed9d640 1 --2- 172.21.15.98:0/1239502605 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19d006b2a0 0x7f19d006d760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f19e00015d0 tx=0x7f19e0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:46.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.417+0000 7f19e77fe640 1 -- 172.21.15.98:0/1239502605 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f19d8054d50 con 0x7f19f00ca250 2023-11-28T17:45:46.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.678+0000 7f19f5151640 1 -- 172.21.15.98:0/1239502605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f19f00c8a20 con 0x7f19f00ca250 2023-11-28T17:45:46.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.679+0000 7f19e77fe640 1 -- 172.21.15.98:0/1239502605 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f19d801f080 con 0x7f19f00ca250 2023-11-28T17:45:46.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.681+0000 7f19f5151640 1 -- 172.21.15.98:0/1239502605 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19d006b2a0 msgr2=0x7f19d006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:46.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.681+0000 7f19f5151640 1 --2- 172.21.15.98:0/1239502605 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19d006b2a0 0x7f19d006d760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f19e00015d0 tx=0x7f19e0009040 comp rx=0 tx=0).stop 2023-11-28T17:45:46.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.681+0000 7f19f5151640 1 -- 172.21.15.98:0/1239502605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f00ca250 msgr2=0x7f19f014e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:46.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.681+0000 7f19f5151640 1 --2- 172.21.15.98:0/1239502605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f00ca250 0x7f19f014e620 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f19d800c7f0 tx=0x7f19d800ccb0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.682+0000 7f19f5151640 1 -- 172.21.15.98:0/1239502605 shutdown_connections 2023-11-28T17:45:46.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.682+0000 7f19f5151640 1 --2- 172.21.15.98:0/1239502605 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19d006b2a0 0x7f19d006d760 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.682+0000 7f19f5151640 1 --2- 172.21.15.98:0/1239502605 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19f01582e0 0x7f19f01f4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.682+0000 7f19f5151640 1 --2- 172.21.15.98:0/1239502605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f00ca250 0x7f19f014e620 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.682+0000 7f19f5151640 1 --2- 172.21.15.98:0/1239502605 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f19f00c5f10 0x7f19f0156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:46.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.682+0000 7f19f5151640 1 -- 172.21.15.98:0/1239502605 >> 172.21.15.98:0/1239502605 conn(0x7f19f006a4d0 msgr2=0x7f19f015c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:46.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.682+0000 7f19f5151640 1 -- 172.21.15.98:0/1239502605 shutdown_connections 2023-11-28T17:45:46.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:46.682+0000 7f19f5151640 1 -- 172.21.15.98:0/1239502605 wait complete. 2023-11-28T17:45:46.701 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2023-11-28T17:45:52.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~61s 2023-11-28T17:45:56.702 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:45:56.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:45:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.928+0000 7f289332e640 1 Processor -- start 2023-11-28T17:45:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.929+0000 7f289332e640 1 -- start start 2023-11-28T17:45:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.930+0000 7f289332e640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f288c1518d0 0x7f288c158240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.930+0000 7f289332e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f288c0728a0 0x7f288c070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.930+0000 7f289332e640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f288c15bc80 con 0x7f288c06d950 2023-11-28T17:45:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.930+0000 7f289332e640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f288c06ade0 con 0x7f288c1518d0 2023-11-28T17:45:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.930+0000 7f289332e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f288c151cb0 con 0x7f288c0728a0 2023-11-28T17:45:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.931+0000 7f2891b2b640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f288c0728a0 0x7f288c070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.931+0000 7f2892b2d640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f288c06d950 0x7f288c06ddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.98:42386/0 (socket says 172.21.15.98:42386) 2023-11-28T17:45:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.931+0000 7f2892b2d640 1 -- 172.21.15.98:0/3223102890 learned_addr learned my addr 172.21.15.98:0/3223102890 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:45:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.931+0000 7f2891b2b640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f288c0728a0 0x7f288c070bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:43200/0 (socket says 172.21.15.98:43200) 2023-11-28T17:45:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.931+0000 7f289232c640 1 --2- 172.21.15.98:0/3223102890 >> v2:172.21.15.113:3300/0 conn(0x7f288c1518d0 0x7f288c158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.932+0000 7f2891b2b640 1 -- 172.21.15.98:0/3223102890 >> v2:172.21.15.113:3300/0 conn(0x7f288c1518d0 msgr2=0x7f288c158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.932+0000 7f2891b2b640 1 --2- 172.21.15.98:0/3223102890 >> v2:172.21.15.113:3300/0 conn(0x7f288c1518d0 0x7f288c158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.932+0000 7f2891b2b640 1 -- 172.21.15.98:0/3223102890 >> v1:172.21.15.113:6789/0 conn(0x7f288c06d950 legacy=0x7f288c06ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.932+0000 7f2891b2b640 1 -- 172.21.15.98:0/3223102890 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288c06b030 con 0x7f288c0728a0 2023-11-28T17:45:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.933+0000 7f2891b2b640 1 --2- 172.21.15.98:0/3223102890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f288c0728a0 0x7f288c070bc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f287c0097a0 tx=0x7f287c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79b05d656cc7b4e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.933+0000 7f289132a640 1 -- 172.21.15.98:0/3223102890 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c02e070 con 0x7f288c0728a0 2023-11-28T17:45:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.933+0000 7f289132a640 1 -- 172.21.15.98:0/3223102890 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287c0315a0 con 0x7f288c0728a0 2023-11-28T17:45:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.933+0000 7f289132a640 1 -- 172.21.15.98:0/3223102890 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c031930 con 0x7f288c0728a0 2023-11-28T17:45:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.934+0000 7f289332e640 1 -- 172.21.15.98:0/3223102890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f288c0728a0 msgr2=0x7f288c070bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.934+0000 7f289332e640 1 --2- 172.21.15.98:0/3223102890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f288c0728a0 0x7f288c070bc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f287c0097a0 tx=0x7f287c02b840 comp rx=0 tx=0).stop 2023-11-28T17:45:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.934+0000 7f289332e640 1 -- 172.21.15.98:0/3223102890 shutdown_connections 2023-11-28T17:45:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.934+0000 7f289332e640 1 --2- 172.21.15.98:0/3223102890 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f288c0728a0 0x7f288c070bc0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.934+0000 7f289332e640 1 --2- 172.21.15.98:0/3223102890 >> v2:172.21.15.113:3300/0 conn(0x7f288c1518d0 0x7f288c158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.934+0000 7f289332e640 1 -- 172.21.15.98:0/3223102890 >> 172.21.15.98:0/3223102890 conn(0x7f288c066230 msgr2=0x7f288c06a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.934+0000 7f289332e640 1 -- 172.21.15.98:0/3223102890 shutdown_connections 2023-11-28T17:45:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.934+0000 7f289332e640 1 -- 172.21.15.98:0/3223102890 wait complete. 2023-11-28T17:45:56.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.936+0000 7f289332e640 1 Processor -- start 2023-11-28T17:45:56.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.936+0000 7f289332e640 1 -- start start 2023-11-28T17:45:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.937+0000 7f289332e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f288c06d950 0x7f288c1f3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.937+0000 7f289332e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f288c0728a0 0x7f288c1efc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.937+0000 7f289332e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f288c1518d0 0x7f288c1e4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.937+0000 7f289332e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288c1e38e0 con 0x7f288c06d950 2023-11-28T17:45:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.937+0000 7f289332e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288c0c75b0 con 0x7f288c1518d0 2023-11-28T17:45:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.937+0000 7f289332e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f288c1f3640 con 0x7f288c0728a0 2023-11-28T17:45:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.937+0000 7f289232c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f288c06d950 0x7f288c1f3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:56.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.937+0000 7f289232c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f288c06d950 0x7f288c1f3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58156/0 (socket says 172.21.15.98:58156) 2023-11-28T17:45:56.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.937+0000 7f289232c640 1 -- 172.21.15.98:0/713207042 learned_addr learned my addr 172.21.15.98:0/713207042 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.938+0000 7f2891b2b640 1 --2- 172.21.15.98:0/713207042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f288c0728a0 0x7f288c1efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.938+0000 7f2892b2d640 1 --2- 172.21.15.98:0/713207042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f288c1518d0 0x7f288c1e4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.938+0000 7f289232c640 1 -- 172.21.15.98:0/713207042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f288c0728a0 msgr2=0x7f288c1efc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.938+0000 7f289232c640 1 --2- 172.21.15.98:0/713207042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f288c0728a0 0x7f288c1efc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.938+0000 7f289232c640 1 -- 172.21.15.98:0/713207042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f288c1518d0 msgr2=0x7f288c1e4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.938+0000 7f289232c640 1 --2- 172.21.15.98:0/713207042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f288c1518d0 0x7f288c1e4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.938+0000 7f289232c640 1 -- 172.21.15.98:0/713207042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287c009400 con 0x7f288c06d950 2023-11-28T17:45:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.938+0000 7f289232c640 1 --2- 172.21.15.98:0/713207042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f288c06d950 0x7f288c1f3100 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f288800e9b0 tx=0x7f288800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.938+0000 7f2882ffd640 1 -- 172.21.15.98:0/713207042 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2888009670 con 0x7f288c06d950 2023-11-28T17:45:56.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.938+0000 7f2882ffd640 1 -- 172.21.15.98:0/713207042 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f288800c9a0 con 0x7f288c06d950 2023-11-28T17:45:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.938+0000 7f2882ffd640 1 -- 172.21.15.98:0/713207042 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288800cd50 con 0x7f288c06d950 2023-11-28T17:45:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.938+0000 7f289332e640 1 -- 172.21.15.98:0/713207042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f288c1e4940 con 0x7f288c06d950 2023-11-28T17:45:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.939+0000 7f289332e640 1 -- 172.21.15.98:0/713207042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288c1ef1d0 con 0x7f288c06d950 2023-11-28T17:45:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.940+0000 7f289332e640 1 -- 172.21.15.98:0/713207042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288c0c7b20 con 0x7f288c06d950 2023-11-28T17:45:56.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.941+0000 7f2882ffd640 1 -- 172.21.15.98:0/713207042 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2888013470 con 0x7f288c06d950 2023-11-28T17:45:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.942+0000 7f2882ffd640 1 --2- 172.21.15.98:0/713207042 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f287006b2a0 0x7f287006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.942+0000 7f2882ffd640 1 -- 172.21.15.98:0/713207042 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2888089f40 con 0x7f288c06d950 2023-11-28T17:45:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.942+0000 7f2891b2b640 1 --2- 172.21.15.98:0/713207042 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f287006b2a0 0x7f287006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.942+0000 7f2891b2b640 1 --2- 172.21.15.98:0/713207042 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f287006b2a0 0x7f287006d760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f287c036000 tx=0x7f287c03d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:56.946+0000 7f2882ffd640 1 -- 172.21.15.98:0/713207042 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2888054c90 con 0x7f288c06d950 2023-11-28T17:45:57.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.196+0000 7f289332e640 1 -- 172.21.15.98:0/713207042 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f288c06eab0 con 0x7f288c06d950 2023-11-28T17:45:57.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.196+0000 7f2882ffd640 1 -- 172.21.15.98:0/713207042 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f2888021620 con 0x7f288c06d950 2023-11-28T17:45:57.197 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:45:57.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.198+0000 7f289332e640 1 -- 172.21.15.98:0/713207042 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f287006b2a0 msgr2=0x7f287006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.198+0000 7f289332e640 1 --2- 172.21.15.98:0/713207042 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f287006b2a0 0x7f287006d760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f287c036000 tx=0x7f287c03d040 comp rx=0 tx=0).stop 2023-11-28T17:45:57.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.198+0000 7f289332e640 1 -- 172.21.15.98:0/713207042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f288c06d950 msgr2=0x7f288c1f3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.198+0000 7f289332e640 1 --2- 172.21.15.98:0/713207042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f288c06d950 0x7f288c1f3100 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f288800e9b0 tx=0x7f288800ee70 comp rx=0 tx=0).stop 2023-11-28T17:45:57.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.199+0000 7f289332e640 1 -- 172.21.15.98:0/713207042 shutdown_connections 2023-11-28T17:45:57.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.199+0000 7f289332e640 1 --2- 172.21.15.98:0/713207042 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f287006b2a0 0x7f287006d760 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.199+0000 7f289332e640 1 --2- 172.21.15.98:0/713207042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f288c1518d0 0x7f288c1e4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.199+0000 7f289332e640 1 --2- 172.21.15.98:0/713207042 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f288c0728a0 0x7f288c1efc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.199+0000 7f289332e640 1 --2- 172.21.15.98:0/713207042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f288c06d950 0x7f288c1f3100 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.199+0000 7f289332e640 1 -- 172.21.15.98:0/713207042 >> 172.21.15.98:0/713207042 conn(0x7f288c066230 msgr2=0x7f288c159c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:57.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.199+0000 7f289332e640 1 -- 172.21.15.98:0/713207042 shutdown_connections 2023-11-28T17:45:57.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.199+0000 7f289332e640 1 -- 172.21.15.98:0/713207042 wait complete. 2023-11-28T17:45:57.218 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:45:57.218 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-11-28T17:45:57.218 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:45:57.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:45:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.422+0000 7ff36bf1d640 1 Processor -- start 2023-11-28T17:45:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.422+0000 7ff36bf1d640 1 -- start start 2023-11-28T17:45:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.423+0000 7ff36bf1d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff3641540d0 0x7ff36415dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.423+0000 7ff36bf1d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3640cb070 0x7ff36406b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.423+0000 7ff36bf1d640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36406a660 con 0x7ff3640ca280 2023-11-28T17:45:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.423+0000 7ff36bf1d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3640cbdd0 con 0x7ff3641540d0 2023-11-28T17:45:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.423+0000 7ff36bf1d640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3641544b0 con 0x7ff3640cb070 2023-11-28T17:45:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.424+0000 7ff369cb9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff3641540d0 0x7ff36415dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.424+0000 7ff369cb9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff3641540d0 0x7ff36415dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58172/0 (socket says 172.21.15.98:58172) 2023-11-28T17:45:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.424+0000 7ff369cb9640 1 -- 172.21.15.98:0/618773310 learned_addr learned my addr 172.21.15.98:0/618773310 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.424+0000 7ff369cb9640 1 -- 172.21.15.98:0/618773310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3640cb070 msgr2=0x7ff36406b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.424+0000 7ff369cb9640 1 --2- 172.21.15.98:0/618773310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3640cb070 0x7ff36406b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.424+0000 7ff369cb9640 1 -- 172.21.15.98:0/618773310 >> v1:172.21.15.113:6789/0 conn(0x7ff3640ca280 legacy=0x7ff364159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.424+0000 7ff369cb9640 1 -- 172.21.15.98:0/618773310 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36415e1b0 con 0x7ff3641540d0 2023-11-28T17:45:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.425+0000 7ff369cb9640 1 --2- 172.21.15.98:0/618773310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3641540d0 0x7ff36415dc70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff360009870 tx=0x7ff36002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf83b5de625b652 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.425+0000 7ff368cb7640 1 -- 172.21.15.98:0/618773310 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360031640 con 0x7ff3641540d0 2023-11-28T17:45:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.425+0000 7ff368cb7640 1 -- 172.21.15.98:0/618773310 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3600317a0 con 0x7ff3641540d0 2023-11-28T17:45:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.425+0000 7ff368cb7640 1 -- 172.21.15.98:0/618773310 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360031b30 con 0x7ff3641540d0 2023-11-28T17:45:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.426+0000 7ff36bf1d640 1 -- 172.21.15.98:0/618773310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3641540d0 msgr2=0x7ff36415dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.426+0000 7ff36bf1d640 1 --2- 172.21.15.98:0/618773310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3641540d0 0x7ff36415dc70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff360009870 tx=0x7ff36002b8f0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.426+0000 7ff36bf1d640 1 -- 172.21.15.98:0/618773310 shutdown_connections 2023-11-28T17:45:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.426+0000 7ff36bf1d640 1 --2- 172.21.15.98:0/618773310 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3640cb070 0x7ff36406b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.426+0000 7ff36bf1d640 1 --2- 172.21.15.98:0/618773310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3641540d0 0x7ff36415dc70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.426+0000 7ff36bf1d640 1 -- 172.21.15.98:0/618773310 >> 172.21.15.98:0/618773310 conn(0x7ff3640cf490 msgr2=0x7ff3640cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.426+0000 7ff36bf1d640 1 -- 172.21.15.98:0/618773310 shutdown_connections 2023-11-28T17:45:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.426+0000 7ff36bf1d640 1 -- 172.21.15.98:0/618773310 wait complete. 2023-11-28T17:45:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.428+0000 7ff36bf1d640 1 Processor -- start 2023-11-28T17:45:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.429+0000 7ff36bf1d640 1 -- start start 2023-11-28T17:45:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.430+0000 7ff36bf1d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3640ca280 0x7ff364156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.430+0000 7ff36bf1d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3640cb070 0x7ff3641f10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.430+0000 7ff36bf1d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3641540d0 0x7ff3640c3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.430+0000 7ff36bf1d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff364156580 con 0x7ff3640cb070 2023-11-28T17:45:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.430+0000 7ff36bf1d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36414f830 con 0x7ff3640ca280 2023-11-28T17:45:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.430+0000 7ff36bf1d640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3641f1620 con 0x7ff3641540d0 2023-11-28T17:45:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.430+0000 7ff3694b8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3640cb070 0x7ff3641f10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.430+0000 7ff3694b8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3640cb070 0x7ff3641f10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58188/0 (socket says 172.21.15.98:58188) 2023-11-28T17:45:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.430+0000 7ff3694b8640 1 -- 172.21.15.98:0/3152053478 learned_addr learned my addr 172.21.15.98:0/3152053478 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.430+0000 7ff369cb9640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3640ca280 0x7ff364156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.430+0000 7ff36a4ba640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3641540d0 0x7ff3640c3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.431+0000 7ff3694b8640 1 -- 172.21.15.98:0/3152053478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3641540d0 msgr2=0x7ff3640c3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.431+0000 7ff3694b8640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3641540d0 0x7ff3640c3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.431+0000 7ff3694b8640 1 -- 172.21.15.98:0/3152053478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3640ca280 msgr2=0x7ff364156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.431+0000 7ff3694b8640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3640ca280 0x7ff364156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.431+0000 7ff3694b8640 1 -- 172.21.15.98:0/3152053478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3600094d0 con 0x7ff3640cb070 2023-11-28T17:45:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.431+0000 7ff369cb9640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3640ca280 0x7ff364156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:45:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.431+0000 7ff36a4ba640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff3641540d0 0x7ff3640c3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:45:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.431+0000 7ff3694b8640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3640cb070 0x7ff3641f10e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff35400b4a0 tx=0x7ff35400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.431+0000 7ff35a7fc640 1 -- 172.21.15.98:0/3152053478 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354010610 con 0x7ff3640cb070 2023-11-28T17:45:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.431+0000 7ff35a7fc640 1 -- 172.21.15.98:0/3152053478 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff354010770 con 0x7ff3640cb070 2023-11-28T17:45:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.431+0000 7ff36bf1d640 1 -- 172.21.15.98:0/3152053478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff364154d90 con 0x7ff3640cb070 2023-11-28T17:45:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.431+0000 7ff35a7fc640 1 -- 172.21.15.98:0/3152053478 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354010b20 con 0x7ff3640cb070 2023-11-28T17:45:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.431+0000 7ff36bf1d640 1 -- 172.21.15.98:0/3152053478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3640c4d30 con 0x7ff3640cb070 2023-11-28T17:45:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.432+0000 7ff36bf1d640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff364154600 0x7ff3640c5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.432+0000 7ff36bf1d640 1 -- 172.21.15.98:0/3152053478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff364074110 con 0x7ff364154600 2023-11-28T17:45:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.433+0000 7ff369cb9640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff364154600 0x7ff3640c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.433+0000 7ff369cb9640 1 -- 172.21.15.98:0/3152053478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3640c9120 con 0x7ff364154600 2023-11-28T17:45:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.433+0000 7ff369cb9640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff364154600 0x7ff3640c5530 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff3640b5070 tx=0x7ff360004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:57.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.434+0000 7ff35a7fc640 1 -- 172.21.15.98:0/3152053478 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36003aa00 con 0x7ff364154600 2023-11-28T17:45:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.434+0000 7ff35a7fc640 1 -- 172.21.15.98:0/3152053478 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7ff36003ab60 con 0x7ff364154600 2023-11-28T17:45:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.434+0000 7ff35a7fc640 1 -- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff364154600 msgr2=0x7ff3640c5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.434+0000 7ff35a7fc640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff364154600 0x7ff3640c5530 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff3640b5070 tx=0x7ff360004730 comp rx=0 tx=0).stop 2023-11-28T17:45:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.434+0000 7ff35a7fc640 1 -- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff364154600 msgr2=0x7ff3640c5530 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.434+0000 7ff35a7fc640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff364154600 0x7ff3640c5530 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff3640b5070 tx=0x7ff360004730 comp rx=0 tx=0).stop 2023-11-28T17:45:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.434+0000 7ff35a7fc640 1 -- 172.21.15.98:0/3152053478 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff354010c80 con 0x7ff3640cb070 2023-11-28T17:45:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.435+0000 7ff35a7fc640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff34806b050 0x7ff34806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.435+0000 7ff35a7fc640 1 -- 172.21.15.98:0/3152053478 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff3540897a0 con 0x7ff3640cb070 2023-11-28T17:45:57.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.435+0000 7ff369cb9640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff34806b050 0x7ff34806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.436+0000 7ff369cb9640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff34806b050 0x7ff34806d510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff36406c860 tx=0x7ff360041000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.442+0000 7ff36bf1d640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3640c4ec0 0x7ff3640c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.442+0000 7ff36bf1d640 1 -- 172.21.15.98:0/3152053478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3640ce490 con 0x7ff3640c4ec0 2023-11-28T17:45:57.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.442+0000 7ff36a4ba640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3640c4ec0 0x7ff3640c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.442+0000 7ff36a4ba640 1 -- 172.21.15.98:0/3152053478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff36406ae70 con 0x7ff3640c4ec0 2023-11-28T17:45:57.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.443+0000 7ff36a4ba640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3640c4ec0 0x7ff3640c5a70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00d760 tx=0x7ff35c00dc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:57.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.443+0000 7ff35a7fc640 1 -- 172.21.15.98:0/3152053478 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c016070 con 0x7ff3640c4ec0 2023-11-28T17:45:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.443+0000 7ff35a7fc640 1 -- 172.21.15.98:0/3152053478 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4468 (secure 0 0 0) 0x7ff35c00de70 con 0x7ff3640c4ec0 2023-11-28T17:45:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.443+0000 7ff35a7fc640 1 -- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3640c4ec0 msgr2=0x7ff3640c5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.443+0000 7ff35a7fc640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3640c4ec0 0x7ff3640c5a70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00d760 tx=0x7ff35c00dc20 comp rx=0 tx=0).stop 2023-11-28T17:45:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.443+0000 7ff35a7fc640 1 -- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3640c4ec0 msgr2=0x7ff3640c5a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.443+0000 7ff35a7fc640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3640c4ec0 0x7ff3640c5a70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.445 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:45:57.445 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:45:57.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:45:57.445 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:45:57.445 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 48, 2023-11-28T17:45:57.445 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:45:57.445 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:45:57.445 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:45:57.445 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:45:57.445 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.445 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 12, 2023-11-28T17:45:57.446 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:57.446 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:45:57.446 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:45:57.446 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:57.446 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:57.446 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:57.446 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:57.446 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:57.446 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:57.446 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:57.447 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:57.447 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:57.447 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:57.447 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.447 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:45:57.447 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:45:57.447 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:57.447 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:57.447 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:57.447 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:57.447 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:57.447 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:57.448 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:57.448 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:57.448 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:57.448 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:57.448 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.448 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:45:57.449 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:45:57.449 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:45:57.449 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:45:57.449 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:45:57.450 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:45:57.450 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:57.450 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:57.450 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:45:57.450 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:45:57.450 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:45:57.450 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:45:57.450 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:45:57.450 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:45:57.450 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:45:57.450 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:57.451 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:45:57.451 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:57.451 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:57.451 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:57.451 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:57.451 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:57.451 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:57.451 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:57.451 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:57.451 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:57.451 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:57.451 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.451 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:45:57.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.452 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:45:57.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:45:57.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:45:57.452 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:57.452 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:57.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:57.452 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:45:57.452 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:57.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:45:57.453 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.453 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:45:57.453 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:45:57.453 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:57.453 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:57.454 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:57.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:45:57.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:45:57.454 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:57.454 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:57.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:57.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:45:57.455 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:57.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:45:57.455 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.455 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:45:57.456 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:45:57.456 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:57.456 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:57.456 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:57.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:45:57.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:45:57.456 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:57.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:57.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:57.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:45:57.457 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:57.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:45:57.457 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.458 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:45:57.458 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:45:57.458 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:57.458 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:57.458 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:57.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.459 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.459 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:45:57.459 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:45:57.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.459 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:57.459 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:57.459 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:45:57.459 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.459 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.460 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:45:57.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.460 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:57.460 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:57.460 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:45:57.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.460 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.460 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:45:57.460 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:45:57.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.444+0000 7ff36bf1d640 1 -- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff34806b050 msgr2=0x7ff34806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.444+0000 7ff36bf1d640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff34806b050 0x7ff34806d510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff36406c860 tx=0x7ff360041000 comp rx=0 tx=0).stop 2023-11-28T17:45:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.444+0000 7ff36bf1d640 1 -- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3640cb070 msgr2=0x7ff3641f10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.444+0000 7ff36a4ba640 1 -- 172.21.15.98:0/3152053478 reap_dead start 2023-11-28T17:45:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.444+0000 7ff36bf1d640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3640cb070 0x7ff3641f10e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff35400b4a0 tx=0x7ff35400b960 comp rx=0 tx=0).stop 2023-11-28T17:45:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.444+0000 7ff36bf1d640 1 -- 172.21.15.98:0/3152053478 shutdown_connections 2023-11-28T17:45:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.444+0000 7ff36bf1d640 1 --2- 172.21.15.98:0/3152053478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3640cb070 0x7ff3641f10e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.444+0000 7ff36bf1d640 1 -- 172.21.15.98:0/3152053478 >> 172.21.15.98:0/3152053478 conn(0x7ff3640cf490 msgr2=0x7ff3640ced00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.444+0000 7ff36bf1d640 1 -- 172.21.15.98:0/3152053478 shutdown_connections 2023-11-28T17:45:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.444+0000 7ff36bf1d640 1 -- 172.21.15.98:0/3152053478 wait complete. 2023-11-28T17:45:57.463 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:45:57.463 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:45:57.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:45:57.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.647+0000 7ff9a14ea640 1 Processor -- start 2023-11-28T17:45:57.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.647+0000 7ff9a14ea640 1 -- start start 2023-11-28T17:45:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.648+0000 7ff9a14ea640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff99c150df0 0x7ff99c06a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.648+0000 7ff9a14ea640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99c157480 con 0x7ff99c06b210 2023-11-28T17:45:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.648+0000 7ff9a14ea640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99c070a40 con 0x7ff99c070ce0 2023-11-28T17:45:57.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.648+0000 7ff9a14ea640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff99c1511d0 con 0x7ff99c150df0 2023-11-28T17:45:57.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.648+0000 7ff99bfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff99c150df0 0x7ff99c06a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.649+0000 7ff99bfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff99c150df0 0x7ff99c06a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58212/0 (socket says 172.21.15.98:58212) 2023-11-28T17:45:57.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.649+0000 7ff99bfff640 1 -- 172.21.15.98:0/1925336467 learned_addr learned my addr 172.21.15.98:0/1925336467 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:57.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.649+0000 7ff99bfff640 1 -- 172.21.15.98:0/1925336467 >> v1:172.21.15.113:6789/0 conn(0x7ff99c070ce0 legacy=0x7ff99c071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:57.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.649+0000 7ff99bfff640 1 -- 172.21.15.98:0/1925336467 >> v1:172.21.15.98:6789/0 conn(0x7ff99c06b210 legacy=0x7ff99c161380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.649+0000 7ff99bfff640 1 -- 172.21.15.98:0/1925336467 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99c1579b0 con 0x7ff99c150df0 2023-11-28T17:45:57.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.650+0000 7ff99bfff640 1 --2- 172.21.15.98:0/1925336467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c150df0 0x7ff99c06a4d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009870 tx=0x7ff98c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15d5e9aac6a00b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:57.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.650+0000 7ff99affd640 1 -- 172.21.15.98:0/1925336467 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c030860 con 0x7ff99c150df0 2023-11-28T17:45:57.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.650+0000 7ff99affd640 1 -- 172.21.15.98:0/1925336467 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c0309c0 con 0x7ff99c150df0 2023-11-28T17:45:57.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.650+0000 7ff99affd640 1 -- 172.21.15.98:0/1925336467 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c030d50 con 0x7ff99c150df0 2023-11-28T17:45:57.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.651+0000 7ff9a14ea640 1 -- 172.21.15.98:0/1925336467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c150df0 msgr2=0x7ff99c06a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.651+0000 7ff9a14ea640 1 --2- 172.21.15.98:0/1925336467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c150df0 0x7ff99c06a4d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009870 tx=0x7ff98c02b9a0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.651+0000 7ff9a14ea640 1 -- 172.21.15.98:0/1925336467 shutdown_connections 2023-11-28T17:45:57.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.651+0000 7ff9a14ea640 1 --2- 172.21.15.98:0/1925336467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c150df0 0x7ff99c06a4d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.651+0000 7ff9a14ea640 1 -- 172.21.15.98:0/1925336467 >> 172.21.15.98:0/1925336467 conn(0x7ff99c0cafe0 msgr2=0x7ff99c0cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:57.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.651+0000 7ff9a14ea640 1 -- 172.21.15.98:0/1925336467 shutdown_connections 2023-11-28T17:45:57.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.651+0000 7ff9a14ea640 1 -- 172.21.15.98:0/1925336467 wait complete. 2023-11-28T17:45:57.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.653+0000 7ff9a14ea640 1 Processor -- start 2023-11-28T17:45:57.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.653+0000 7ff9a14ea640 1 -- start start 2023-11-28T17:45:57.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff9a14ea640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff99c06b210 0x7ff99c157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff9a14ea640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c070ce0 0x7ff99c14db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff9a14ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c150df0 0x7ff99c149860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff9a14ea640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c14bc40 con 0x7ff99c150df0 2023-11-28T17:45:57.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff9a14ea640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c151540 con 0x7ff99c070ce0 2023-11-28T17:45:57.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff9a14ea640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c14b090 con 0x7ff99c06b210 2023-11-28T17:45:57.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff9a0ce9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c150df0 0x7ff99c149860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff9a0ce9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c150df0 0x7ff99c149860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58214/0 (socket says 172.21.15.98:58214) 2023-11-28T17:45:57.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff9a0ce9640 1 -- 172.21.15.98:0/4182761332 learned_addr learned my addr 172.21.15.98:0/4182761332 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:57.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff99bfff640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff99c06b210 0x7ff99c157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff9a0ce9640 1 -- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff99c06b210 msgr2=0x7ff99c157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff99b7fe640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c070ce0 0x7ff99c14db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff9a0ce9640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff99c06b210 0x7ff99c157bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff9a0ce9640 1 -- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c070ce0 msgr2=0x7ff99c14db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff9a0ce9640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c070ce0 0x7ff99c14db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.655+0000 7ff9a0ce9640 1 -- 172.21.15.98:0/4182761332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c0094d0 con 0x7ff99c150df0 2023-11-28T17:45:57.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.656+0000 7ff9a0ce9640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c150df0 0x7ff99c149860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff99400c680 tx=0x7ff99400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:57.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.656+0000 7ff998ff9640 1 -- 172.21.15.98:0/4182761332 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994003b70 con 0x7ff99c150df0 2023-11-28T17:45:57.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.656+0000 7ff998ff9640 1 -- 172.21.15.98:0/4182761332 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994003cd0 con 0x7ff99c150df0 2023-11-28T17:45:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.656+0000 7ff998ff9640 1 -- 172.21.15.98:0/4182761332 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994005310 con 0x7ff99c150df0 2023-11-28T17:45:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.656+0000 7ff9a14ea640 1 -- 172.21.15.98:0/4182761332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff99c15ae70 con 0x7ff99c150df0 2023-11-28T17:45:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.656+0000 7ff9a14ea640 1 -- 172.21.15.98:0/4182761332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99c14b2a0 con 0x7ff99c150df0 2023-11-28T17:45:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.657+0000 7ff9a14ea640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c14cbc0 0x7ff99c149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.657+0000 7ff9a14ea640 1 -- 172.21.15.98:0/4182761332 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c074310 con 0x7ff99c14cbc0 2023-11-28T17:45:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.658+0000 7ff99bfff640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c14cbc0 0x7ff99c149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.658+0000 7ff998ff9640 1 -- 172.21.15.98:0/4182761332 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff994005470 con 0x7ff99c150df0 2023-11-28T17:45:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.659+0000 7ff99bfff640 1 -- 172.21.15.98:0/4182761332 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff99c0c7430 con 0x7ff99c14cbc0 2023-11-28T17:45:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.659+0000 7ff99bfff640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c14cbc0 0x7ff99c149da0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff99c1545f0 tx=0x7ff98c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.659+0000 7ff998ff9640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff98806b2a0 0x7ff98806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.659+0000 7ff998ff9640 1 -- 172.21.15.98:0/4182761332 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff994089a80 con 0x7ff99c150df0 2023-11-28T17:45:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.659+0000 7ff99b7fe640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff98806b2a0 0x7ff98806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.659+0000 7ff998ff9640 1 -- 172.21.15.98:0/4182761332 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c048070 con 0x7ff99c14cbc0 2023-11-28T17:45:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.660+0000 7ff99b7fe640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff98806b2a0 0x7ff98806d760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff99c0c9020 tx=0x7ff990009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.660+0000 7ff998ff9640 1 -- 172.21.15.98:0/4182761332 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7ff98c044470 con 0x7ff99c14cbc0 2023-11-28T17:45:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.660+0000 7ff998ff9640 1 -- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c14cbc0 msgr2=0x7ff99c149da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.660+0000 7ff998ff9640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c14cbc0 0x7ff99c149da0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff99c1545f0 tx=0x7ff98c004730 comp rx=0 tx=0).stop 2023-11-28T17:45:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.660+0000 7ff998ff9640 1 -- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c14cbc0 msgr2=0x7ff99c149da0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.660+0000 7ff998ff9640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c14cbc0 0x7ff99c149da0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff99c1545f0 tx=0x7ff98c004730 comp rx=0 tx=0).stop 2023-11-28T17:45:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.667+0000 7ff9a14ea640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c14b4c0 0x7ff99c14bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.667+0000 7ff9a14ea640 1 -- 172.21.15.98:0/4182761332 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c1544e0 con 0x7ff99c14b4c0 2023-11-28T17:45:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.668+0000 7ff99bfff640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c14b4c0 0x7ff99c14bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.668+0000 7ff99bfff640 1 -- 172.21.15.98:0/4182761332 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff99c0715f0 con 0x7ff99c14b4c0 2023-11-28T17:45:57.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.669+0000 7ff99bfff640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c14b4c0 0x7ff99c14bf60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009870 tx=0x7ff98c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:57.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.669+0000 7ff998ff9640 1 -- 172.21.15.98:0/4182761332 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c04c930 con 0x7ff99c14b4c0 2023-11-28T17:45:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.670+0000 7ff998ff9640 1 -- 172.21.15.98:0/4182761332 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4636 (secure 0 0 0) 0x7ff98c04ca90 con 0x7ff99c14b4c0 2023-11-28T17:45:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.670+0000 7ff998ff9640 1 -- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c14b4c0 msgr2=0x7ff99c14bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.670+0000 7ff998ff9640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c14b4c0 0x7ff99c14bf60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009870 tx=0x7ff98c0046c0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.670+0000 7ff998ff9640 1 -- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c14b4c0 msgr2=0x7ff99c14bf60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.670+0000 7ff998ff9640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff99c14b4c0 0x7ff99c14bf60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.672 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:45:57.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:45:57.672 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:45:57.672 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:45:57.672 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 48, 2023-11-28T17:45:57.672 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:45:57.672 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:45:57.673 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:45:57.673 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:45:57.673 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.673 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 12, 2023-11-28T17:45:57.673 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:57.673 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:45:57.673 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:45:57.673 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:57.673 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:57.673 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:57.673 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:57.673 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:57.674 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:57.674 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:57.674 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:57.674 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:57.674 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:57.674 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.674 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:45:57.674 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:45:57.674 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:57.674 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:57.674 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:57.674 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:57.675 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:57.675 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:57.675 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:57.675 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:57.675 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:57.675 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:57.675 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.675 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:45:57.675 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:45:57.675 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:45:57.675 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:45:57.675 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:45:57.676 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:45:57.676 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:57.676 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:57.676 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:45:57.676 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:45:57.676 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:45:57.676 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:45:57.676 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:45:57.676 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:45:57.676 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:45:57.676 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:57.676 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:45:57.676 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:57.677 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:57.677 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:57.677 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:57.677 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:57.677 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:57.677 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:57.677 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:57.677 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:57.677 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:57.677 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.677 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:45:57.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.678 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:45:57.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.678 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:45:57.678 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:45:57.678 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:57.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:57.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:57.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:45:57.678 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:57.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:45:57.679 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.679 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.679 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:45:57.679 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:45:57.679 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:57.679 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:57.679 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:57.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.680 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:45:57.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:45:57.680 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:57.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:57.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:57.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:45:57.680 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:57.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:45:57.680 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.681 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:45:57.681 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:45:57.681 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:57.681 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:57.681 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:57.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.681 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:45:57.681 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:45:57.682 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:57.682 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:57.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:57.682 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:45:57.682 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:57.682 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:45:57.682 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.682 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:45:57.683 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:45:57.683 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:57.683 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:57.683 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:57.683 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.683 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.683 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:45:57.683 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:45:57.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.683 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:57.684 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:57.684 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:45:57.684 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.684 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.684 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:45:57.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.684 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:57.684 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:57.684 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:45:57.684 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.684 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.684 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:45:57.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.685 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:57.685 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:57.685 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:45:57.685 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.685 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.685 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:45:57.685 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:45:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.671+0000 7ff9a14ea640 1 -- 172.21.15.98:0/4182761332 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff98806b2a0 msgr2=0x7ff98806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.671+0000 7ff9a14ea640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff98806b2a0 0x7ff98806d760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff99c0c9020 tx=0x7ff990009040 comp rx=0 tx=0).stop 2023-11-28T17:45:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.671+0000 7ff9a14ea640 1 -- 172.21.15.98:0/4182761332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c150df0 msgr2=0x7ff99c149860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.671+0000 7ff9a14ea640 1 --2- 172.21.15.98:0/4182761332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff99c150df0 0x7ff99c149860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff99400c680 tx=0x7ff99400cb40 comp rx=0 tx=0).stop 2023-11-28T17:45:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.671+0000 7ff9a0ce9640 1 -- 172.21.15.98:0/4182761332 reap_dead start 2023-11-28T17:45:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.671+0000 7ff9a0ce9640 1 -- 172.21.15.98:0/4182761332 reap_dead start 2023-11-28T17:45:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.671+0000 7ff9a14ea640 1 -- 172.21.15.98:0/4182761332 shutdown_connections 2023-11-28T17:45:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.671+0000 7ff9a14ea640 1 -- 172.21.15.98:0/4182761332 >> 172.21.15.98:0/4182761332 conn(0x7ff99c0cafe0 msgr2=0x7ff99c156880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.671+0000 7ff9a14ea640 1 -- 172.21.15.98:0/4182761332 shutdown_connections 2023-11-28T17:45:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.671+0000 7ff9a14ea640 1 -- 172.21.15.98:0/4182761332 wait complete. 2023-11-28T17:45:57.687 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:45:57.688 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:45:57.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:45:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.886+0000 7f6ffa251640 1 Processor -- start 2023-11-28T17:45:57.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.887+0000 7f6ffa251640 1 -- start start 2023-11-28T17:45:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.888+0000 7f6ffa251640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6ff4151be0 0x7f6ff40c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.888+0000 7f6ffa251640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff40c8150 con 0x7f6ff40c5a60 2023-11-28T17:45:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.888+0000 7f6ffa251640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff40c8850 con 0x7f6ff41584d0 2023-11-28T17:45:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.888+0000 7f6ffa251640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff40c5f50 con 0x7f6ff4151be0 2023-11-28T17:45:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.888+0000 7f6ff2ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6ff40c5a60 0x7f6ff40c4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57308/0 (socket says 172.21.15.98:57308) 2023-11-28T17:45:57.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.888+0000 7f6ff2ffd640 1 -- 172.21.15.98:0/580144688 learned_addr learned my addr 172.21.15.98:0/580144688 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:45:57.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.889+0000 7f6ff37fe640 1 --2- 172.21.15.98:0/580144688 >> v2:172.21.15.113:3300/0 conn(0x7f6ff4151be0 0x7f6ff40c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.889+0000 7f6ff27fc640 1 -- 172.21.15.98:0/580144688 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285372037 0 0) 0x7f6ff40c8150 con 0x7f6ff40c5a60 2023-11-28T17:45:57.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.889+0000 7f6ff27fc640 1 -- 172.21.15.98:0/580144688 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe0003520 con 0x7f6ff40c5a60 2023-11-28T17:45:57.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.889+0000 7f6ff27fc640 1 -- 172.21.15.98:0/580144688 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 121094104 0 0) 0x7f6fe0003520 con 0x7f6ff40c5a60 2023-11-28T17:45:57.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.889+0000 7f6ff27fc640 1 -- 172.21.15.98:0/580144688 >> v2:172.21.15.113:3300/0 conn(0x7f6ff4151be0 msgr2=0x7f6ff40c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:57.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.889+0000 7f6ff27fc640 1 --2- 172.21.15.98:0/580144688 >> v2:172.21.15.113:3300/0 conn(0x7f6ff4151be0 0x7f6ff40c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.889+0000 7f6ff27fc640 1 -- 172.21.15.98:0/580144688 >> v1:172.21.15.113:6789/0 conn(0x7f6ff41584d0 legacy=0x7f6ff406dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.889+0000 7f6ff27fc640 1 -- 172.21.15.98:0/580144688 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff40c7b90 con 0x7f6ff40c5a60 2023-11-28T17:45:57.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.889+0000 7f6ff27fc640 1 -- 172.21.15.98:0/580144688 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f6fdc001670 con 0x7f6ff40c5a60 2023-11-28T17:45:57.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.889+0000 7f6ff27fc640 1 -- 172.21.15.98:0/580144688 >> v1:172.21.15.98:6789/0 conn(0x7f6ff40c5a60 legacy=0x7f6ff40c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.889+0000 7f6ff27fc640 1 --2- 172.21.15.98:0/580144688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe00039c0 0x7f6fe0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.890+0000 7f6ff27fc640 1 --2- 172.21.15.98:0/580144688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe00047c0 0x7f6fe0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.890+0000 7f6ff27fc640 1 --2- 172.21.15.98:0/580144688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe0005180 0x7f6fe000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.890+0000 7f6ff27fc640 1 -- 172.21.15.98:0/580144688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40c5f50 con 0x7f6fe00039c0 2023-11-28T17:45:57.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.890+0000 7f6ff27fc640 1 -- 172.21.15.98:0/580144688 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe000c080 con 0x7f6fe0005180 2023-11-28T17:45:57.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.890+0000 7f6ff27fc640 1 -- 172.21.15.98:0/580144688 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe000c240 con 0x7f6fe00047c0 2023-11-28T17:45:57.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.890+0000 7f6ff37fe640 1 --2- 172.21.15.98:0/580144688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe00039c0 0x7f6fe0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.890+0000 7f6ff2ffd640 1 --2- 172.21.15.98:0/580144688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe00047c0 0x7f6fe0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.890+0000 7f6ff3fff640 1 --2- 172.21.15.98:0/580144688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe0005180 0x7f6fe000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.890+0000 7f6ff37fe640 1 -- 172.21.15.98:0/580144688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe00047c0 msgr2=0x7f6fe0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:57.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.890+0000 7f6ff37fe640 1 --2- 172.21.15.98:0/580144688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe00047c0 0x7f6fe0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.890+0000 7f6ff37fe640 1 -- 172.21.15.98:0/580144688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe0005180 msgr2=0x7f6fe000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:45:57.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.890+0000 7f6ff37fe640 1 --2- 172.21.15.98:0/580144688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe0005180 0x7f6fe000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.890+0000 7f6ff37fe640 1 -- 172.21.15.98:0/580144688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe0003520 con 0x7f6fe00039c0 2023-11-28T17:45:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.890+0000 7f6ff3fff640 1 --2- 172.21.15.98:0/580144688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe0005180 0x7f6fe000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:45:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.890+0000 7f6ff3fff640 1 -- 172.21.15.98:0/580144688 reap_dead start 2023-11-28T17:45:57.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.891+0000 7f6ff37fe640 1 --2- 172.21.15.98:0/580144688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe00039c0 0x7f6fe0003de0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4009fc0 tx=0x7f6fe00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e7fe466c2d91bb6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:57.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.892+0000 7f6ff27fc640 1 -- 172.21.15.98:0/580144688 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe402cc20 con 0x7f6fe00039c0 2023-11-28T17:45:57.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.892+0000 7f6ff27fc640 1 -- 172.21.15.98:0/580144688 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe400f040 con 0x7f6fe00039c0 2023-11-28T17:45:57.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.892+0000 7f6ff27fc640 1 -- 172.21.15.98:0/580144688 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4034440 con 0x7f6fe00039c0 2023-11-28T17:45:57.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.892+0000 7f6ffa251640 1 -- 172.21.15.98:0/580144688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe00039c0 msgr2=0x7f6fe0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.892+0000 7f6ffa251640 1 --2- 172.21.15.98:0/580144688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe00039c0 0x7f6fe0003de0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4009fc0 tx=0x7f6fe00057f0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.892+0000 7f6ffa251640 1 -- 172.21.15.98:0/580144688 shutdown_connections 2023-11-28T17:45:57.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.892+0000 7f6ffa251640 1 --2- 172.21.15.98:0/580144688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe00039c0 0x7f6fe0003de0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.892+0000 7f6ffa251640 1 -- 172.21.15.98:0/580144688 >> 172.21.15.98:0/580144688 conn(0x7f6ff406c0f0 msgr2=0x7f6ff406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:57.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.892+0000 7f6ffa251640 1 -- 172.21.15.98:0/580144688 shutdown_connections 2023-11-28T17:45:57.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.892+0000 7f6ffa251640 1 -- 172.21.15.98:0/580144688 wait complete. 2023-11-28T17:45:57.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.894+0000 7f6ffa251640 1 Processor -- start 2023-11-28T17:45:57.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.894+0000 7f6ffa251640 1 -- start start 2023-11-28T17:45:57.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.895+0000 7f6ffa251640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe00039c0 0x7f6ff414d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.895+0000 7f6ffa251640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff4156c60 0x7f6ff41547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.895+0000 7f6ffa251640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ff406e060 0x7f6ff414c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.895+0000 7f6ffa251640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff414efb0 con 0x7f6ff4156c60 2023-11-28T17:45:57.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.895+0000 7f6ffa251640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff41ef910 con 0x7f6ff406e060 2023-11-28T17:45:57.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.895+0000 7f6ffa251640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff415ba20 con 0x7f6fe00039c0 2023-11-28T17:45:57.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.895+0000 7f6ff2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff4156c60 0x7f6ff41547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.895+0000 7f6ff2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff4156c60 0x7f6ff41547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58218/0 (socket says 172.21.15.98:58218) 2023-11-28T17:45:57.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.895+0000 7f6ff2ffd640 1 -- 172.21.15.98:0/2400780239 learned_addr learned my addr 172.21.15.98:0/2400780239 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:57.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.896+0000 7f6ff37fe640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe00039c0 0x7f6ff414d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.896+0000 7f6ff2ffd640 1 -- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe00039c0 msgr2=0x7f6ff414d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.896+0000 7f6ff3fff640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ff406e060 0x7f6ff414c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.896+0000 7f6ff2ffd640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe00039c0 0x7f6ff414d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.896+0000 7f6ff2ffd640 1 -- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ff406e060 msgr2=0x7f6ff414c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.896+0000 7f6ff2ffd640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ff406e060 0x7f6ff414c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:57.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.896+0000 7f6ff2ffd640 1 -- 172.21.15.98:0/2400780239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff415bd00 con 0x7f6ff4156c60 2023-11-28T17:45:57.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.896+0000 7f6ff2ffd640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff4156c60 0x7f6ff41547d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc00d7a0 tx=0x7f6fdc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:57.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.896+0000 7f6febfff640 1 -- 172.21.15.98:0/2400780239 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0074e0 con 0x7f6ff4156c60 2023-11-28T17:45:57.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.896+0000 7f6febfff640 1 -- 172.21.15.98:0/2400780239 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fdc007640 con 0x7f6ff4156c60 2023-11-28T17:45:57.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.896+0000 7f6febfff640 1 -- 172.21.15.98:0/2400780239 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0049e0 con 0x7f6ff4156c60 2023-11-28T17:45:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.896+0000 7f6ffa251640 1 -- 172.21.15.98:0/2400780239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff41efbd0 con 0x7f6ff4156c60 2023-11-28T17:45:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.896+0000 7f6ffa251640 1 -- 172.21.15.98:0/2400780239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff41f0190 con 0x7f6ff4156c60 2023-11-28T17:45:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.897+0000 7f6ffa251640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ff415b450 0x7f6ff414cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.897+0000 7f6ffa251640 1 -- 172.21.15.98:0/2400780239 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff40744d0 con 0x7f6ff415b450 2023-11-28T17:45:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.898+0000 7f6ff37fe640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ff415b450 0x7f6ff414cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.899+0000 7f6febfff640 1 -- 172.21.15.98:0/2400780239 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6fdc004b40 con 0x7f6ff4156c60 2023-11-28T17:45:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.899+0000 7f6ff37fe640 1 -- 172.21.15.98:0/2400780239 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ff40c6240 con 0x7f6ff415b450 2023-11-28T17:45:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.899+0000 7f6ff37fe640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ff415b450 0x7f6ff414cd40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4009f90 tx=0x7f6fe4009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.899+0000 7f6febfff640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6fd406b2a0 0x7f6fd406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.900+0000 7f6febfff640 1 -- 172.21.15.98:0/2400780239 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6fdc089530 con 0x7f6ff4156c60 2023-11-28T17:45:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.900+0000 7f6ff3fff640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6fd406b2a0 0x7f6fd406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.900+0000 7f6febfff640 1 -- 172.21.15.98:0/2400780239 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4034df0 con 0x7f6ff415b450 2023-11-28T17:45:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.900+0000 7f6febfff640 1 -- 172.21.15.98:0/2400780239 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f6fe4003990 con 0x7f6ff415b450 2023-11-28T17:45:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.900+0000 7f6febfff640 1 -- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ff415b450 msgr2=0x7f6ff414cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.900+0000 7f6febfff640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ff415b450 0x7f6ff414cd40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4009f90 tx=0x7f6fe4009750 comp rx=0 tx=0).stop 2023-11-28T17:45:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.900+0000 7f6febfff640 1 -- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ff415b450 msgr2=0x7f6ff414cd40 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.900+0000 7f6febfff640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ff415b450 0x7f6ff414cd40 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4009f90 tx=0x7f6fe4009750 comp rx=0 tx=0).stop 2023-11-28T17:45:57.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.900+0000 7f6ff3fff640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6fd406b2a0 0x7f6fd406d760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40c5240 tx=0x7f6fec009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:57.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.907+0000 7f6ffa251640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ff41f2e80 0x7f6ff41f3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:57.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.907+0000 7f6ffa251640 1 -- 172.21.15.98:0/2400780239 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff415b830 con 0x7f6ff41f2e80 2023-11-28T17:45:57.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.908+0000 7f6ff37fe640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ff41f2e80 0x7f6ff41f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:57.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.908+0000 7f6ff37fe640 1 -- 172.21.15.98:0/2400780239 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6ff4158920 con 0x7f6ff41f2e80 2023-11-28T17:45:57.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.909+0000 7f6ff37fe640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ff41f2e80 0x7f6ff41f3290 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6fe402c5c0 tx=0x7f6fe4030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:57.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.909+0000 7f6febfff640 1 -- 172.21.15.98:0/2400780239 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4004850 con 0x7f6ff41f2e80 2023-11-28T17:45:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.910+0000 7f6febfff640 1 -- 172.21.15.98:0/2400780239 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4976 (secure 0 0 0) 0x7f6fe404c070 con 0x7f6ff41f2e80 2023-11-28T17:45:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.910+0000 7f6febfff640 1 -- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ff41f2e80 msgr2=0x7f6ff41f3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.910+0000 7f6febfff640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ff41f2e80 0x7f6ff41f3290 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6fe402c5c0 tx=0x7f6fe4030000 comp rx=0 tx=0).stop 2023-11-28T17:45:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.910+0000 7f6febfff640 1 -- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ff41f2e80 msgr2=0x7f6ff41f3290 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:45:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.910+0000 7f6febfff640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6ff41f2e80 0x7f6ff41f3290 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6fe402c5c0 tx=0x7f6fe4030000 comp rx=0 tx=0).stop 2023-11-28T17:45:57.911 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:45:57.911 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:45:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:45:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:45:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 48, 2023-11-28T17:45:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:45:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:45:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:45:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:45:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 12, 2023-11-28T17:45:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:57.912 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:45:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:45:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:57.913 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.914 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:45:57.914 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:45:57.914 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:57.914 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:57.914 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:57.914 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:57.914 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:57.914 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:57.914 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:57.914 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:57.915 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:57.915 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:57.915 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.915 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:45:57.915 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:45:57.915 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:45:57.915 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:45:57.915 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:45:57.915 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:45:57.915 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:57.915 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:45:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:45:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:45:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:45:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:45:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:45:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:45:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:45:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:45:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:45:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:45:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:45:57.916 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:45:57.917 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:45:57.917 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:45:57.917 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:45:57.917 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:45:57.917 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:45:57.917 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:45:57.917 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:45:57.917 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.917 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:45:57.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.917 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:45:57.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.918 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:45:57.918 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:45:57.918 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:57.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:57.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:57.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:45:57.918 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:57.919 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:45:57.919 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.919 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.919 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.919 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:45:57.919 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:45:57.919 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:57.919 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:57.919 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:57.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.920 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:45:57.920 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:45:57.920 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:57.920 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:57.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.920 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:57.920 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:45:57.920 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:57.921 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:45:57.921 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.921 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.921 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.921 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:45:57.921 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:45:57.922 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:57.922 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:57.922 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:57.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.922 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:45:57.922 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:45:57.922 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:45:57.922 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:45:57.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.922 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:45:57.922 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:45:57.923 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.923 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:45:57.923 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:45:57.923 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:45:57.923 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.923 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.923 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:45:57.924 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:45:57.924 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:45:57.924 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:45:57.924 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:45:57.924 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.924 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.924 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:45:57.924 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:45:57.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.925 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:57.925 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:57.925 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:45:57.925 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.925 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.925 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:45:57.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.925 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:57.925 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:57.925 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:45:57.926 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.926 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.926 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:45:57.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.926 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:57.926 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:57.926 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 8 2023-11-28T17:45:57.926 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.926 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.926 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:45:57.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.926 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:57.927 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:57.927 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:45:57.927 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.927 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:45:57.927 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:45:57.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:45:57.927 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:45:57.927 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:45:57.927 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:45:57.927 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:45:57.927 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:45:57.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:45:57.928 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:45:57.928 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:45:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.910+0000 7f6ffa251640 1 -- 172.21.15.98:0/2400780239 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6fd406b2a0 msgr2=0x7f6fd406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.910+0000 7f6ffa251640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6fd406b2a0 0x7f6fd406d760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40c5240 tx=0x7f6fec009040 comp rx=0 tx=0).stop 2023-11-28T17:45:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.910+0000 7f6ffa251640 1 -- 172.21.15.98:0/2400780239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff4156c60 msgr2=0x7f6ff41547d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.910+0000 7f6ffa251640 1 --2- 172.21.15.98:0/2400780239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ff4156c60 0x7f6ff41547d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc00d7a0 tx=0x7f6fdc00dc60 comp rx=0 tx=0).stop 2023-11-28T17:45:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.910+0000 7f6ff3fff640 1 -- 172.21.15.98:0/2400780239 reap_dead start 2023-11-28T17:45:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.911+0000 7f6ff3fff640 1 -- 172.21.15.98:0/2400780239 reap_dead start 2023-11-28T17:45:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.911+0000 7f6ffa251640 1 -- 172.21.15.98:0/2400780239 shutdown_connections 2023-11-28T17:45:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.911+0000 7f6ffa251640 1 -- 172.21.15.98:0/2400780239 >> 172.21.15.98:0/2400780239 conn(0x7f6ff406c0f0 msgr2=0x7f6ff41558f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.911+0000 7f6ffa251640 1 -- 172.21.15.98:0/2400780239 shutdown_connections 2023-11-28T17:45:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:57.911+0000 7f6ffa251640 1 -- 172.21.15.98:0/2400780239 wait complete. 2023-11-28T17:45:57.930 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:45:57.931 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2023-11-28T17:45:57.931 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-11-28T17:45:57.931 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2023-11-28T17:45:57.931 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-11-28T17:45:57.931 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-28T17:45:57.931 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:45:57.969 INFO:tasks.ceph.mon.b:Stopped 2023-11-28T17:45:57.969 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-11-28T17:45:57.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:45:58.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.183+0000 7f29cc248640 1 Processor -- start 2023-11-28T17:45:58.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.184+0000 7f29cc248640 1 -- start start 2023-11-28T17:45:58.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.185+0000 7f29cc248640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f29c40c5c00 0x7f29c40c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:58.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.185+0000 7f29cc248640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29c40c4a20 0x7f29c40c6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:58.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.185+0000 7f29cc248640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c4157850 con 0x7f29c4150c00 2023-11-28T17:45:58.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.185+0000 7f29cc248640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c40c2e10 con 0x7f29c40c5c00 2023-11-28T17:45:58.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.185+0000 7f29cc248640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c4156980 con 0x7f29c40c4a20 2023-11-28T17:45:58.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.185+0000 7f29c97e3640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f29c40c5c00 msgr2=0x7f29c40c44e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:58.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.185+0000 7f29c9fe4640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f29c4150c00 legacy=0x7f29c40c2700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:45:58.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.185+0000 7f29c97e3640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f29c40c5c00 0x7f29c40c44e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:58.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.186+0000 7f29ca7e5640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29c40c4a20 0x7f29c40c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:58.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.186+0000 7f29ca7e5640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29c40c4a20 0x7f29c40c6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:43282/0 (socket says 172.21.15.98:43282) 2023-11-28T17:45:58.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.186+0000 7f29ca7e5640 1 -- 172.21.15.98:0/566591051 learned_addr learned my addr 172.21.15.98:0/566591051 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:58.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.186+0000 7f29ca7e5640 1 -- 172.21.15.98:0/566591051 >> v2:172.21.15.113:3300/0 conn(0x7f29c40c5c00 msgr2=0x7f29c40c44e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:58.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.186+0000 7f29ca7e5640 1 --2- 172.21.15.98:0/566591051 >> v2:172.21.15.113:3300/0 conn(0x7f29c40c5c00 0x7f29c40c44e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:58.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.186+0000 7f29ca7e5640 1 -- 172.21.15.98:0/566591051 >> v1:172.21.15.113:6789/0 conn(0x7f29c4150c00 legacy=0x7f29c40c2700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:45:58.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.186+0000 7f29ca7e5640 1 -- 172.21.15.98:0/566591051 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c4156c40 con 0x7f29c40c4a20 2023-11-28T17:45:58.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.188+0000 7f29ca7e5640 1 --2- 172.21.15.98:0/566591051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29c40c4a20 0x7f29c40c6600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009e50 tx=0x7f29bc02bf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38f330a46830fac4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:58.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.189+0000 7f29c8fe2640 1 -- 172.21.15.98:0/566591051 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc02caa0 con 0x7f29c40c4a20 2023-11-28T17:45:58.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.189+0000 7f29c8fe2640 1 -- 172.21.15.98:0/566591051 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29bc02cc00 con 0x7f29c40c4a20 2023-11-28T17:45:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.189+0000 7f29c8fe2640 1 -- 172.21.15.98:0/566591051 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc0343f0 con 0x7f29c40c4a20 2023-11-28T17:45:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.190+0000 7f29cc248640 1 -- 172.21.15.98:0/566591051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29c40c4a20 msgr2=0x7f29c40c6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.190+0000 7f29cc248640 1 --2- 172.21.15.98:0/566591051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29c40c4a20 0x7f29c40c6600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009e50 tx=0x7f29bc02bf00 comp rx=0 tx=0).stop 2023-11-28T17:45:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.190+0000 7f29cc248640 1 -- 172.21.15.98:0/566591051 shutdown_connections 2023-11-28T17:45:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.190+0000 7f29cc248640 1 --2- 172.21.15.98:0/566591051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29c40c4a20 0x7f29c40c6600 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.190+0000 7f29cc248640 1 --2- 172.21.15.98:0/566591051 >> v2:172.21.15.113:3300/0 conn(0x7f29c40c5c00 0x7f29c40c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.190+0000 7f29cc248640 1 -- 172.21.15.98:0/566591051 >> 172.21.15.98:0/566591051 conn(0x7f29c406a4d0 msgr2=0x7f29c406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.190+0000 7f29cc248640 1 -- 172.21.15.98:0/566591051 shutdown_connections 2023-11-28T17:45:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.190+0000 7f29cc248640 1 -- 172.21.15.98:0/566591051 wait complete. 2023-11-28T17:45:58.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.192+0000 7f29cc248640 1 Processor -- start 2023-11-28T17:45:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.192+0000 7f29cc248640 1 -- start start 2023-11-28T17:45:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29cc248640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29c40c4a20 0x7f29c41efef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29cc248640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29c40c5c00 0x7f29c4152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29cc248640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29c4150c00 0x7f29c4155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:58.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29cc248640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c41f0430 con 0x7f29c4150c00 2023-11-28T17:45:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29cc248640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c4072210 con 0x7f29c40c5c00 2023-11-28T17:45:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29cc248640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c4071150 con 0x7f29c40c4a20 2023-11-28T17:45:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29c97e3640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29c40c5c00 msgr2=0x7f29c4152860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:45:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29c97e3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29c40c5c00 0x7f29c4152860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:45:58.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29ca7e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29c4150c00 0x7f29c4155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29ca7e5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29c4150c00 0x7f29c4155ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58230/0 (socket says 172.21.15.98:58230) 2023-11-28T17:45:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29ca7e5640 1 -- 172.21.15.98:0/809304001 learned_addr learned my addr 172.21.15.98:0/809304001 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:45:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29c9fe4640 1 --2- 172.21.15.98:0/809304001 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29c40c4a20 0x7f29c41efef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29ca7e5640 1 -- 172.21.15.98:0/809304001 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29c40c4a20 msgr2=0x7f29c41efef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:58.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29ca7e5640 1 --2- 172.21.15.98:0/809304001 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29c40c4a20 0x7f29c41efef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29ca7e5640 1 -- 172.21.15.98:0/809304001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29c40c5c00 msgr2=0x7f29c4152860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:45:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29ca7e5640 1 --2- 172.21.15.98:0/809304001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29c40c5c00 0x7f29c4152860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.194+0000 7f29ca7e5640 1 -- 172.21.15.98:0/809304001 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29bc009b30 con 0x7f29c4150c00 2023-11-28T17:45:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.195+0000 7f29ca7e5640 1 --2- 172.21.15.98:0/809304001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29c4150c00 0x7f29c4155ce0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009e50 tx=0x7f29bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.195+0000 7f29b67fc640 1 -- 172.21.15.98:0/809304001 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc009510 con 0x7f29c4150c00 2023-11-28T17:45:58.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.195+0000 7f29b67fc640 1 -- 172.21.15.98:0/809304001 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29bc03e450 con 0x7f29c4150c00 2023-11-28T17:45:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.195+0000 7f29cc248640 1 -- 172.21.15.98:0/809304001 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c4152da0 con 0x7f29c4150c00 2023-11-28T17:45:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.195+0000 7f29b67fc640 1 -- 172.21.15.98:0/809304001 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc03e7e0 con 0x7f29c4150c00 2023-11-28T17:45:58.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.195+0000 7f29cc248640 1 -- 172.21.15.98:0/809304001 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c4072460 con 0x7f29c4150c00 2023-11-28T17:45:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.196+0000 7f29cc248640 1 -- 172.21.15.98:0/809304001 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c4067bf0 con 0x7f29c4150c00 2023-11-28T17:45:58.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.197+0000 7f29b67fc640 1 -- 172.21.15.98:0/809304001 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f29bc03e940 con 0x7f29c4150c00 2023-11-28T17:45:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.199+0000 7f29b67fc640 1 --2- 172.21.15.98:0/809304001 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f29a806b250 0x7f29a806d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:45:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.199+0000 7f29b67fc640 1 -- 172.21.15.98:0/809304001 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f29bc0aca90 con 0x7f29c4150c00 2023-11-28T17:45:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.199+0000 7f29c9fe4640 1 --2- 172.21.15.98:0/809304001 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f29a806b250 0x7f29a806d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:45:58.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.200+0000 7f29c9fe4640 1 --2- 172.21.15.98:0/809304001 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f29a806b250 0x7f29a806d710 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f29c00038b0 tx=0x7f29c0009110 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:45:58.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.205+0000 7f29b67fc640 1 -- 172.21.15.98:0/809304001 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29bc077750 con 0x7f29c4150c00 2023-11-28T17:45:58.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.467+0000 7f29cc248640 1 -- 172.21.15.98:0/809304001 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f29c4074f60 con 0x7f29c4150c00 2023-11-28T17:45:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.468+0000 7f29b67fc640 1 -- 172.21.15.98:0/809304001 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f29bc043070 con 0x7f29c4150c00 2023-11-28T17:45:58.469 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:45:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.471+0000 7f29cc248640 1 -- 172.21.15.98:0/809304001 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f29a806b250 msgr2=0x7f29a806d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.471+0000 7f29cc248640 1 --2- 172.21.15.98:0/809304001 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f29a806b250 0x7f29a806d710 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f29c00038b0 tx=0x7f29c0009110 comp rx=0 tx=0).stop 2023-11-28T17:45:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.471+0000 7f29cc248640 1 -- 172.21.15.98:0/809304001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29c4150c00 msgr2=0x7f29c4155ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:45:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.471+0000 7f29cc248640 1 --2- 172.21.15.98:0/809304001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29c4150c00 0x7f29c4155ce0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009e50 tx=0x7f29bc004a40 comp rx=0 tx=0).stop 2023-11-28T17:45:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.471+0000 7f29cc248640 1 -- 172.21.15.98:0/809304001 shutdown_connections 2023-11-28T17:45:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.471+0000 7f29cc248640 1 --2- 172.21.15.98:0/809304001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29c4150c00 0x7f29c4155ce0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.471+0000 7f29cc248640 1 --2- 172.21.15.98:0/809304001 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f29a806b250 0x7f29a806d710 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.471+0000 7f29cc248640 1 --2- 172.21.15.98:0/809304001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f29c40c5c00 0x7f29c4152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.471+0000 7f29cc248640 1 --2- 172.21.15.98:0/809304001 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f29c40c4a20 0x7f29c41efef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:45:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.471+0000 7f29cc248640 1 -- 172.21.15.98:0/809304001 >> 172.21.15.98:0/809304001 conn(0x7f29c406a4d0 msgr2=0x7f29c40c7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:45:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.471+0000 7f29cc248640 1 -- 172.21.15.98:0/809304001 shutdown_connections 2023-11-28T17:45:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:45:58.471+0000 7f29cc248640 1 -- 172.21.15.98:0/809304001 wait complete. 2023-11-28T17:45:59.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~68s 2023-11-28T17:46:01.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:46:01.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.713+0000 7f01989b0640 1 Processor -- start 2023-11-28T17:46:01.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.713+0000 7f01989b0640 1 -- start start 2023-11-28T17:46:01.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.715+0000 7f01989b0640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0194157b60 0x7f019415bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:01.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.715+0000 7f01989b0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019415b150 con 0x7f0194151d80 2023-11-28T17:46:01.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.715+0000 7f01989b0640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019406e1f0 con 0x7f01940c4910 2023-11-28T17:46:01.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.715+0000 7f01989b0640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0194157f90 con 0x7f0194157b60 2023-11-28T17:46:01.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.715+0000 7f018bfff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f0194157b60 msgr2=0x7f019415bc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:46:01.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.715+0000 7f018bfff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0194157b60 0x7f019415bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:01.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.715+0000 7f0192d9d640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f01940c4910 legacy=0x7f01941624e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:46:01.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.715+0000 7f019259c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0194151d80 0x7f01940c5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50530/0 (socket says 172.21.15.98:50530) 2023-11-28T17:46:01.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.715+0000 7f019259c640 1 -- 172.21.15.98:0/1862301024 learned_addr learned my addr 172.21.15.98:0/1862301024 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:46:01.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f0191d9b640 1 -- 172.21.15.98:0/1862301024 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306197595 0 0) 0x7f019415b150 con 0x7f0194151d80 2023-11-28T17:46:01.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f0191d9b640 1 -- 172.21.15.98:0/1862301024 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0180003520 con 0x7f0194151d80 2023-11-28T17:46:01.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f0191d9b640 1 -- 172.21.15.98:0/1862301024 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1415312947 0 0) 0x7f0180003520 con 0x7f0194151d80 2023-11-28T17:46:01.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f0191d9b640 1 -- 172.21.15.98:0/1862301024 >> v2:172.21.15.113:3300/0 conn(0x7f0194157b60 msgr2=0x7f019415bc10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:46:01.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f0191d9b640 1 --2- 172.21.15.98:0/1862301024 >> v2:172.21.15.113:3300/0 conn(0x7f0194157b60 0x7f019415bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:01.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f0191d9b640 1 -- 172.21.15.98:0/1862301024 >> v1:172.21.15.113:6789/0 conn(0x7f01940c4910 legacy=0x7f01941624e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:46:01.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f0191d9b640 1 -- 172.21.15.98:0/1862301024 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019406e4d0 con 0x7f0194151d80 2023-11-28T17:46:01.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f0191d9b640 1 -- 172.21.15.98:0/1862301024 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f0184001670 con 0x7f0194151d80 2023-11-28T17:46:01.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f0191d9b640 1 -- 172.21.15.98:0/1862301024 >> v1:172.21.15.98:6789/0 conn(0x7f0194151d80 legacy=0x7f01940c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:01.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f0191d9b640 1 --2- 172.21.15.98:0/1862301024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0180003970 0x7f0180003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:01.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f0191d9b640 1 --2- 172.21.15.98:0/1862301024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01800047c0 0x7f0180004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:01.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f0191d9b640 1 --2- 172.21.15.98:0/1862301024 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01800051d0 0x7f018000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:01.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f0191d9b640 1 -- 172.21.15.98:0/1862301024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018000c0e0 con 0x7f01800047c0 2023-11-28T17:46:01.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f0191d9b640 1 -- 172.21.15.98:0/1862301024 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018000c220 con 0x7f0180003970 2023-11-28T17:46:01.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f0191d9b640 1 -- 172.21.15.98:0/1862301024 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018000c3e0 con 0x7f01800051d0 2023-11-28T17:46:01.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f019259c640 1 -- 172.21.15.98:0/1862301024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0180003970 msgr2=0x7f0180003d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:46:01.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f018bfff640 1 --2- 172.21.15.98:0/1862301024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01800047c0 0x7f0180004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:01.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.716+0000 7f019259c640 1 --2- 172.21.15.98:0/1862301024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0180003970 0x7f0180003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:01.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.717+0000 7f018bfff640 1 -- 172.21.15.98:0/1862301024 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01800051d0 msgr2=0x7f018000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:01.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.717+0000 7f018bfff640 1 --2- 172.21.15.98:0/1862301024 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01800051d0 0x7f018000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:01.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.717+0000 7f018bfff640 1 -- 172.21.15.98:0/1862301024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0180003970 msgr2=0x7f0180003d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:46:01.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.717+0000 7f018bfff640 1 --2- 172.21.15.98:0/1862301024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0180003970 0x7f0180003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:01.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.717+0000 7f018bfff640 1 -- 172.21.15.98:0/1862301024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0180003520 con 0x7f01800047c0 2023-11-28T17:46:01.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.717+0000 7f0192d9d640 1 -- 172.21.15.98:0/1862301024 reap_dead start 2023-11-28T17:46:01.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.718+0000 7f018bfff640 1 --2- 172.21.15.98:0/1862301024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01800047c0 0x7f0180004c40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f017c009900 tx=0x7f01800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f52263df7521ff server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:01.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.718+0000 7f0191d9b640 1 -- 172.21.15.98:0/1862301024 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c003bb0 con 0x7f01800047c0 2023-11-28T17:46:01.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.718+0000 7f0191d9b640 1 -- 172.21.15.98:0/1862301024 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017c003d10 con 0x7f01800047c0 2023-11-28T17:46:01.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.718+0000 7f0191d9b640 1 -- 172.21.15.98:0/1862301024 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c0052b0 con 0x7f01800047c0 2023-11-28T17:46:01.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.719+0000 7f01989b0640 1 -- 172.21.15.98:0/1862301024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01800047c0 msgr2=0x7f0180004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:01.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.719+0000 7f01989b0640 1 --2- 172.21.15.98:0/1862301024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01800047c0 0x7f0180004c40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f017c009900 tx=0x7f01800057f0 comp rx=0 tx=0).stop 2023-11-28T17:46:01.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.719+0000 7f01989b0640 1 -- 172.21.15.98:0/1862301024 shutdown_connections 2023-11-28T17:46:01.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.719+0000 7f01989b0640 1 --2- 172.21.15.98:0/1862301024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01800047c0 0x7f0180004c40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:01.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.719+0000 7f01989b0640 1 -- 172.21.15.98:0/1862301024 >> 172.21.15.98:0/1862301024 conn(0x7f019406a4d0 msgr2=0x7f019406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:01.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.719+0000 7f01989b0640 1 -- 172.21.15.98:0/1862301024 shutdown_connections 2023-11-28T17:46:01.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.719+0000 7f01989b0640 1 -- 172.21.15.98:0/1862301024 wait complete. 2023-11-28T17:46:01.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.721+0000 7f01989b0640 1 Processor -- start 2023-11-28T17:46:01.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.721+0000 7f01989b0640 1 -- start start 2023-11-28T17:46:01.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.722+0000 7f01989b0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01800047c0 0x7f01940cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:01.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.722+0000 7f01989b0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01941e3290 0x7f01941e1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:01.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.722+0000 7f01989b0640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01941e1c70 0x7f01941f06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:01.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.722+0000 7f01989b0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01940c19c0 con 0x7f01800047c0 2023-11-28T17:46:01.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.722+0000 7f01989b0640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01941ed140 con 0x7f01941e3290 2023-11-28T17:46:01.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.722+0000 7f01989b0640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01941ee110 con 0x7f01941e1c70 2023-11-28T17:46:01.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.723+0000 7f019259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01800047c0 0x7f01940cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:01.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.723+0000 7f018bfff640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01941e3290 msgr2=0x7f01941e1730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:46:01.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.723+0000 7f018bfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01941e3290 0x7f01941e1730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:01.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.723+0000 7f019259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01800047c0 0x7f01940cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46426/0 (socket says 172.21.15.98:46426) 2023-11-28T17:46:01.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.723+0000 7f019259c640 1 -- 172.21.15.98:0/2140706934 learned_addr learned my addr 172.21.15.98:0/2140706934 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:01.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.723+0000 7f0192d9d640 1 --2- 172.21.15.98:0/2140706934 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01941e1c70 0x7f01941f06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:01.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.723+0000 7f019259c640 1 -- 172.21.15.98:0/2140706934 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01941e1c70 msgr2=0x7f01941f06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:01.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.723+0000 7f019259c640 1 --2- 172.21.15.98:0/2140706934 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01941e1c70 0x7f01941f06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:01.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.723+0000 7f019259c640 1 -- 172.21.15.98:0/2140706934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01941e3290 msgr2=0x7f01941e1730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:46:01.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.723+0000 7f019259c640 1 --2- 172.21.15.98:0/2140706934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01941e3290 0x7f01941e1730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:01.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.723+0000 7f019259c640 1 -- 172.21.15.98:0/2140706934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01940c1700 con 0x7f01800047c0 2023-11-28T17:46:01.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.723+0000 7f0192d9d640 1 --2- 172.21.15.98:0/2140706934 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01941e1c70 0x7f01941f06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:46:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.723+0000 7f019259c640 1 --2- 172.21.15.98:0/2140706934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01800047c0 0x7f01940cb470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f018400c750 tx=0x7f018400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.723+0000 7f018affd640 1 -- 172.21.15.98:0/2140706934 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01840074e0 con 0x7f01800047c0 2023-11-28T17:46:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.724+0000 7f018affd640 1 -- 172.21.15.98:0/2140706934 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0184007640 con 0x7f01800047c0 2023-11-28T17:46:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.724+0000 7f018affd640 1 -- 172.21.15.98:0/2140706934 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0184004a20 con 0x7f01800047c0 2023-11-28T17:46:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.724+0000 7f01989b0640 1 -- 172.21.15.98:0/2140706934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01941ed350 con 0x7f01800047c0 2023-11-28T17:46:01.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.724+0000 7f01989b0640 1 -- 172.21.15.98:0/2140706934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01941ed910 con 0x7f01800047c0 2023-11-28T17:46:01.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.725+0000 7f01989b0640 1 -- 172.21.15.98:0/2140706934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0194067bf0 con 0x7f01800047c0 2023-11-28T17:46:01.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.726+0000 7f018affd640 1 -- 172.21.15.98:0/2140706934 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0184004b80 con 0x7f01800047c0 2023-11-28T17:46:01.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.727+0000 7f018affd640 1 --2- 172.21.15.98:0/2140706934 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f017406b280 0x7f017406d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:01.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.727+0000 7f018affd640 1 -- 172.21.15.98:0/2140706934 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0184089060 con 0x7f01800047c0 2023-11-28T17:46:01.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.727+0000 7f018bfff640 1 --2- 172.21.15.98:0/2140706934 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f017406b280 0x7f017406d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:01.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.728+0000 7f018bfff640 1 --2- 172.21.15.98:0/2140706934 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f017406b280 0x7f017406d740 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f017c009900 tx=0x7f017c005000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:01.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.731+0000 7f018affd640 1 -- 172.21.15.98:0/2140706934 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01840bbe90 con 0x7f01800047c0 2023-11-28T17:46:01.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.989+0000 7f01989b0640 1 -- 172.21.15.98:0/2140706934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0194074f60 con 0x7f01800047c0 2023-11-28T17:46:01.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.990+0000 7f018affd640 1 -- 172.21.15.98:0/2140706934 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f0184018070 con 0x7f01800047c0 2023-11-28T17:46:01.991 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:46:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.993+0000 7f01989b0640 1 -- 172.21.15.98:0/2140706934 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f017406b280 msgr2=0x7f017406d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.993+0000 7f01989b0640 1 --2- 172.21.15.98:0/2140706934 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f017406b280 0x7f017406d740 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f017c009900 tx=0x7f017c005000 comp rx=0 tx=0).stop 2023-11-28T17:46:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.993+0000 7f01989b0640 1 -- 172.21.15.98:0/2140706934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01800047c0 msgr2=0x7f01940cb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.993+0000 7f01989b0640 1 --2- 172.21.15.98:0/2140706934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01800047c0 0x7f01940cb470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f018400c750 tx=0x7f018400cc10 comp rx=0 tx=0).stop 2023-11-28T17:46:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.993+0000 7f01989b0640 1 -- 172.21.15.98:0/2140706934 shutdown_connections 2023-11-28T17:46:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.993+0000 7f01989b0640 1 --2- 172.21.15.98:0/2140706934 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f01941e1c70 0x7f01941f06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.993+0000 7f01989b0640 1 --2- 172.21.15.98:0/2140706934 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f017406b280 0x7f017406d740 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.993+0000 7f01989b0640 1 --2- 172.21.15.98:0/2140706934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01941e3290 0x7f01941e1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.993+0000 7f01989b0640 1 --2- 172.21.15.98:0/2140706934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01800047c0 0x7f01940cb470 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.993+0000 7f01989b0640 1 -- 172.21.15.98:0/2140706934 >> 172.21.15.98:0/2140706934 conn(0x7f019406a4d0 msgr2=0x7f019406ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.993+0000 7f01989b0640 1 -- 172.21.15.98:0/2140706934 shutdown_connections 2023-11-28T17:46:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:01.993+0000 7f01989b0640 1 -- 172.21.15.98:0/2140706934 wait complete. 2023-11-28T17:46:05.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:46:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.238+0000 7f3b06a9b640 1 Processor -- start 2023-11-28T17:46:05.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.239+0000 7f3b06a9b640 1 -- start start 2023-11-28T17:46:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.240+0000 7f3b06a9b640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b000cfab0 0x7f3b00159e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.240+0000 7f3b06a9b640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3b000cb780 0x7f3b0015a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.240+0000 7f3b06a9b640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b0015aa10 con 0x7f3b000cf6d0 2023-11-28T17:46:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.240+0000 7f3b06a9b640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b000c7d10 con 0x7f3b000cb780 2023-11-28T17:46:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.240+0000 7f3b06a9b640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0015acf0 con 0x7f3b000cfab0 2023-11-28T17:46:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.241+0000 7f3b04837640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f3b000cf6d0 legacy=0x7f3b000c7600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:46:05.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.241+0000 7f3b05038640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f3b000cb780 msgr2=0x7f3b0015a3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:46:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.241+0000 7f3b05038640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f3b000cb780 0x7f3b0015a3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.241+0000 7f3afffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b000cfab0 0x7f3b00159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.241+0000 7f3afffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b000cfab0 0x7f3b00159e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:51858/0 (socket says 172.21.15.98:51858) 2023-11-28T17:46:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.241+0000 7f3afffff640 1 -- 172.21.15.98:0/3074076003 learned_addr learned my addr 172.21.15.98:0/3074076003 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.242+0000 7f3afffff640 1 -- 172.21.15.98:0/3074076003 >> v2:172.21.15.113:3300/0 conn(0x7f3b000cb780 msgr2=0x7f3b0015a3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:46:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.242+0000 7f3afffff640 1 --2- 172.21.15.98:0/3074076003 >> v2:172.21.15.113:3300/0 conn(0x7f3b000cb780 0x7f3b0015a3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.242+0000 7f3afffff640 1 -- 172.21.15.98:0/3074076003 >> v1:172.21.15.113:6789/0 conn(0x7f3b000cf6d0 legacy=0x7f3b000c7600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:46:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.242+0000 7f3afffff640 1 -- 172.21.15.98:0/3074076003 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0015d430 con 0x7f3b000cfab0 2023-11-28T17:46:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.243+0000 7f3afffff640 1 --2- 172.21.15.98:0/3074076003 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b000cfab0 0x7f3b00159e80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3aec009870 tx=0x7f3aec02b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23c60ead30c36d00 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.244+0000 7f3aff7fe640 1 -- 172.21.15.98:0/3074076003 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec031870 con 0x7f3b000cfab0 2023-11-28T17:46:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.244+0000 7f3aff7fe640 1 -- 172.21.15.98:0/3074076003 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aec0319d0 con 0x7f3b000cfab0 2023-11-28T17:46:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.244+0000 7f3aff7fe640 1 -- 172.21.15.98:0/3074076003 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec031d60 con 0x7f3b000cfab0 2023-11-28T17:46:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.245+0000 7f3b06a9b640 1 -- 172.21.15.98:0/3074076003 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b000cfab0 msgr2=0x7f3b00159e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.245+0000 7f3b06a9b640 1 --2- 172.21.15.98:0/3074076003 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b000cfab0 0x7f3b00159e80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3aec009870 tx=0x7f3aec02b910 comp rx=0 tx=0).stop 2023-11-28T17:46:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.245+0000 7f3b06a9b640 1 -- 172.21.15.98:0/3074076003 shutdown_connections 2023-11-28T17:46:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.245+0000 7f3b06a9b640 1 --2- 172.21.15.98:0/3074076003 >> v2:172.21.15.113:3300/0 conn(0x7f3b000cb780 0x7f3b0015a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.245+0000 7f3b06a9b640 1 --2- 172.21.15.98:0/3074076003 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b000cfab0 0x7f3b00159e80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.245+0000 7f3b06a9b640 1 -- 172.21.15.98:0/3074076003 >> 172.21.15.98:0/3074076003 conn(0x7f3b0006af10 msgr2=0x7f3b0006b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.245+0000 7f3b06a9b640 1 -- 172.21.15.98:0/3074076003 shutdown_connections 2023-11-28T17:46:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.245+0000 7f3b06a9b640 1 -- 172.21.15.98:0/3074076003 wait complete. 2023-11-28T17:46:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.247+0000 7f3b06a9b640 1 Processor -- start 2023-11-28T17:46:05.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.247+0000 7f3b06a9b640 1 -- start start 2023-11-28T17:46:05.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.248+0000 7f3b06a9b640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b000cb780 0x7f3b000c5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:05.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.248+0000 7f3b06a9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b000cf6d0 0x7f3b001547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:05.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.248+0000 7f3b06a9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b000cfab0 0x7f3b00154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:05.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.248+0000 7f3b06a9b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b001e3290 con 0x7f3b000cfab0 2023-11-28T17:46:05.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.248+0000 7f3b06a9b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b001f0780 con 0x7f3b000cf6d0 2023-11-28T17:46:05.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.248+0000 7f3b06a9b640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b001e34e0 con 0x7f3b000cb780 2023-11-28T17:46:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.248+0000 7f3afffff640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b000cf6d0 msgr2=0x7f3b001547f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:46:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.249+0000 7f3afffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b000cf6d0 0x7f3b001547f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.249+0000 7f3b05038640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b000cfab0 0x7f3b00154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.249+0000 7f3b05038640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b000cfab0 0x7f3b00154d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46442/0 (socket says 172.21.15.98:46442) 2023-11-28T17:46:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.249+0000 7f3b05038640 1 -- 172.21.15.98:0/927753661 learned_addr learned my addr 172.21.15.98:0/927753661 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.249+0000 7f3b04837640 1 --2- 172.21.15.98:0/927753661 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b000cb780 0x7f3b000c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.249+0000 7f3b05038640 1 -- 172.21.15.98:0/927753661 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b000cb780 msgr2=0x7f3b000c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.249+0000 7f3b05038640 1 --2- 172.21.15.98:0/927753661 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b000cb780 0x7f3b000c5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.249+0000 7f3b05038640 1 -- 172.21.15.98:0/927753661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b000cf6d0 msgr2=0x7f3b001547f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:46:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.249+0000 7f3b05038640 1 --2- 172.21.15.98:0/927753661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b000cf6d0 0x7f3b001547f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.249+0000 7f3b05038640 1 -- 172.21.15.98:0/927753661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec0094d0 con 0x7f3b000cfab0 2023-11-28T17:46:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.250+0000 7f3b05038640 1 --2- 172.21.15.98:0/927753661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b000cfab0 0x7f3b00154d30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3af800b990 tx=0x7f3af800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.250+0000 7f3afd7fa640 1 -- 172.21.15.98:0/927753661 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af800cb20 con 0x7f3b000cfab0 2023-11-28T17:46:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.250+0000 7f3afd7fa640 1 -- 172.21.15.98:0/927753661 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af800cc80 con 0x7f3b000cfab0 2023-11-28T17:46:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.250+0000 7f3b06a9b640 1 -- 172.21.15.98:0/927753661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b001e37c0 con 0x7f3b000cfab0 2023-11-28T17:46:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.250+0000 7f3afd7fa640 1 -- 172.21.15.98:0/927753661 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af80156b0 con 0x7f3b000cfab0 2023-11-28T17:46:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.250+0000 7f3b06a9b640 1 -- 172.21.15.98:0/927753661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b001e3d80 con 0x7f3b000cfab0 2023-11-28T17:46:05.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.252+0000 7f3b06a9b640 1 -- 172.21.15.98:0/927753661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b000676d0 con 0x7f3b000cfab0 2023-11-28T17:46:05.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.252+0000 7f3afd7fa640 1 -- 172.21.15.98:0/927753661 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3af8012070 con 0x7f3b000cfab0 2023-11-28T17:46:05.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.253+0000 7f3afd7fa640 1 --2- 172.21.15.98:0/927753661 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3ae406b2a0 0x7f3ae406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:05.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.253+0000 7f3afd7fa640 1 -- 172.21.15.98:0/927753661 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3af8089cb0 con 0x7f3b000cfab0 2023-11-28T17:46:05.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.254+0000 7f3b04837640 1 --2- 172.21.15.98:0/927753661 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3ae406b2a0 0x7f3ae406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:05.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.254+0000 7f3b04837640 1 --2- 172.21.15.98:0/927753661 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3ae406b2a0 0x7f3ae406d760 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3af4003930 tx=0x7f3af4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:05.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.258+0000 7f3afd7fa640 1 -- 172.21.15.98:0/927753661 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3af8054a80 con 0x7f3b000cfab0 2023-11-28T17:46:05.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.515+0000 7f3b06a9b640 1 -- 172.21.15.98:0/927753661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3b001e3f70 con 0x7f3b000cfab0 2023-11-28T17:46:05.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.516+0000 7f3afd7fa640 1 -- 172.21.15.98:0/927753661 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f3af80580a0 con 0x7f3b000cfab0 2023-11-28T17:46:05.517 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:46:05.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.518+0000 7f3b06a9b640 1 -- 172.21.15.98:0/927753661 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3ae406b2a0 msgr2=0x7f3ae406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:05.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.518+0000 7f3b06a9b640 1 --2- 172.21.15.98:0/927753661 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3ae406b2a0 0x7f3ae406d760 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3af4003930 tx=0x7f3af4008040 comp rx=0 tx=0).stop 2023-11-28T17:46:05.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.518+0000 7f3b06a9b640 1 -- 172.21.15.98:0/927753661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b000cfab0 msgr2=0x7f3b00154d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:05.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.518+0000 7f3b06a9b640 1 --2- 172.21.15.98:0/927753661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b000cfab0 0x7f3b00154d30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3af800b990 tx=0x7f3af800be50 comp rx=0 tx=0).stop 2023-11-28T17:46:05.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.519+0000 7f3b06a9b640 1 -- 172.21.15.98:0/927753661 shutdown_connections 2023-11-28T17:46:05.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.519+0000 7f3b06a9b640 1 --2- 172.21.15.98:0/927753661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b000cfab0 0x7f3b00154d30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:05.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.519+0000 7f3b06a9b640 1 --2- 172.21.15.98:0/927753661 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3ae406b2a0 0x7f3ae406d760 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:05.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.519+0000 7f3b06a9b640 1 --2- 172.21.15.98:0/927753661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b000cf6d0 0x7f3b001547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:05.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.519+0000 7f3b06a9b640 1 --2- 172.21.15.98:0/927753661 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b000cb780 0x7f3b000c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:05.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.519+0000 7f3b06a9b640 1 -- 172.21.15.98:0/927753661 >> 172.21.15.98:0/927753661 conn(0x7f3b0006af10 msgr2=0x7f3b00066230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:05.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.519+0000 7f3b06a9b640 1 -- 172.21.15.98:0/927753661 shutdown_connections 2023-11-28T17:46:05.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:05.519+0000 7f3b06a9b640 1 -- 172.21.15.98:0/927753661 wait complete. 2023-11-28T17:46:05.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~74s 2023-11-28T17:46:08.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:46:08.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.755+0000 7f312287c640 1 Processor -- start 2023-11-28T17:46:08.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.755+0000 7f312287c640 1 -- start start 2023-11-28T17:46:08.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.757+0000 7f312287c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 0x7f311c0c2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:08.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.757+0000 7f312287c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f311c0c6600 0x7f311c0c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:08.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.757+0000 7f312287c640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c157850 con 0x7f311c0c5c00 2023-11-28T17:46:08.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.757+0000 7f312287c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311c0c6fd0 con 0x7f311c150c00 2023-11-28T17:46:08.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.757+0000 7f312287c640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311c156980 con 0x7f311c0c6600 2023-11-28T17:46:08.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.757+0000 7f3113fff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f311c0c5c00 legacy=0x7f311c0c44e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:46:08.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.757+0000 7f311bfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 0x7f311c0c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:08.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.757+0000 7f3120e19640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f311c0c6600 msgr2=0x7f311c0c6a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:46:08.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.757+0000 7f311bfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 0x7f311c0c2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46454/0 (socket says 172.21.15.98:46454) 2023-11-28T17:46:08.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.757+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 learned_addr learned my addr 172.21.15.98:0/551863161 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:08.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.757+0000 7f3120e19640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.113:3300/0 conn(0x7f311c0c6600 0x7f311c0c6a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:08.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.757+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 msgr2=0x7f311c0c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:46:08.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.757+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 msgr2=0x7f311c0c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:08.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.757+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 0x7f311c0c2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:08.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.757+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 0x7f311c0c2700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:08.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.958+0000 7f3113fff640 1 -- 172.21.15.98:0/551863161 >> v1:172.21.15.113:6789/0 conn(0x7f311c0c5c00 legacy=0x7f311c0c44e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:46:09.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.958+0000 7f3120e19640 1 -- 172.21.15.98:0/551863161 >> v2:172.21.15.113:3300/0 conn(0x7f311c0c6600 msgr2=0x7f311c0c6a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:46:09.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.958+0000 7f3120e19640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.113:3300/0 conn(0x7f311c0c6600 0x7f311c0c6a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:46:09.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:46:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.958+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 0x7f311c0c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.959+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 msgr2=0x7f311c0c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:46:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.959+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 msgr2=0x7f311c0c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.959+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 0x7f311c0c2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:08.959+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 0x7f311c0c2700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:46:09.014 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:46:09.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:09.359+0000 7f3113fff640 1 -- 172.21.15.98:0/551863161 >> v1:172.21.15.113:6789/0 conn(0x7f311c0c5c00 legacy=0x7f311c0c44e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:46:09.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:09.359+0000 7f3120e19640 1 -- 172.21.15.98:0/551863161 >> v2:172.21.15.113:3300/0 conn(0x7f311c0c6600 msgr2=0x7f311c0c6a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:46:09.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:09.359+0000 7f3120e19640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.113:3300/0 conn(0x7f311c0c6600 0x7f311c0c6a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:46:09.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:09.360+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 0x7f311c0c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:09.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:09.361+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 msgr2=0x7f311c0c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:46:09.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:09.361+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 msgr2=0x7f311c0c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:09.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:09.361+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 0x7f311c0c2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:09.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:09.361+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 0x7f311c0c2700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:46:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:10.160+0000 7f3113fff640 1 -- 172.21.15.98:0/551863161 >> v1:172.21.15.113:6789/0 conn(0x7f311c0c5c00 legacy=0x7f311c0c44e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:46:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:10.160+0000 7f3120e19640 1 -- 172.21.15.98:0/551863161 >> v2:172.21.15.113:3300/0 conn(0x7f311c0c6600 msgr2=0x7f311c0c6a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:46:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:10.160+0000 7f3120e19640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.113:3300/0 conn(0x7f311c0c6600 0x7f311c0c6a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:46:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:10.162+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 0x7f311c0c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:10.163+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 msgr2=0x7f311c0c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:46:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:10.163+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 msgr2=0x7f311c0c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:10.163+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 0x7f311c0c2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:10.163+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 0x7f311c0c2700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:46:11.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.757+0000 7f311a7fc640 1 -- 172.21.15.98:0/551863161 >> v2:172.21.15.113:3300/0 conn(0x7f311c0c6600 msgr2=0x7f311c0c6a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:46:11.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.757+0000 7f311a7fc640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.113:3300/0 conn(0x7f311c0c6600 0x7f311c0c6a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:11.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.757+0000 7f311a7fc640 1 -- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 msgr2=0x7f311c0c2700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:46:11.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.757+0000 7f311a7fc640 1 --2- 172.21.15.98:0/551863161 >> v2:172.21.15.98:3300/0 conn(0x7f311c150c00 0x7f311c0c2700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.757+0000 7f311a7fc640 1 -- 172.21.15.98:0/551863161 >> v1:172.21.15.113:6789/0 conn(0x7f311c0c5c00 legacy=0x7f311c0c44e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:46:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.757+0000 7f311a7fc640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 0x7f30fc005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.757+0000 7f311a7fc640 1 -- 172.21.15.98:0/551863161 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c156c40 con 0x7f30fc006130 2023-11-28T17:46:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.757+0000 7f311a7fc640 1 -- 172.21.15.98:0/551863161 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0c6fd0 con 0x7f30fc0036c0 2023-11-28T17:46:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.758+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 0x7f30fc005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.758+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 msgr2=0x7f30fc005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:46:11.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.758+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 msgr2=0x7f30fc005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:11.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.758+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 0x7f30fc005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:11.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.759+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 0x7f30fc005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.960+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 0x7f30fc005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.961+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 msgr2=0x7f30fc005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:46:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.961+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 msgr2=0x7f30fc005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.961+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 0x7f30fc005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:11.961+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 0x7f30fc005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:46:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.362+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 0x7f30fc005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.363+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 msgr2=0x7f30fc005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:46:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.363+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 msgr2=0x7f30fc005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.363+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 0x7f30fc005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:12.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.363+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 0x7f30fc005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:46:12.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~81s 2023-11-28T17:46:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.681+0000 7f311b7fe640 1 -- 172.21.15.98:0/551863161 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261980778 0 0) 0x7f311c156c40 con 0x7f30fc006130 2023-11-28T17:46:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.681+0000 7f311b7fe640 1 -- 172.21.15.98:0/551863161 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3108003520 con 0x7f30fc006130 2023-11-28T17:46:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.682+0000 7f311b7fe640 1 -- 172.21.15.98:0/551863161 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2018321808 0 0) 0x7f3108003520 con 0x7f30fc006130 2023-11-28T17:46:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.682+0000 7f311b7fe640 1 -- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 msgr2=0x7f30fc005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:46:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.682+0000 7f311b7fe640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f30fc0036c0 0x7f30fc005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.682+0000 7f311b7fe640 1 -- 172.21.15.98:0/551863161 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c157850 con 0x7f30fc006130 2023-11-28T17:46:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.682+0000 7f311b7fe640 1 -- 172.21.15.98:0/551863161 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f31040042c0 con 0x7f30fc006130 2023-11-28T17:46:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.682+0000 7f311b7fe640 1 -- 172.21.15.98:0/551863161 >> v1:172.21.15.98:6789/0 conn(0x7f30fc006130 legacy=0x7f30fc0085f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.682+0000 7f3120e19640 1 -- 172.21.15.98:0/551863161 reap_dead start 2023-11-28T17:46:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.682+0000 7f311b7fe640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3108003970 0x7f3108003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.683+0000 7f311b7fe640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31080047c0 0x7f3108004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.683+0000 7f311b7fe640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31080051d0 0x7f310800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.683+0000 7f311b7fe640 1 -- 172.21.15.98:0/551863161 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0c6fd0 con 0x7f31080047c0 2023-11-28T17:46:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.683+0000 7f311b7fe640 1 -- 172.21.15.98:0/551863161 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310800c100 con 0x7f31080051d0 2023-11-28T17:46:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.683+0000 7f311b7fe640 1 -- 172.21.15.98:0/551863161 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310800c2c0 con 0x7f3108003970 2023-11-28T17:46:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.683+0000 7f3120e19640 1 -- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31080051d0 msgr2=0x7f310800b980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:46:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.683+0000 7f3120e19640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31080051d0 0x7f310800b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.684+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3108003970 0x7f3108003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.686+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31080051d0 msgr2=0x7f310800b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:46:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.686+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31080051d0 0x7f310800b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.686+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31080047c0 msgr2=0x7f3108004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.686+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31080047c0 0x7f3108004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.686+0000 7f311bfff640 1 -- 172.21.15.98:0/551863161 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3108003520 con 0x7f3108003970 2023-11-28T17:46:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.688+0000 7f311bfff640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3108003970 0x7f3108003d70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f310c009100 tx=0x7f31080057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5dfe43584820f4d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.688+0000 7f311b7fe640 1 -- 172.21.15.98:0/551863161 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c038410 con 0x7f3108003970 2023-11-28T17:46:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.689+0000 7f311b7fe640 1 -- 172.21.15.98:0/551863161 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310c032030 con 0x7f3108003970 2023-11-28T17:46:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.689+0000 7f311b7fe640 1 -- 172.21.15.98:0/551863161 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c0387b0 con 0x7f3108003970 2023-11-28T17:46:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.690+0000 7f312287c640 1 -- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3108003970 msgr2=0x7f3108003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.690+0000 7f312287c640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3108003970 0x7f3108003d70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f310c009100 tx=0x7f31080057f0 comp rx=0 tx=0).stop 2023-11-28T17:46:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.690+0000 7f312287c640 1 -- 172.21.15.98:0/551863161 shutdown_connections 2023-11-28T17:46:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.690+0000 7f312287c640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31080051d0 0x7f310800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.690+0000 7f312287c640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31080047c0 0x7f3108004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.690+0000 7f312287c640 1 --2- 172.21.15.98:0/551863161 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3108003970 0x7f3108003d70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.690+0000 7f312287c640 1 -- 172.21.15.98:0/551863161 >> 172.21.15.98:0/551863161 conn(0x7f311c06a4d0 msgr2=0x7f311c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.690+0000 7f312287c640 1 -- 172.21.15.98:0/551863161 shutdown_connections 2023-11-28T17:46:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.691+0000 7f312287c640 1 -- 172.21.15.98:0/551863161 wait complete. 2023-11-28T17:46:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.693+0000 7f312287c640 1 Processor -- start 2023-11-28T17:46:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.694+0000 7f312287c640 1 -- start start 2023-11-28T17:46:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.695+0000 7f312287c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3108003970 0x7f311c0c27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.695+0000 7f312287c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31080047c0 0x7f311c149cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.695+0000 7f312287c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31080051d0 0x7f311c070840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.695+0000 7f312287c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c149160 con 0x7f31080051d0 2023-11-28T17:46:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.695+0000 7f312287c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c1ef8d0 con 0x7f3108003970 2023-11-28T17:46:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.695+0000 7f312287c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c14a5a0 con 0x7f31080047c0 2023-11-28T17:46:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.695+0000 7f311bfff640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3108003970 msgr2=0x7f311c0c27c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:46:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.695+0000 7f311bfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3108003970 0x7f311c0c27c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.696+0000 7f3120e19640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31080051d0 0x7f311c070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.696+0000 7f3120e19640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31080051d0 0x7f311c070840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53030/0 (socket says 172.21.15.98:53030) 2023-11-28T17:46:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.696+0000 7f3120e19640 1 -- 172.21.15.98:0/3907065563 learned_addr learned my addr 172.21.15.98:0/3907065563 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.696+0000 7f3113fff640 1 --2- 172.21.15.98:0/3907065563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31080047c0 0x7f311c149cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.696+0000 7f3120e19640 1 -- 172.21.15.98:0/3907065563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31080047c0 msgr2=0x7f311c149cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.696+0000 7f3120e19640 1 --2- 172.21.15.98:0/3907065563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31080047c0 0x7f311c149cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.696+0000 7f3120e19640 1 -- 172.21.15.98:0/3907065563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3108003970 msgr2=0x7f311c0c27c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:46:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.696+0000 7f3120e19640 1 --2- 172.21.15.98:0/3907065563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3108003970 0x7f311c0c27c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.696+0000 7f3120e19640 1 -- 172.21.15.98:0/3907065563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c1efb70 con 0x7f31080051d0 2023-11-28T17:46:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.697+0000 7f3120e19640 1 --2- 172.21.15.98:0/3907065563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31080051d0 0x7f311c070840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009d90 tx=0x7f311400c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.697+0000 7f31197fa640 1 -- 172.21.15.98:0/3907065563 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114016630 con 0x7f31080051d0 2023-11-28T17:46:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.697+0000 7f312287c640 1 -- 172.21.15.98:0/3907065563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311c1efe50 con 0x7f31080051d0 2023-11-28T17:46:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.697+0000 7f31197fa640 1 -- 172.21.15.98:0/3907065563 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31140167b0 con 0x7f31080051d0 2023-11-28T17:46:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.697+0000 7f31197fa640 1 -- 172.21.15.98:0/3907065563 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114016ba0 con 0x7f31080051d0 2023-11-28T17:46:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.698+0000 7f312287c640 1 -- 172.21.15.98:0/3907065563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c14a860 con 0x7f31080051d0 2023-11-28T17:46:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.699+0000 7f312287c640 1 -- 172.21.15.98:0/3907065563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311c067bf0 con 0x7f31080051d0 2023-11-28T17:46:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.702+0000 7f31197fa640 1 -- 172.21.15.98:0/3907065563 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3114012020 con 0x7f31080051d0 2023-11-28T17:46:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.703+0000 7f31197fa640 1 --2- 172.21.15.98:0/3907065563 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f30fc06b920 0x7f30fc06dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.704+0000 7f31197fa640 1 -- 172.21.15.98:0/3907065563 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f311408a4d0 con 0x7f31080051d0 2023-11-28T17:46:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.704+0000 7f311bfff640 1 --2- 172.21.15.98:0/3907065563 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f30fc06b920 0x7f30fc06dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.705+0000 7f311bfff640 1 --2- 172.21.15.98:0/3907065563 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f30fc06b920 0x7f30fc06dde0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f310c039000 tx=0x7f310c01c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.719+0000 7f31197fa640 1 -- 172.21.15.98:0/3907065563 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3114055400 con 0x7f31080051d0 2023-11-28T17:46:12.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.993+0000 7f312287c640 1 -- 172.21.15.98:0/3907065563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f311c0c3db0 con 0x7f31080051d0 2023-11-28T17:46:12.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.994+0000 7f31197fa640 1 -- 172.21.15.98:0/3907065563 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f311401b620 con 0x7f31080051d0 2023-11-28T17:46:12.995 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:46:12.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.996+0000 7f312287c640 1 -- 172.21.15.98:0/3907065563 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f30fc06b920 msgr2=0x7f30fc06dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:12.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.996+0000 7f312287c640 1 --2- 172.21.15.98:0/3907065563 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f30fc06b920 0x7f30fc06dde0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f310c039000 tx=0x7f310c01c040 comp rx=0 tx=0).stop 2023-11-28T17:46:12.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.996+0000 7f312287c640 1 -- 172.21.15.98:0/3907065563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31080051d0 msgr2=0x7f311c070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:12.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.996+0000 7f312287c640 1 --2- 172.21.15.98:0/3907065563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31080051d0 0x7f311c070840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009d90 tx=0x7f311400c420 comp rx=0 tx=0).stop 2023-11-28T17:46:12.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.996+0000 7f312287c640 1 -- 172.21.15.98:0/3907065563 shutdown_connections 2023-11-28T17:46:12.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.996+0000 7f312287c640 1 --2- 172.21.15.98:0/3907065563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31080051d0 0x7f311c070840 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:12.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.996+0000 7f312287c640 1 --2- 172.21.15.98:0/3907065563 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f31080047c0 0x7f311c149cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:12.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.996+0000 7f312287c640 1 --2- 172.21.15.98:0/3907065563 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f30fc06b920 0x7f30fc06dde0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:12.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.996+0000 7f312287c640 1 --2- 172.21.15.98:0/3907065563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3108003970 0x7f311c0c27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:12.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.996+0000 7f312287c640 1 -- 172.21.15.98:0/3907065563 >> 172.21.15.98:0/3907065563 conn(0x7f311c06a4d0 msgr2=0x7f311c0c4470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:12.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.997+0000 7f312287c640 1 -- 172.21.15.98:0/3907065563 shutdown_connections 2023-11-28T17:46:12.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:12.997+0000 7f312287c640 1 -- 172.21.15.98:0/3907065563 wait complete. 2023-11-28T17:46:13.017 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-11-28T17:46:13.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-11-28T17:46:13.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:46:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.230+0000 7f6180190640 1 Processor -- start 2023-11-28T17:46:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.230+0000 7f6180190640 1 -- start start 2023-11-28T17:46:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f6180190640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6178153f40 0x7f61780c8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f6180190640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f61780c90e0 0x7f617806e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f6180190640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617806d4e0 con 0x7f61780c6c30 2023-11-28T17:46:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f6180190640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61780c7c30 con 0x7f61780c90e0 2023-11-28T17:46:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f6180190640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f617806dcf0 con 0x7f6178153f40 2023-11-28T17:46:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f617df2c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f6178153f40 msgr2=0x7f61780c8050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:46:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f617df2c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6178153f40 0x7f61780c8050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f617e72d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f61780c90e0 0x7f617806e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f617e72d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f61780c90e0 0x7f617806e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53036/0 (socket says 172.21.15.98:53036) 2023-11-28T17:46:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f617e72d640 1 -- 172.21.15.98:0/3195378308 learned_addr learned my addr 172.21.15.98:0/3195378308 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f617d72b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f61780c6c30 0x7f6178159cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45978/0 (socket says 172.21.15.98:45978) 2023-11-28T17:46:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f617e72d640 1 -- 172.21.15.98:0/3195378308 >> v2:172.21.15.113:3300/0 conn(0x7f6178153f40 msgr2=0x7f61780c8050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:46:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f617e72d640 1 --2- 172.21.15.98:0/3195378308 >> v2:172.21.15.113:3300/0 conn(0x7f6178153f40 0x7f61780c8050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f617e72d640 1 -- 172.21.15.98:0/3195378308 >> v1:172.21.15.98:6789/0 conn(0x7f61780c6c30 legacy=0x7f6178159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f617e72d640 1 -- 172.21.15.98:0/3195378308 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617806d730 con 0x7f61780c90e0 2023-11-28T17:46:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.232+0000 7f617cf2a640 1 -- 172.21.15.98:0/3195378308 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231205450 0 0) 0x7f617806d4e0 con 0x7f61780c6c30 2023-11-28T17:46:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.234+0000 7f617e72d640 1 --2- 172.21.15.98:0/3195378308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61780c90e0 0x7f617806e0d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f61780cd2e0 tx=0x7f617002ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b683b8b1a99941dc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.234+0000 7f617cf2a640 1 -- 172.21.15.98:0/3195378308 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617002f870 con 0x7f61780c90e0 2023-11-28T17:46:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.234+0000 7f617cf2a640 1 -- 172.21.15.98:0/3195378308 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f617002f9d0 con 0x7f61780c90e0 2023-11-28T17:46:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.234+0000 7f617cf2a640 1 -- 172.21.15.98:0/3195378308 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617002fd60 con 0x7f61780c90e0 2023-11-28T17:46:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.234+0000 7f6180190640 1 -- 172.21.15.98:0/3195378308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61780c90e0 msgr2=0x7f617806e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.235+0000 7f6180190640 1 --2- 172.21.15.98:0/3195378308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61780c90e0 0x7f617806e0d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f61780cd2e0 tx=0x7f617002ba50 comp rx=0 tx=0).stop 2023-11-28T17:46:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.235+0000 7f6180190640 1 -- 172.21.15.98:0/3195378308 shutdown_connections 2023-11-28T17:46:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.235+0000 7f6180190640 1 --2- 172.21.15.98:0/3195378308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61780c90e0 0x7f617806e0d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.235+0000 7f6180190640 1 --2- 172.21.15.98:0/3195378308 >> v2:172.21.15.113:3300/0 conn(0x7f6178153f40 0x7f61780c8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.235+0000 7f6180190640 1 -- 172.21.15.98:0/3195378308 >> 172.21.15.98:0/3195378308 conn(0x7f617806ef10 msgr2=0x7f617806f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.235+0000 7f6180190640 1 -- 172.21.15.98:0/3195378308 shutdown_connections 2023-11-28T17:46:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.235+0000 7f6180190640 1 -- 172.21.15.98:0/3195378308 wait complete. 2023-11-28T17:46:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.236+0000 7f6180190640 1 Processor -- start 2023-11-28T17:46:13.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.237+0000 7f6180190640 1 -- start start 2023-11-28T17:46:13.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.238+0000 7f6180190640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61780c6c30 0x7f617814bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.238+0000 7f6180190640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61780c90e0 0x7f6178147000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.238+0000 7f6180190640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6178153f40 0x7f617814b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.238+0000 7f6180190640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617814e400 con 0x7f61780c6c30 2023-11-28T17:46:13.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.238+0000 7f6180190640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617815db50 con 0x7f61780c90e0 2023-11-28T17:46:13.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.238+0000 7f6180190640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617815d250 con 0x7f6178153f40 2023-11-28T17:46:13.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.238+0000 7f617df2c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61780c6c30 0x7f617814bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:13.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.238+0000 7f617d72b640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61780c90e0 msgr2=0x7f6178147000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:46:13.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.238+0000 7f617d72b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61780c90e0 0x7f6178147000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.238+0000 7f617df2c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61780c6c30 0x7f617814bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53046/0 (socket says 172.21.15.98:53046) 2023-11-28T17:46:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.238+0000 7f617df2c640 1 -- 172.21.15.98:0/3092697451 learned_addr learned my addr 172.21.15.98:0/3092697451 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.239+0000 7f617e72d640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6178153f40 0x7f617814b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.239+0000 7f617df2c640 1 -- 172.21.15.98:0/3092697451 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6178153f40 msgr2=0x7f617814b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.239+0000 7f617df2c640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6178153f40 0x7f617814b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.239+0000 7f617df2c640 1 -- 172.21.15.98:0/3092697451 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61780c90e0 msgr2=0x7f6178147000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:46:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.239+0000 7f617df2c640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61780c90e0 0x7f6178147000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.239+0000 7f617df2c640 1 -- 172.21.15.98:0/3092697451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61700094d0 con 0x7f61780c6c30 2023-11-28T17:46:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.239+0000 7f617e72d640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6178153f40 0x7f617814b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:46:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.239+0000 7f617df2c640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61780c6c30 0x7f617814bba0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f616c00ba10 tx=0x7f616c00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.239+0000 7f61767fc640 1 -- 172.21.15.98:0/3092697451 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c00cbd0 con 0x7f61780c6c30 2023-11-28T17:46:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.239+0000 7f61767fc640 1 -- 172.21.15.98:0/3092697451 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616c00cd50 con 0x7f61780c6c30 2023-11-28T17:46:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.239+0000 7f61767fc640 1 -- 172.21.15.98:0/3092697451 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c0156b0 con 0x7f61780c6c30 2023-11-28T17:46:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.239+0000 7f6180190640 1 -- 172.21.15.98:0/3092697451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617815b510 con 0x7f61780c6c30 2023-11-28T17:46:13.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.239+0000 7f6180190640 1 -- 172.21.15.98:0/3092697451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617815b7f0 con 0x7f61780c6c30 2023-11-28T17:46:13.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.241+0000 7f6180190640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f617815bc60 0x7f617815d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.241+0000 7f6180190640 1 -- 172.21.15.98:0/3092697451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6178074460 con 0x7f617815bc60 2023-11-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.241+0000 7f617d72b640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f617815bc60 0x7f617815d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.241+0000 7f617d72b640 1 -- 172.21.15.98:0/3092697451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61780c7850 con 0x7f617815bc60 2023-11-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.242+0000 7f617d72b640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f617815bc60 0x7f617815d4a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f61780cb3d0 tx=0x7f616400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.242+0000 7f61767fc640 1 -- 172.21.15.98:0/3092697451 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61640074e0 con 0x7f617815bc60 2023-11-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.242+0000 7f61767fc640 1 -- 172.21.15.98:0/3092697451 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f6164007640 con 0x7f617815bc60 2023-11-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.242+0000 7f61767fc640 1 -- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f617815bc60 msgr2=0x7f617815d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.242+0000 7f61767fc640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f617815bc60 0x7f617815d4a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f61780cb3d0 tx=0x7f616400c830 comp rx=0 tx=0).stop 2023-11-28T17:46:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.242+0000 7f61767fc640 1 -- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f617815bc60 msgr2=0x7f617815d4a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.243+0000 7f61767fc640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f617815bc60 0x7f617815d4a0 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f61780cb3d0 tx=0x7f616400c830 comp rx=0 tx=0).stop 2023-11-28T17:46:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.243+0000 7f61767fc640 1 -- 172.21.15.98:0/3092697451 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f616c012070 con 0x7f61780c6c30 2023-11-28T17:46:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.244+0000 7f61767fc640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f615c06b020 0x7f615c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.244+0000 7f61767fc640 1 -- 172.21.15.98:0/3092697451 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f616c089ca0 con 0x7f61780c6c30 2023-11-28T17:46:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.244+0000 7f617d72b640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f615c06b020 0x7f615c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.245+0000 7f617d72b640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f615c06b020 0x7f615c06d4e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6164013000 tx=0x7f6164009000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.251+0000 7f6180190640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f617814b650 0x7f6178143ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.251+0000 7f6180190640 1 -- 172.21.15.98:0/3092697451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61780c7570 con 0x7f617814b650 2023-11-28T17:46:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.251+0000 7f617e72d640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f617814b650 0x7f6178143ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.252+0000 7f617e72d640 1 -- 172.21.15.98:0/3092697451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6178154320 con 0x7f617814b650 2023-11-28T17:46:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.252+0000 7f617e72d640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f617814b650 0x7f6178143ac0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f61780cd280 tx=0x7f6170004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.252+0000 7f61767fc640 1 -- 172.21.15.98:0/3092697451 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6170046070 con 0x7f617814b650 2023-11-28T17:46:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.253+0000 7f61767fc640 1 -- 172.21.15.98:0/3092697451 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4626 (secure 0 0 0) 0x7f617002f810 con 0x7f617814b650 2023-11-28T17:46:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.253+0000 7f61767fc640 1 -- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f617814b650 msgr2=0x7f6178143ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.253+0000 7f61767fc640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f617814b650 0x7f6178143ac0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f61780cd280 tx=0x7f6170004730 comp rx=0 tx=0).stop 2023-11-28T17:46:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.253+0000 7f61767fc640 1 -- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f617814b650 msgr2=0x7f6178143ac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.253+0000 7f61767fc640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f617814b650 0x7f6178143ac0 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f61780cd280 tx=0x7f6170004730 comp rx=0 tx=0).stop 2023-11-28T17:46:13.254 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:46:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:46:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:46:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:46:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 52, 2023-11-28T17:46:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:46:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:46:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:46:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:46:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:46:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:46:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:46:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:46:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:46:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:46:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:46:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:46:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:46:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:46:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:46:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:46:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:46:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:46:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:46:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:46:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:46:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:46:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:46:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:46:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:46:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:46:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:46:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:46:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:46:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:46:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:46:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:46:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:46:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:46:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:46:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:46:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:46:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:46:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:46:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:46:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:46:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:46:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:46:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:46:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:46:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:46:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:46:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:46:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:46:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:46:13.267 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:46:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.253+0000 7f6180190640 1 -- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f615c06b020 msgr2=0x7f615c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.253+0000 7f6180190640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f615c06b020 0x7f615c06d4e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6164013000 tx=0x7f6164009000 comp rx=0 tx=0).stop 2023-11-28T17:46:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.253+0000 7f6180190640 1 -- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61780c6c30 msgr2=0x7f617814bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.253+0000 7f6180190640 1 --2- 172.21.15.98:0/3092697451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61780c6c30 0x7f617814bba0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f616c00ba10 tx=0x7f616c00bed0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.253+0000 7f617e72d640 1 -- 172.21.15.98:0/3092697451 reap_dead start 2023-11-28T17:46:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.253+0000 7f617e72d640 1 -- 172.21.15.98:0/3092697451 reap_dead start 2023-11-28T17:46:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.253+0000 7f6180190640 1 -- 172.21.15.98:0/3092697451 shutdown_connections 2023-11-28T17:46:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.253+0000 7f6180190640 1 -- 172.21.15.98:0/3092697451 >> 172.21.15.98:0/3092697451 conn(0x7f617806ef10 msgr2=0x7f61781530b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.253+0000 7f6180190640 1 -- 172.21.15.98:0/3092697451 shutdown_connections 2023-11-28T17:46:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.253+0000 7f6180190640 1 -- 172.21.15.98:0/3092697451 wait complete. 2023-11-28T17:46:13.270 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:46:13.270 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-11-28T17:46:13.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:46:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.467+0000 7f0d2af89640 1 Processor -- start 2023-11-28T17:46:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.467+0000 7f0d2af89640 1 -- start start 2023-11-28T17:46:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.469+0000 7f0d2af89640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d24153f20 0x7f0d240c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.469+0000 7f0d2af89640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d240c6780 0x7f0d2415a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.469+0000 7f0d2af89640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d24154300 con 0x7f0d240c6350 2023-11-28T17:46:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.469+0000 7f0d2af89640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d24159d80 con 0x7f0d24153f20 2023-11-28T17:46:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.469+0000 7f0d2af89640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d24071790 con 0x7f0d240c6780 2023-11-28T17:46:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.470+0000 7f0d28d25640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d24153f20 0x7f0d240c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.470+0000 7f0d23fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0d240c6350 0x7f0d240c6ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45990/0 (socket says 172.21.15.98:45990) 2023-11-28T17:46:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.470+0000 7f0d23fff640 1 -- 172.21.15.98:0/2102824859 learned_addr learned my addr 172.21.15.98:0/2102824859 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:46:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.470+0000 7f0d29526640 1 --2- 172.21.15.98:0/2102824859 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d240c6780 0x7f0d2415a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.470+0000 7f0d28d25640 1 -- 172.21.15.98:0/2102824859 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d240c6780 msgr2=0x7f0d2415a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.470+0000 7f0d28d25640 1 --2- 172.21.15.98:0/2102824859 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d240c6780 0x7f0d2415a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.470+0000 7f0d28d25640 1 -- 172.21.15.98:0/2102824859 >> v1:172.21.15.98:6789/0 conn(0x7f0d240c6350 legacy=0x7f0d240c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.470+0000 7f0d28d25640 1 -- 172.21.15.98:0/2102824859 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d24071a70 con 0x7f0d24153f20 2023-11-28T17:46:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.471+0000 7f0d28d25640 1 --2- 172.21.15.98:0/2102824859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d24153f20 0x7f0d240c7db0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0d180097a0 tx=0x7f0d1802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73be6e66678c1c19 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.472+0000 7f0d237fe640 1 -- 172.21.15.98:0/2102824859 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18030800 con 0x7f0d24153f20 2023-11-28T17:46:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.472+0000 7f0d237fe640 1 -- 172.21.15.98:0/2102824859 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d18030960 con 0x7f0d24153f20 2023-11-28T17:46:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.472+0000 7f0d237fe640 1 -- 172.21.15.98:0/2102824859 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18030cf0 con 0x7f0d24153f20 2023-11-28T17:46:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.472+0000 7f0d2af89640 1 -- 172.21.15.98:0/2102824859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d24153f20 msgr2=0x7f0d240c7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.472+0000 7f0d2af89640 1 --2- 172.21.15.98:0/2102824859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d24153f20 0x7f0d240c7db0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0d180097a0 tx=0x7f0d1802b940 comp rx=0 tx=0).stop 2023-11-28T17:46:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.473+0000 7f0d2af89640 1 -- 172.21.15.98:0/2102824859 shutdown_connections 2023-11-28T17:46:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.473+0000 7f0d2af89640 1 --2- 172.21.15.98:0/2102824859 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d240c6780 0x7f0d2415a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.473+0000 7f0d2af89640 1 --2- 172.21.15.98:0/2102824859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d24153f20 0x7f0d240c7db0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.473+0000 7f0d2af89640 1 -- 172.21.15.98:0/2102824859 >> 172.21.15.98:0/2102824859 conn(0x7f0d2406e7b0 msgr2=0x7f0d2406eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.473+0000 7f0d2af89640 1 -- 172.21.15.98:0/2102824859 shutdown_connections 2023-11-28T17:46:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.473+0000 7f0d2af89640 1 -- 172.21.15.98:0/2102824859 wait complete. 2023-11-28T17:46:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.474+0000 7f0d2af89640 1 Processor -- start 2023-11-28T17:46:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.475+0000 7f0d2af89640 1 -- start start 2023-11-28T17:46:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d2af89640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d240c6350 0x7f0d241efcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d2af89640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d240c6780 0x7f0d241f0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d2af89640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d24153f20 0x7f0d24154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d2af89640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d24158610 con 0x7f0d240c6350 2023-11-28T17:46:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d2af89640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d241ef880 con 0x7f0d240c6780 2023-11-28T17:46:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d2af89640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d241f3d30 con 0x7f0d24153f20 2023-11-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d23fff640 1 -- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d240c6780 msgr2=0x7f0d241f0210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d28d25640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d240c6350 0x7f0d241efcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d23fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d240c6780 0x7f0d241f0210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d28d25640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d240c6350 0x7f0d241efcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53080/0 (socket says 172.21.15.98:53080) 2023-11-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d28d25640 1 -- 172.21.15.98:0/3898075952 learned_addr learned my addr 172.21.15.98:0/3898075952 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d29526640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d24153f20 0x7f0d24154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d28d25640 1 -- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d24153f20 msgr2=0x7f0d24154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d28d25640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d24153f20 0x7f0d24154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d28d25640 1 -- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d240c6780 msgr2=0x7f0d241f0210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d28d25640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d240c6780 0x7f0d241f0210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.476+0000 7f0d28d25640 1 -- 172.21.15.98:0/3898075952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d18009400 con 0x7f0d240c6350 2023-11-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.477+0000 7f0d28d25640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d240c6350 0x7f0d241efcd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0d1802bf50 tx=0x7f0d180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.477+0000 7f0d217fa640 1 -- 172.21.15.98:0/3898075952 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1803e070 con 0x7f0d240c6350 2023-11-28T17:46:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.477+0000 7f0d217fa640 1 -- 172.21.15.98:0/3898075952 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d18039ad0 con 0x7f0d240c6350 2023-11-28T17:46:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.477+0000 7f0d217fa640 1 -- 172.21.15.98:0/3898075952 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18039e60 con 0x7f0d240c6350 2023-11-28T17:46:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.477+0000 7f0d2af89640 1 -- 172.21.15.98:0/3898075952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d241f37b0 con 0x7f0d240c6350 2023-11-28T17:46:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.477+0000 7f0d2af89640 1 -- 172.21.15.98:0/3898075952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d241595e0 con 0x7f0d240c6350 2023-11-28T17:46:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.478+0000 7f0d2af89640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d2406aa70 0x7f0d24154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.478+0000 7f0d2af89640 1 -- 172.21.15.98:0/3898075952 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d240744a0 con 0x7f0d2406aa70 2023-11-28T17:46:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.479+0000 7f0d23fff640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d2406aa70 0x7f0d24154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.479+0000 7f0d217fa640 1 -- 172.21.15.98:0/3898075952 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0d1804b400 con 0x7f0d240c6350 2023-11-28T17:46:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.479+0000 7f0d23fff640 1 -- 172.21.15.98:0/3898075952 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d24071450 con 0x7f0d2406aa70 2023-11-28T17:46:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.480+0000 7f0d23fff640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d2406aa70 0x7f0d24154c80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0d10004810 tx=0x7f0d1000c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.480+0000 7f0d217fa640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d0806b250 0x7f0d0806d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.480+0000 7f0d217fa640 1 -- 172.21.15.98:0/3898075952 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0d180ac170 con 0x7f0d240c6350 2023-11-28T17:46:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.480+0000 7f0d29526640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d0806b250 0x7f0d0806d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.480+0000 7f0d217fa640 1 -- 172.21.15.98:0/3898075952 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10005ce0 con 0x7f0d2406aa70 2023-11-28T17:46:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.481+0000 7f0d217fa640 1 -- 172.21.15.98:0/3898075952 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f0d10005e40 con 0x7f0d2406aa70 2023-11-28T17:46:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.481+0000 7f0d217fa640 1 -- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d2406aa70 msgr2=0x7f0d24154c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.481+0000 7f0d217fa640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d2406aa70 0x7f0d24154c80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0d10004810 tx=0x7f0d1000c8e0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.481+0000 7f0d217fa640 1 -- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d2406aa70 msgr2=0x7f0d24154c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.481+0000 7f0d217fa640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d2406aa70 0x7f0d24154c80 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0d10004810 tx=0x7f0d1000c8e0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.481+0000 7f0d29526640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d0806b250 0x7f0d0806d710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0d24072a80 tx=0x7f0d1c00b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.488+0000 7f0d2af89640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d240ca7e0 0x7f0d240cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.488+0000 7f0d2af89640 1 -- 172.21.15.98:0/3898075952 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d24159490 con 0x7f0d240ca7e0 2023-11-28T17:46:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.489+0000 7f0d23fff640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d240ca7e0 0x7f0d240cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.489+0000 7f0d23fff640 1 -- 172.21.15.98:0/3898075952 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0d24072030 con 0x7f0d240ca7e0 2023-11-28T17:46:13.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.490+0000 7f0d23fff640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d240ca7e0 0x7f0d240cabf0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0d10012000 tx=0x7f0d10000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:13.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.490+0000 7f0d217fa640 1 -- 172.21.15.98:0/3898075952 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d10023630 con 0x7f0d240ca7e0 2023-11-28T17:46:13.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.491+0000 7f0d217fa640 1 -- 172.21.15.98:0/3898075952 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4964 (secure 0 0 0) 0x7f0d1001be90 con 0x7f0d240ca7e0 2023-11-28T17:46:13.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.491+0000 7f0d217fa640 1 -- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d240ca7e0 msgr2=0x7f0d240cabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:13.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.491+0000 7f0d217fa640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d240ca7e0 0x7f0d240cabf0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0d10012000 tx=0x7f0d10000ed0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.491+0000 7f0d217fa640 1 -- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d240ca7e0 msgr2=0x7f0d240cabf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:13.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.491+0000 7f0d217fa640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d240ca7e0 0x7f0d240cabf0 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0d10012000 tx=0x7f0d10000ed0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.495 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:46:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:46:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:46:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:46:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 52, 2023-11-28T17:46:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:46:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:46:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:46:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:46:13.495 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:46:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:46:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:13.496 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:46:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:46:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:13.497 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:46:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:46:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:46:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:46:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:46:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:46:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:46:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:46:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:46:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:46:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:46:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:46:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:46:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:46:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:13.499 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:46:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.500 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:46:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:46:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:46:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:46:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:46:13.501 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:46:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:46:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.502 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:46:13.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:46:13.503 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:13.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:13.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:13.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:46:13.503 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:13.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:13.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:46:13.503 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:13.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.503 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:13.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:46:13.504 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:46:13.504 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:13.504 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:13.504 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:13.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.504 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:46:13.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:46:13.504 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:13.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:13.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:13.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:46:13.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:13.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:13.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:46:13.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:13.507 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.507 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:13.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:46:13.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:46:13.507 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:13.507 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:13.507 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:13.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:13.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.508 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:46:13.508 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:46:13.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.508 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:13.508 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:13.509 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:46:13.509 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.509 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:13.509 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:46:13.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.509 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:13.509 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:13.510 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:46:13.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.510 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:13.510 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:46:13.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.510 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:13.510 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:13.510 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 8 2023-11-28T17:46:13.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.511 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:13.511 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:46:13.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.511 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:13.511 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:13.511 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:46:13.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.511 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:13.512 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:46:13.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:13.512 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:13.512 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:13.512 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:46:13.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:13.512 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:13.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:13.513 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:46:13.513 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:46:13.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.491+0000 7f0d2af89640 1 -- 172.21.15.98:0/3898075952 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d0806b250 msgr2=0x7f0d0806d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:13.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.491+0000 7f0d2af89640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d0806b250 0x7f0d0806d710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0d24072a80 tx=0x7f0d1c00b040 comp rx=0 tx=0).stop 2023-11-28T17:46:13.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.491+0000 7f0d2af89640 1 -- 172.21.15.98:0/3898075952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d240c6350 msgr2=0x7f0d241efcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:13.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.491+0000 7f0d2af89640 1 --2- 172.21.15.98:0/3898075952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d240c6350 0x7f0d241efcd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0d1802bf50 tx=0x7f0d180046c0 comp rx=0 tx=0).stop 2023-11-28T17:46:13.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.491+0000 7f0d29526640 1 -- 172.21.15.98:0/3898075952 reap_dead start 2023-11-28T17:46:13.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.491+0000 7f0d29526640 1 -- 172.21.15.98:0/3898075952 reap_dead start 2023-11-28T17:46:13.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.492+0000 7f0d2af89640 1 -- 172.21.15.98:0/3898075952 shutdown_connections 2023-11-28T17:46:13.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.492+0000 7f0d2af89640 1 -- 172.21.15.98:0/3898075952 >> 172.21.15.98:0/3898075952 conn(0x7f0d2406e7b0 msgr2=0x7f0d2414eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:13.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.492+0000 7f0d2af89640 1 -- 172.21.15.98:0/3898075952 shutdown_connections 2023-11-28T17:46:13.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:13.492+0000 7f0d2af89640 1 -- 172.21.15.98:0/3898075952 wait complete. 2023-11-28T17:46:13.516 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:46:13.516 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-11-28T17:46:19.276 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~88s 2023-11-28T17:46:25.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~94s 2023-11-28T17:46:32.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~101s 2023-11-28T17:46:33.518 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2023-11-28T17:46:33.518 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2023-11-28T17:46:33.518 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-28T17:46:33.518 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-28T17:46:33.528 INFO:tasks.ceph.mon.b:Started 2023-11-28T17:46:33.528 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:46:33.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:46:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.746+0000 7efe6b51d640 1 Processor -- start 2023-11-28T17:46:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.746+0000 7efe6b51d640 1 -- start start 2023-11-28T17:46:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.747+0000 7efe6b51d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efe64151d80 0x7efe64157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.747+0000 7efe6b51d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efe6415bc10 0x7efe640c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.747+0000 7efe6b51d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efe640c6140 0x7efe640c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe6b51d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe6415b150 con 0x7efe640c6140 2023-11-28T17:46:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe6b51d640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe6406e1f0 con 0x7efe64151d80 2023-11-28T17:46:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe6b51d640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6406e4d0 con 0x7efe6415bc10 2023-11-28T17:46:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe69aba640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efe640c6140 0x7efe640c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe69aba640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efe640c6140 0x7efe640c4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51332/0 (socket says 172.21.15.98:51332) 2023-11-28T17:46:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe692b9640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7efe64151d80 0x7efe64157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe69aba640 1 -- 172.21.15.98:0/1970307557 learned_addr learned my addr 172.21.15.98:0/1970307557 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe68ab8640 1 --2- 172.21.15.98:0/1970307557 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efe6415bc10 0x7efe640c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe692b9640 1 -- 172.21.15.98:0/1970307557 >> v2:172.21.15.113:3300/0 conn(0x7efe64151d80 msgr2=0x7efe64157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:46:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe692b9640 1 -- 172.21.15.98:0/1970307557 >> v2:172.21.15.113:3300/0 conn(0x7efe64151d80 msgr2=0x7efe64157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe692b9640 1 --2- 172.21.15.98:0/1970307557 >> v2:172.21.15.113:3300/0 conn(0x7efe64151d80 0x7efe64157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe692b9640 1 --2- 172.21.15.98:0/1970307557 >> v2:172.21.15.113:3300/0 conn(0x7efe64151d80 0x7efe64157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe69aba640 1 -- 172.21.15.98:0/1970307557 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efe6415bc10 msgr2=0x7efe640c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe69aba640 1 --2- 172.21.15.98:0/1970307557 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efe6415bc10 0x7efe640c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe69aba640 1 -- 172.21.15.98:0/1970307557 >> v2:172.21.15.113:3300/0 conn(0x7efe64151d80 msgr2=0x7efe64157b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:46:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe69aba640 1 --2- 172.21.15.98:0/1970307557 >> v2:172.21.15.113:3300/0 conn(0x7efe64151d80 0x7efe64157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe69aba640 1 -- 172.21.15.98:0/1970307557 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6406e7b0 con 0x7efe640c6140 2023-11-28T17:46:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.748+0000 7efe68ab8640 1 --2- 172.21.15.98:0/1970307557 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efe6415bc10 0x7efe640c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:46:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.750+0000 7efe69aba640 1 --2- 172.21.15.98:0/1970307557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe640c6140 0x7efe640c4910 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efe640c7c60 tx=0x7efe5c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7161df67e054a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.750+0000 7efe5bfff640 1 -- 172.21.15.98:0/1970307557 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5c02cb00 con 0x7efe640c6140 2023-11-28T17:46:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.750+0000 7efe5bfff640 1 -- 172.21.15.98:0/1970307557 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5c02cc60 con 0x7efe640c6140 2023-11-28T17:46:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.750+0000 7efe5bfff640 1 -- 172.21.15.98:0/1970307557 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5c0343f0 con 0x7efe640c6140 2023-11-28T17:46:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.750+0000 7efe6b51d640 1 -- 172.21.15.98:0/1970307557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe640c6140 msgr2=0x7efe640c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.750+0000 7efe6b51d640 1 --2- 172.21.15.98:0/1970307557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe640c6140 0x7efe640c4910 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efe640c7c60 tx=0x7efe5c02bf60 comp rx=0 tx=0).stop 2023-11-28T17:46:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.750+0000 7efe6b51d640 1 -- 172.21.15.98:0/1970307557 shutdown_connections 2023-11-28T17:46:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.750+0000 7efe6b51d640 1 --2- 172.21.15.98:0/1970307557 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efe6415bc10 0x7efe640c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.750+0000 7efe6b51d640 1 --2- 172.21.15.98:0/1970307557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe640c6140 0x7efe640c4910 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.750+0000 7efe6b51d640 1 --2- 172.21.15.98:0/1970307557 >> v2:172.21.15.113:3300/0 conn(0x7efe64151d80 0x7efe64157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.751+0000 7efe6b51d640 1 -- 172.21.15.98:0/1970307557 >> 172.21.15.98:0/1970307557 conn(0x7efe6406a4d0 msgr2=0x7efe6406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.751+0000 7efe6b51d640 1 -- 172.21.15.98:0/1970307557 shutdown_connections 2023-11-28T17:46:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.751+0000 7efe6b51d640 1 -- 172.21.15.98:0/1970307557 wait complete. 2023-11-28T17:46:33.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.752+0000 7efe6b51d640 1 Processor -- start 2023-11-28T17:46:33.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.752+0000 7efe6b51d640 1 -- start start 2023-11-28T17:46:33.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.753+0000 7efe6b51d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efe640c6140 0x7efe641f0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:33.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.753+0000 7efe6b51d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe64151d80 0x7efe641ee300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:33.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.753+0000 7efe6b51d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe6415bc10 0x7efe64071230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:33.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.753+0000 7efe6b51d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe641ede70 con 0x7efe64151d80 2023-11-28T17:46:33.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.753+0000 7efe6b51d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe641f1db0 con 0x7efe6415bc10 2023-11-28T17:46:33.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.753+0000 7efe6b51d640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe641f1510 con 0x7efe640c6140 2023-11-28T17:46:33.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.754+0000 7efe68ab8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe64151d80 0x7efe641ee300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:33.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.754+0000 7efe68ab8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe64151d80 0x7efe641ee300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51344/0 (socket says 172.21.15.98:51344) 2023-11-28T17:46:33.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.754+0000 7efe68ab8640 1 -- 172.21.15.98:0/3500067415 learned_addr learned my addr 172.21.15.98:0/3500067415 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:33.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.754+0000 7efe692b9640 1 --2- 172.21.15.98:0/3500067415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efe640c6140 0x7efe641f0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:33.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.754+0000 7efe68ab8640 1 -- 172.21.15.98:0/3500067415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efe640c6140 msgr2=0x7efe641f0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:33.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.754+0000 7efe68ab8640 1 --2- 172.21.15.98:0/3500067415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efe640c6140 0x7efe641f0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:33.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.754+0000 7efe68ab8640 1 -- 172.21.15.98:0/3500067415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe6415bc10 msgr2=0x7efe64071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:33.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.754+0000 7efe68ab8640 1 --2- 172.21.15.98:0/3500067415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe6415bc10 0x7efe64071230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:33.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.754+0000 7efe68ab8640 1 -- 172.21.15.98:0/3500067415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe54009400 con 0x7efe64151d80 2023-11-28T17:46:33.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.754+0000 7efe68ab8640 1 --2- 172.21.15.98:0/3500067415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe64151d80 0x7efe641ee300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efe5400e960 tx=0x7efe5400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:33.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.755+0000 7efe59ffb640 1 -- 172.21.15.98:0/3500067415 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5400ca00 con 0x7efe64151d80 2023-11-28T17:46:33.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.755+0000 7efe59ffb640 1 -- 172.21.15.98:0/3500067415 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5400cb60 con 0x7efe64151d80 2023-11-28T17:46:33.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.755+0000 7efe6b51d640 1 -- 172.21.15.98:0/3500067415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5c009b30 con 0x7efe64151d80 2023-11-28T17:46:33.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.755+0000 7efe59ffb640 1 -- 172.21.15.98:0/3500067415 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe54013470 con 0x7efe64151d80 2023-11-28T17:46:33.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.755+0000 7efe6b51d640 1 -- 172.21.15.98:0/3500067415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe641f1ad0 con 0x7efe64151d80 2023-11-28T17:46:33.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.756+0000 7efe6b51d640 1 -- 172.21.15.98:0/3500067415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe64067bf0 con 0x7efe64151d80 2023-11-28T17:46:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.757+0000 7efe59ffb640 1 -- 172.21.15.98:0/3500067415 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7efe540135d0 con 0x7efe64151d80 2023-11-28T17:46:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.758+0000 7efe59ffb640 1 --2- 172.21.15.98:0/3500067415 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7efe4806b2a0 0x7efe4806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.758+0000 7efe59ffb640 1 -- 172.21.15.98:0/3500067415 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7efe5408a2f0 con 0x7efe64151d80 2023-11-28T17:46:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.758+0000 7efe692b9640 1 --2- 172.21.15.98:0/3500067415 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7efe4806b2a0 0x7efe4806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.759+0000 7efe692b9640 1 --2- 172.21.15.98:0/3500067415 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7efe4806b2a0 0x7efe4806d760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efe60006fd0 tx=0x7efe60008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:33.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:33.762+0000 7efe59ffb640 1 -- 172.21.15.98:0/3500067415 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efe54055040 con 0x7efe64151d80 2023-11-28T17:46:34.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:34.022+0000 7efe6b51d640 1 -- 172.21.15.98:0/3500067415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efe6406b2d0 con 0x7efe64151d80 2023-11-28T17:46:34.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:34.023+0000 7efe59ffb640 1 -- 172.21.15.98:0/3500067415 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7efe54058660 con 0x7efe64151d80 2023-11-28T17:46:34.025 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:46:34.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:34.026+0000 7efe6b51d640 1 -- 172.21.15.98:0/3500067415 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7efe4806b2a0 msgr2=0x7efe4806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:34.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:34.026+0000 7efe6b51d640 1 --2- 172.21.15.98:0/3500067415 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7efe4806b2a0 0x7efe4806d760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efe60006fd0 tx=0x7efe60008040 comp rx=0 tx=0).stop 2023-11-28T17:46:34.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:34.026+0000 7efe6b51d640 1 -- 172.21.15.98:0/3500067415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe64151d80 msgr2=0x7efe641ee300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:34.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:34.026+0000 7efe6b51d640 1 --2- 172.21.15.98:0/3500067415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe64151d80 0x7efe641ee300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efe5400e960 tx=0x7efe5400ee20 comp rx=0 tx=0).stop 2023-11-28T17:46:34.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:34.026+0000 7efe6b51d640 1 -- 172.21.15.98:0/3500067415 shutdown_connections 2023-11-28T17:46:34.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:34.026+0000 7efe6b51d640 1 --2- 172.21.15.98:0/3500067415 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7efe4806b2a0 0x7efe4806d760 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:34.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:34.026+0000 7efe6b51d640 1 --2- 172.21.15.98:0/3500067415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efe6415bc10 0x7efe64071230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:34.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:34.026+0000 7efe6b51d640 1 --2- 172.21.15.98:0/3500067415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efe64151d80 0x7efe641ee300 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:34.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:34.026+0000 7efe6b51d640 1 --2- 172.21.15.98:0/3500067415 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7efe640c6140 0x7efe641f0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:34.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:34.026+0000 7efe6b51d640 1 -- 172.21.15.98:0/3500067415 >> 172.21.15.98:0/3500067415 conn(0x7efe6406a4d0 msgr2=0x7efe64150960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:34.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:34.026+0000 7efe6b51d640 1 -- 172.21.15.98:0/3500067415 shutdown_connections 2023-11-28T17:46:34.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:34.026+0000 7efe6b51d640 1 -- 172.21.15.98:0/3500067415 wait complete. 2023-11-28T17:46:37.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:46:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.267+0000 7f2034fd5640 1 Processor -- start 2023-11-28T17:46:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.267+0000 7f2034fd5640 1 -- start start 2023-11-28T17:46:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.269+0000 7f2034fd5640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 0x7f20300c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.269+0000 7f2034fd5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 0x7f20300c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.269+0000 7f2034fd5640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20300cab90 con 0x7f20300ca250 2023-11-28T17:46:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.269+0000 7f2034fd5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20300c7470 con 0x7f20300c5f10 2023-11-28T17:46:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.269+0000 7f2034fd5640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20300c66b0 con 0x7f20301582e0 2023-11-28T17:46:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.269+0000 7f2025d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 0x7f20300c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.269+0000 7f2025d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 0x7f20300c9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51346/0 (socket says 172.21.15.98:51346) 2023-11-28T17:46:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.269+0000 7f2025d9b640 1 -- 172.21.15.98:0/3126795537 learned_addr learned my addr 172.21.15.98:0/3126795537 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.270+0000 7f2025d9b640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 msgr2=0x7f20300c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:46:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.270+0000 7f2025d9b640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 msgr2=0x7f20300c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.270+0000 7f2025d9b640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 0x7f20300c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.270+0000 7f202e59c640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 0x7f20300c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.270+0000 7f2025d9b640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 0x7f20300c9d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.270+0000 7f202e59c640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 msgr2=0x7f20300c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:46:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.270+0000 7f202e59c640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 msgr2=0x7f20300c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.270+0000 7f202e59c640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 0x7f20300c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.270+0000 7f202e59c640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 0x7f20300c9000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.471+0000 7f2025d9b640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 0x7f20300c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.471+0000 7f202e59c640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 0x7f20300c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.472+0000 7f2025d9b640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 msgr2=0x7f20300c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:46:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.472+0000 7f2025d9b640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 msgr2=0x7f20300c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.472+0000 7f2025d9b640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 0x7f20300c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.472+0000 7f2025d9b640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 0x7f20300c9d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:46:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.472+0000 7f202e59c640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 msgr2=0x7f20300c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:46:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.472+0000 7f202e59c640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 msgr2=0x7f20300c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.472+0000 7f202e59c640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 0x7f20300c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.472+0000 7f202e59c640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 0x7f20300c9000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:46:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.873+0000 7f2025d9b640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 0x7f20300c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.873+0000 7f202e59c640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 0x7f20300c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.874+0000 7f2025d9b640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 msgr2=0x7f20300c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:46:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.874+0000 7f2025d9b640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 msgr2=0x7f20300c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.874+0000 7f2025d9b640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 0x7f20300c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.874+0000 7f2025d9b640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 0x7f20300c9d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:46:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.874+0000 7f202e59c640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 msgr2=0x7f20300c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:46:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.874+0000 7f202e59c640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 msgr2=0x7f20300c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.874+0000 7f202e59c640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 0x7f20300c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:37.874+0000 7f202e59c640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 0x7f20300c9000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:46:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:38.675+0000 7f2025d9b640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 0x7f20300c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:38.676+0000 7f202e59c640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 0x7f20300c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:38.676+0000 7f2025d9b640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 msgr2=0x7f20300c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:46:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:38.676+0000 7f2025d9b640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 msgr2=0x7f20300c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:38.676+0000 7f2025d9b640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 0x7f20300c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:38.676+0000 7f2025d9b640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 0x7f20300c9d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:46:38.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:38.676+0000 7f202e59c640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 msgr2=0x7f20300c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:46:38.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:38.676+0000 7f202e59c640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 msgr2=0x7f20300c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:46:38.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:38.676+0000 7f202e59c640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 0x7f20300c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:46:38.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:38.676+0000 7f202e59c640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 0x7f20300c9000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:46:39.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:46:39.188 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:46:39.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~108s 2023-11-28T17:46:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.197+0000 7f202dd9b640 1 -- 172.21.15.98:0/3126795537 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488434958 0 0) 0x7f20300cab90 con 0x7f20300ca250 2023-11-28T17:46:40.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.197+0000 7f202dd9b640 1 -- 172.21.15.98:0/3126795537 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f201c003520 con 0x7f20300ca250 2023-11-28T17:46:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.198+0000 7f202dd9b640 1 -- 172.21.15.98:0/3126795537 <== mon.1 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1262301514 0 0) 0x7f201c003520 con 0x7f20300ca250 2023-11-28T17:46:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.198+0000 7f202dd9b640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 msgr2=0x7f20300c9000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:46:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.198+0000 7f202dd9b640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.113:3300/0 conn(0x7f20301582e0 0x7f20300c9000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.198+0000 7f202dd9b640 1 -- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 msgr2=0x7f20300c9d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:46:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.198+0000 7f202dd9b640 1 --2- 172.21.15.98:0/3126795537 >> v2:172.21.15.98:3300/0 conn(0x7f20300c5f10 0x7f20300c9d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.198+0000 7f202dd9b640 1 -- 172.21.15.98:0/3126795537 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20300c6990 con 0x7f20300ca250 2023-11-28T17:46:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.198+0000 7f202dd9b640 1 -- 172.21.15.98:0/3126795537 <== mon.1 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f20280019f0 con 0x7f20300ca250 2023-11-28T17:46:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.198+0000 7f202dd9b640 1 -- 172.21.15.98:0/3126795537 >> v1:172.21.15.113:6789/0 conn(0x7f20300ca250 legacy=0x7f2030161680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.199+0000 7f202dd9b640 1 --2- 172.21.15.98:0/3126795537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f201c0039c0 0x7f201c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.199+0000 7f202dd9b640 1 --2- 172.21.15.98:0/3126795537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f201c0047c0 0x7f201c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.199+0000 7f202dd9b640 1 --2- 172.21.15.98:0/3126795537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c005180 0x7f201c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.199+0000 7f202dd9b640 1 -- 172.21.15.98:0/3126795537 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20300c7470 con 0x7f201c005180 2023-11-28T17:46:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.199+0000 7f202dd9b640 1 -- 172.21.15.98:0/3126795537 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20300c66b0 con 0x7f201c0047c0 2023-11-28T17:46:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.199+0000 7f202dd9b640 1 -- 172.21.15.98:0/3126795537 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f201c00c0d0 con 0x7f201c0039c0 2023-11-28T17:46:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.199+0000 7f202ed9d640 1 --2- 172.21.15.98:0/3126795537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c005180 0x7f201c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.199+0000 7f2025d9b640 1 --2- 172.21.15.98:0/3126795537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f201c0047c0 0x7f201c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.200+0000 7f202e59c640 1 --2- 172.21.15.98:0/3126795537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f201c0039c0 0x7f201c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.200+0000 7f202ed9d640 1 -- 172.21.15.98:0/3126795537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f201c0039c0 msgr2=0x7f201c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.200+0000 7f202ed9d640 1 --2- 172.21.15.98:0/3126795537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f201c0039c0 0x7f201c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.200+0000 7f202ed9d640 1 -- 172.21.15.98:0/3126795537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f201c0047c0 msgr2=0x7f201c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.200+0000 7f202ed9d640 1 --2- 172.21.15.98:0/3126795537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f201c0047c0 0x7f201c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.200+0000 7f202ed9d640 1 -- 172.21.15.98:0/3126795537 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201c003520 con 0x7f201c005180 2023-11-28T17:46:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.200+0000 7f202e59c640 1 --2- 172.21.15.98:0/3126795537 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f201c0039c0 0x7f201c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:46:40.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.202+0000 7f202ed9d640 1 -- 172.21.15.98:0/3126795537 reap_dead start 2023-11-28T17:46:40.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.202+0000 7f202ed9d640 1 --2- 172.21.15.98:0/3126795537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c005180 0x7f201c00b980 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f202800c4c0 tx=0x7f201c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237f73fbfb32228e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:40.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.204+0000 7f202dd9b640 1 -- 172.21.15.98:0/3126795537 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202802ccb0 con 0x7f201c005180 2023-11-28T17:46:40.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.205+0000 7f202dd9b640 1 -- 172.21.15.98:0/3126795537 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202802ce30 con 0x7f201c005180 2023-11-28T17:46:40.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.205+0000 7f202dd9b640 1 -- 172.21.15.98:0/3126795537 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2028034680 con 0x7f201c005180 2023-11-28T17:46:40.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.206+0000 7f2034fd5640 1 -- 172.21.15.98:0/3126795537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c005180 msgr2=0x7f201c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.206+0000 7f2034fd5640 1 --2- 172.21.15.98:0/3126795537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c005180 0x7f201c00b980 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f202800c4c0 tx=0x7f201c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.206+0000 7f2034fd5640 1 -- 172.21.15.98:0/3126795537 shutdown_connections 2023-11-28T17:46:40.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.206+0000 7f2034fd5640 1 --2- 172.21.15.98:0/3126795537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c005180 0x7f201c00b980 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.206+0000 7f2034fd5640 1 -- 172.21.15.98:0/3126795537 >> 172.21.15.98:0/3126795537 conn(0x7f203006a4d0 msgr2=0x7f203006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:40.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.206+0000 7f2034fd5640 1 -- 172.21.15.98:0/3126795537 shutdown_connections 2023-11-28T17:46:40.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.206+0000 7f2034fd5640 1 -- 172.21.15.98:0/3126795537 wait complete. 2023-11-28T17:46:40.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.210+0000 7f2034fd5640 1 Processor -- start 2023-11-28T17:46:40.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.210+0000 7f2034fd5640 1 -- start start 2023-11-28T17:46:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.211+0000 7f2034fd5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c005180 0x7f203014da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.212+0000 7f2034fd5640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f20301513a0 0x7f203006e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.212+0000 7f2034fd5640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f203006e540 0x7f2030147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.212+0000 7f2034fd5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20301f50b0 con 0x7f201c005180 2023-11-28T17:46:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.212+0000 7f2034fd5640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20301f4b50 con 0x7f203006e540 2023-11-28T17:46:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.212+0000 7f2034fd5640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203006e9c0 con 0x7f20301513a0 2023-11-28T17:46:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.212+0000 7f202e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c005180 0x7f203014da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.212+0000 7f202e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c005180 0x7f203014da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36690/0 (socket says 172.21.15.98:36690) 2023-11-28T17:46:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.212+0000 7f202e59c640 1 -- 172.21.15.98:0/3948786721 learned_addr learned my addr 172.21.15.98:0/3948786721 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:40.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.212+0000 7f2025d9b640 1 --2- 172.21.15.98:0/3948786721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f20301513a0 0x7f203006e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.212+0000 7f202ed9d640 1 --2- 172.21.15.98:0/3948786721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f203006e540 0x7f2030147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.213+0000 7f202e59c640 1 -- 172.21.15.98:0/3948786721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f20301513a0 msgr2=0x7f203006e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.213+0000 7f202e59c640 1 --2- 172.21.15.98:0/3948786721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f20301513a0 0x7f203006e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.213+0000 7f202e59c640 1 -- 172.21.15.98:0/3948786721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f203006e540 msgr2=0x7f2030147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.213+0000 7f202e59c640 1 --2- 172.21.15.98:0/3948786721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f203006e540 0x7f2030147320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.213+0000 7f202e59c640 1 -- 172.21.15.98:0/3948786721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20301f4dd0 con 0x7f201c005180 2023-11-28T17:46:40.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.213+0000 7f202e59c640 1 --2- 172.21.15.98:0/3948786721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c005180 0x7f203014da40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2020015a30 tx=0x7f2020015e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:40.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.213+0000 7f20277fe640 1 -- 172.21.15.98:0/3948786721 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020013ee0 con 0x7f201c005180 2023-11-28T17:46:40.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.213+0000 7f20277fe640 1 -- 172.21.15.98:0/3948786721 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2020011ab0 con 0x7f201c005180 2023-11-28T17:46:40.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.214+0000 7f20277fe640 1 -- 172.21.15.98:0/3948786721 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020011e60 con 0x7f201c005180 2023-11-28T17:46:40.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.214+0000 7f2034fd5640 1 -- 172.21.15.98:0/3948786721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203014cf60 con 0x7f201c005180 2023-11-28T17:46:40.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.214+0000 7f2034fd5640 1 -- 172.21.15.98:0/3948786721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f203014d520 con 0x7f201c005180 2023-11-28T17:46:40.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.215+0000 7f2034fd5640 1 -- 172.21.15.98:0/3948786721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203014d6e0 con 0x7f201c005180 2023-11-28T17:46:40.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.216+0000 7f20277fe640 1 -- 172.21.15.98:0/3948786721 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2020017440 con 0x7f201c005180 2023-11-28T17:46:40.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.217+0000 7f20277fe640 1 --2- 172.21.15.98:0/3948786721 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f201006b2d0 0x7f201006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.217+0000 7f20277fe640 1 -- 172.21.15.98:0/3948786721 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f202008a9a0 con 0x7f201c005180 2023-11-28T17:46:40.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.218+0000 7f2025d9b640 1 --2- 172.21.15.98:0/3948786721 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f201006b2d0 0x7f201006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.218+0000 7f2025d9b640 1 --2- 172.21.15.98:0/3948786721 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f201006b2d0 0x7f201006d790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f20300c4de0 tx=0x7f2018008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:40.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.222+0000 7f20277fe640 1 -- 172.21.15.98:0/3948786721 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f20200558d0 con 0x7f201c005180 2023-11-28T17:46:40.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.491+0000 7f2034fd5640 1 -- 172.21.15.98:0/3948786721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2030067bf0 con 0x7f201c005180 2023-11-28T17:46:40.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.492+0000 7f20277fe640 1 -- 172.21.15.98:0/3948786721 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7f2020027080 con 0x7f201c005180 2023-11-28T17:46:40.494 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:46:40.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.494+0000 7f2034fd5640 1 -- 172.21.15.98:0/3948786721 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f201006b2d0 msgr2=0x7f201006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.494+0000 7f2034fd5640 1 --2- 172.21.15.98:0/3948786721 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f201006b2d0 0x7f201006d790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f20300c4de0 tx=0x7f2018008040 comp rx=0 tx=0).stop 2023-11-28T17:46:40.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.494+0000 7f2034fd5640 1 -- 172.21.15.98:0/3948786721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c005180 msgr2=0x7f203014da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.494+0000 7f2034fd5640 1 --2- 172.21.15.98:0/3948786721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c005180 0x7f203014da40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2020015a30 tx=0x7f2020015e30 comp rx=0 tx=0).stop 2023-11-28T17:46:40.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.495+0000 7f2034fd5640 1 -- 172.21.15.98:0/3948786721 shutdown_connections 2023-11-28T17:46:40.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.495+0000 7f2034fd5640 1 --2- 172.21.15.98:0/3948786721 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f201006b2d0 0x7f201006d790 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.495+0000 7f2034fd5640 1 --2- 172.21.15.98:0/3948786721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f203006e540 0x7f2030147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.495+0000 7f2034fd5640 1 --2- 172.21.15.98:0/3948786721 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f20301513a0 0x7f203006e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.495+0000 7f2034fd5640 1 --2- 172.21.15.98:0/3948786721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c005180 0x7f203014da40 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.495+0000 7f2034fd5640 1 -- 172.21.15.98:0/3948786721 >> 172.21.15.98:0/3948786721 conn(0x7f203006a4d0 msgr2=0x7f203015d120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:40.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.495+0000 7f2034fd5640 1 -- 172.21.15.98:0/3948786721 shutdown_connections 2023-11-28T17:46:40.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.495+0000 7f2034fd5640 1 -- 172.21.15.98:0/3948786721 wait complete. 2023-11-28T17:46:40.516 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:46:40.517 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:46:40.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:46:40.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.727+0000 7f9e2a6ee640 1 Processor -- start 2023-11-28T17:46:40.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.727+0000 7f9e2a6ee640 1 -- start start 2023-11-28T17:46:40.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.728+0000 7f9e2a6ee640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9e240c4380 0x7f9e240c2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.728+0000 7f9e2a6ee640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e240c4760 0x7f9e240c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.728+0000 7f9e2a6ee640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e2415a0f0 0x7f9e2415a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.728+0000 7f9e2a6ee640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e240c2a60 con 0x7f9e240c4760 2023-11-28T17:46:40.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.728+0000 7f9e2a6ee640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e240c6560 con 0x7f9e240c4380 2023-11-28T17:46:40.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.728+0000 7f9e2a6ee640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2415aaa0 con 0x7f9e2415a0f0 2023-11-28T17:46:40.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.729+0000 7f9e1bfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e240c4760 0x7f9e240c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.729+0000 7f9e1bfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e240c4760 0x7f9e240c6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36696/0 (socket says 172.21.15.98:36696) 2023-11-28T17:46:40.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.729+0000 7f9e1bfff640 1 -- 172.21.15.98:0/1607382566 learned_addr learned my addr 172.21.15.98:0/1607382566 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:40.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.729+0000 7f9e1bfff640 1 -- 172.21.15.98:0/1607382566 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e2415a0f0 msgr2=0x7f9e2415a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:40.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.729+0000 7f9e1bfff640 1 --2- 172.21.15.98:0/1607382566 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e2415a0f0 0x7f9e2415a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.729+0000 7f9e1bfff640 1 -- 172.21.15.98:0/1607382566 >> v2:172.21.15.113:3300/0 conn(0x7f9e240c4380 msgr2=0x7f9e240c2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:40.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.729+0000 7f9e1bfff640 1 --2- 172.21.15.98:0/1607382566 >> v2:172.21.15.113:3300/0 conn(0x7f9e240c4380 0x7f9e240c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.729+0000 7f9e1bfff640 1 -- 172.21.15.98:0/1607382566 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e24156980 con 0x7f9e240c4760 2023-11-28T17:46:40.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.730+0000 7f9e1bfff640 1 --2- 172.21.15.98:0/1607382566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e240c4760 0x7f9e240c6020 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9e24070840 tx=0x7f9e0c02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29bb8ca589300672 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:40.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.731+0000 7f9e237fe640 1 -- 172.21.15.98:0/1607382566 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c030850 con 0x7f9e240c4760 2023-11-28T17:46:40.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.731+0000 7f9e237fe640 1 -- 172.21.15.98:0/1607382566 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0c0309b0 con 0x7f9e240c4760 2023-11-28T17:46:40.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.731+0000 7f9e237fe640 1 -- 172.21.15.98:0/1607382566 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c030d40 con 0x7f9e240c4760 2023-11-28T17:46:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.731+0000 7f9e2a6ee640 1 -- 172.21.15.98:0/1607382566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e240c4760 msgr2=0x7f9e240c6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.731+0000 7f9e2a6ee640 1 --2- 172.21.15.98:0/1607382566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e240c4760 0x7f9e240c6020 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9e24070840 tx=0x7f9e0c02b990 comp rx=0 tx=0).stop 2023-11-28T17:46:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.731+0000 7f9e2a6ee640 1 -- 172.21.15.98:0/1607382566 shutdown_connections 2023-11-28T17:46:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.731+0000 7f9e2a6ee640 1 --2- 172.21.15.98:0/1607382566 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e2415a0f0 0x7f9e2415a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.731+0000 7f9e2a6ee640 1 --2- 172.21.15.98:0/1607382566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e240c4760 0x7f9e240c6020 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.731+0000 7f9e2a6ee640 1 --2- 172.21.15.98:0/1607382566 >> v2:172.21.15.113:3300/0 conn(0x7f9e240c4380 0x7f9e240c2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.731+0000 7f9e2a6ee640 1 -- 172.21.15.98:0/1607382566 >> 172.21.15.98:0/1607382566 conn(0x7f9e2406c0f0 msgr2=0x7f9e2406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.731+0000 7f9e2a6ee640 1 -- 172.21.15.98:0/1607382566 shutdown_connections 2023-11-28T17:46:40.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.732+0000 7f9e2a6ee640 1 -- 172.21.15.98:0/1607382566 wait complete. 2023-11-28T17:46:40.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.733+0000 7f9e2a6ee640 1 Processor -- start 2023-11-28T17:46:40.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.733+0000 7f9e2a6ee640 1 -- start start 2023-11-28T17:46:40.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.734+0000 7f9e2a6ee640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e240c4380 0x7f9e24156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.734+0000 7f9e2a6ee640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e240c4760 0x7f9e2414fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.734+0000 7f9e2a6ee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e2415a0f0 0x7f9e24143b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.734+0000 7f9e2a6ee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24156760 con 0x7f9e2415a0f0 2023-11-28T17:46:40.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.734+0000 7f9e2a6ee640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e241f28f0 con 0x7f9e240c4380 2023-11-28T17:46:40.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.734+0000 7f9e2a6ee640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2414e560 con 0x7f9e240c4760 2023-11-28T17:46:40.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.734+0000 7f9e28c8b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e2415a0f0 0x7f9e24143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.735+0000 7f9e28c8b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e2415a0f0 0x7f9e24143b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36698/0 (socket says 172.21.15.98:36698) 2023-11-28T17:46:40.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.735+0000 7f9e28c8b640 1 -- 172.21.15.98:0/2971842505 learned_addr learned my addr 172.21.15.98:0/2971842505 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:40.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.735+0000 7f9e28c8b640 1 -- 172.21.15.98:0/2971842505 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e240c4760 msgr2=0x7f9e2414fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.735+0000 7f9e28c8b640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9e240c4760 0x7f9e2414fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.735+0000 7f9e28c8b640 1 -- 172.21.15.98:0/2971842505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e240c4380 msgr2=0x7f9e24156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.735+0000 7f9e28c8b640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e240c4380 0x7f9e24156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.735+0000 7f9e28c8b640 1 -- 172.21.15.98:0/2971842505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0c0094d0 con 0x7f9e2415a0f0 2023-11-28T17:46:40.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.735+0000 7f9e28c8b640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e2415a0f0 0x7f9e24143b10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00b970 tx=0x7f9e1c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:40.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.735+0000 7f9e217fa640 1 -- 172.21.15.98:0/2971842505 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c00cb40 con 0x7f9e2415a0f0 2023-11-28T17:46:40.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.735+0000 7f9e217fa640 1 -- 172.21.15.98:0/2971842505 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c00ccc0 con 0x7f9e2415a0f0 2023-11-28T17:46:40.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.735+0000 7f9e217fa640 1 -- 172.21.15.98:0/2971842505 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c014680 con 0x7f9e2415a0f0 2023-11-28T17:46:40.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.736+0000 7f9e2a6ee640 1 -- 172.21.15.98:0/2971842505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e24145dc0 con 0x7f9e2415a0f0 2023-11-28T17:46:40.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.736+0000 7f9e2a6ee640 1 -- 172.21.15.98:0/2971842505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e24146380 con 0x7f9e2415a0f0 2023-11-28T17:46:40.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.737+0000 7f9e2a6ee640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e24154650 0x7f9e24144050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.737+0000 7f9e2a6ee640 1 -- 172.21.15.98:0/2971842505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e240744d0 con 0x7f9e24154650 2023-11-28T17:46:40.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.737+0000 7f9e23fff640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e24154650 0x7f9e24144050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.737+0000 7f9e23fff640 1 -- 172.21.15.98:0/2971842505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e24157200 con 0x7f9e24154650 2023-11-28T17:46:40.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.738+0000 7f9e23fff640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e24154650 0x7f9e24144050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400b340 tx=0x7f9e1400b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:40.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.738+0000 7f9e217fa640 1 -- 172.21.15.98:0/2971842505 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f9e1c007370 con 0x7f9e2415a0f0 2023-11-28T17:46:40.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.739+0000 7f9e217fa640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9e0406b2d0 0x7f9e0406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.739+0000 7f9e217fa640 1 -- 172.21.15.98:0/2971842505 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9e1c089bc0 con 0x7f9e2415a0f0 2023-11-28T17:46:40.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.739+0000 7f9e1bfff640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9e0406b2d0 0x7f9e0406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.739+0000 7f9e217fa640 1 -- 172.21.15.98:0/2971842505 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e14011830 con 0x7f9e24154650 2023-11-28T17:46:40.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.739+0000 7f9e217fa640 1 -- 172.21.15.98:0/2971842505 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f9e14011990 con 0x7f9e24154650 2023-11-28T17:46:40.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.739+0000 7f9e217fa640 1 -- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e24154650 msgr2=0x7f9e24144050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.739+0000 7f9e217fa640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e24154650 0x7f9e24144050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400b340 tx=0x7f9e1400b800 comp rx=0 tx=0).stop 2023-11-28T17:46:40.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.739+0000 7f9e217fa640 1 -- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e24154650 msgr2=0x7f9e24144050 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:40.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.739+0000 7f9e217fa640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e24154650 0x7f9e24144050 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.740+0000 7f9e1bfff640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9e0406b2d0 0x7f9e0406d790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c036000 tx=0x7f9e0c03b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:40.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.746+0000 7f9e2a6ee640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e241f2d80 0x7f9e241f3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.746+0000 7f9e2a6ee640 1 -- 172.21.15.98:0/2971842505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24156f20 con 0x7f9e241f2d80 2023-11-28T17:46:40.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.746+0000 7f9e23fff640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e241f2d80 0x7f9e241f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.747+0000 7f9e23fff640 1 -- 172.21.15.98:0/2971842505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9e241574e0 con 0x7f9e241f2d80 2023-11-28T17:46:40.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.747+0000 7f9e23fff640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e241f2d80 0x7f9e241f3190 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400b340 tx=0x7f9e14004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:40.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.747+0000 7f9e217fa640 1 -- 172.21.15.98:0/2971842505 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e14025680 con 0x7f9e241f2d80 2023-11-28T17:46:40.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.748+0000 7f9e217fa640 1 -- 172.21.15.98:0/2971842505 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4637 (secure 0 0 0) 0x7f9e1401c2e0 con 0x7f9e241f2d80 2023-11-28T17:46:40.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.748+0000 7f9e217fa640 1 -- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e241f2d80 msgr2=0x7f9e241f3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.748+0000 7f9e217fa640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e241f2d80 0x7f9e241f3190 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400b340 tx=0x7f9e14004840 comp rx=0 tx=0).stop 2023-11-28T17:46:40.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.748+0000 7f9e217fa640 1 -- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e241f2d80 msgr2=0x7f9e241f3190 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:40.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.748+0000 7f9e217fa640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e241f2d80 0x7f9e241f3190 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.750 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:46:40.750 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:46:40.750 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:46:40.750 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:46:40.751 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 60, 2023-11-28T17:46:40.751 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:46:40.751 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:46:40.751 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:46:40.751 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:46:40.751 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:40.751 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:46:40.751 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:40.751 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:46:40.751 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:46:40.751 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:40.751 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:40.752 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:40.752 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:40.752 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:40.752 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:40.752 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:40.752 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:40.752 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:40.752 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:40.752 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:40.752 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:46:40.752 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:46:40.752 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:40.752 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:40.753 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:40.753 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:40.753 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:40.753 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:40.753 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:40.753 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:40.753 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:40.753 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:40.753 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:40.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.753 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:46:40.753 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:46:40.753 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:46:40.754 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:46:40.754 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:46:40.754 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:46:40.754 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:40.754 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:40.754 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:46:40.754 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:46:40.754 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:46:40.754 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:46:40.754 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:46:40.754 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:46:40.754 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:46:40.755 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:40.755 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:46:40.755 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:40.755 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:40.755 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:40.755 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:40.755 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:40.755 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:40.755 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:40.755 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:40.755 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:40.755 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:40.756 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:40.756 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:46:40.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.756 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:46:40.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.756 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:46:40.756 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:46:40.756 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:40.756 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:40.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.756 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:40.756 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:46:40.756 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:40.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:40.757 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:46:40.757 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:40.757 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:40.757 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:40.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.757 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:46:40.757 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:46:40.757 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:40.757 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:40.758 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:40.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.758 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:46:40.758 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:46:40.758 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:40.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:40.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:40.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:46:40.758 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:40.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.759 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:40.759 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:46:40.759 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:40.759 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:40.759 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:40.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.759 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:46:40.759 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:46:40.759 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:40.759 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:40.760 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:40.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.760 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:46:40.760 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:46:40.760 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:40.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:40.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:40.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:46:40.760 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:40.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:40.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:46:40.761 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:40.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:40.761 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:40.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:46:40.761 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:46:40.761 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:40.761 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:40.761 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:40.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:40.762 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:40.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.762 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:46:40.762 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:46:40.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.762 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:40.762 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:40.762 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:46:40.762 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:40.762 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:40.762 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:46:40.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.763 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:40.763 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:40.763 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:46:40.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:40.763 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:40.763 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:46:40.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.763 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:40.763 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:40.763 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:46:40.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:40.763 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:40.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.764 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:46:40.764 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:46:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.748+0000 7f9e2a6ee640 1 -- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9e0406b2d0 msgr2=0x7f9e0406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.748+0000 7f9e2a6ee640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9e0406b2d0 0x7f9e0406d790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c036000 tx=0x7f9e0c03b040 comp rx=0 tx=0).stop 2023-11-28T17:46:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.748+0000 7f9e2a6ee640 1 -- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e2415a0f0 msgr2=0x7f9e24143b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.748+0000 7f9e2a6ee640 1 --2- 172.21.15.98:0/2971842505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e2415a0f0 0x7f9e24143b10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00b970 tx=0x7f9e1c00be30 comp rx=0 tx=0).stop 2023-11-28T17:46:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.748+0000 7f9e28c8b640 1 -- 172.21.15.98:0/2971842505 reap_dead start 2023-11-28T17:46:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.748+0000 7f9e28c8b640 1 -- 172.21.15.98:0/2971842505 reap_dead start 2023-11-28T17:46:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.748+0000 7f9e2a6ee640 1 -- 172.21.15.98:0/2971842505 shutdown_connections 2023-11-28T17:46:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.748+0000 7f9e2a6ee640 1 -- 172.21.15.98:0/2971842505 >> 172.21.15.98:0/2971842505 conn(0x7f9e2406c0f0 msgr2=0x7f9e2406dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.748+0000 7f9e2a6ee640 1 -- 172.21.15.98:0/2971842505 shutdown_connections 2023-11-28T17:46:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.748+0000 7f9e2a6ee640 1 -- 172.21.15.98:0/2971842505 wait complete. 2023-11-28T17:46:40.766 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:46:40.766 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:46:40.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:46:40.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.963+0000 7fe6fc333640 1 Processor -- start 2023-11-28T17:46:40.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.963+0000 7fe6fc333640 1 -- start start 2023-11-28T17:46:40.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.965+0000 7fe6fc333640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6f4151be0 0x7fe6f40c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.965+0000 7fe6fc333640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6f40c5a60 0x7fe6f41584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.965+0000 7fe6fc333640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fe6f40c4730 0x7fe6f40c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.965+0000 7fe6fc333640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f4158a10 con 0x7fe6f4151be0 2023-11-28T17:46:40.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.965+0000 7fe6fc333640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f40c8150 con 0x7fe6f40c4730 2023-11-28T17:46:40.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.965+0000 7fe6fc333640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f40c8850 con 0x7fe6f40c5a60 2023-11-28T17:46:40.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.965+0000 7fe6fa0cf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6f4151be0 0x7fe6f40c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.966+0000 7fe6fa0cf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6f4151be0 0x7fe6f40c75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36726/0 (socket says 172.21.15.98:36726) 2023-11-28T17:46:40.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.966+0000 7fe6fa0cf640 1 -- 172.21.15.98:0/468054932 learned_addr learned my addr 172.21.15.98:0/468054932 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:40.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.966+0000 7fe6f98ce640 1 --2- 172.21.15.98:0/468054932 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6f40c5a60 0x7fe6f41584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.966+0000 7fe6fa8d0640 1 --2- 172.21.15.98:0/468054932 >> v2:172.21.15.113:3300/0 conn(0x7fe6f40c4730 0x7fe6f40c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.966+0000 7fe6fa0cf640 1 -- 172.21.15.98:0/468054932 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6f40c5a60 msgr2=0x7fe6f41584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:40.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.966+0000 7fe6fa0cf640 1 --2- 172.21.15.98:0/468054932 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6f40c5a60 0x7fe6f41584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.966+0000 7fe6fa0cf640 1 -- 172.21.15.98:0/468054932 >> v2:172.21.15.113:3300/0 conn(0x7fe6f40c4730 msgr2=0x7fe6f40c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:40.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.966+0000 7fe6fa0cf640 1 --2- 172.21.15.98:0/468054932 >> v2:172.21.15.113:3300/0 conn(0x7fe6f40c4730 0x7fe6f40c4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.966+0000 7fe6fa0cf640 1 -- 172.21.15.98:0/468054932 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f40c5f50 con 0x7fe6f4151be0 2023-11-28T17:46:40.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.967+0000 7fe6fa0cf640 1 --2- 172.21.15.98:0/468054932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f4151be0 0x7fe6f40c75a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4070240 tx=0x7fe6f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c992a67a4f97ec84 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:40.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.967+0000 7fe6f90cd640 1 -- 172.21.15.98:0/468054932 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0031870 con 0x7fe6f4151be0 2023-11-28T17:46:40.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.968+0000 7fe6f90cd640 1 -- 172.21.15.98:0/468054932 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f00319d0 con 0x7fe6f4151be0 2023-11-28T17:46:40.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.968+0000 7fe6f90cd640 1 -- 172.21.15.98:0/468054932 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0031d60 con 0x7fe6f4151be0 2023-11-28T17:46:40.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.968+0000 7fe6fc333640 1 -- 172.21.15.98:0/468054932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f4151be0 msgr2=0x7fe6f40c75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.968+0000 7fe6fc333640 1 --2- 172.21.15.98:0/468054932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f4151be0 0x7fe6f40c75a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4070240 tx=0x7fe6f002b910 comp rx=0 tx=0).stop 2023-11-28T17:46:40.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.968+0000 7fe6fc333640 1 -- 172.21.15.98:0/468054932 shutdown_connections 2023-11-28T17:46:40.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.968+0000 7fe6fc333640 1 --2- 172.21.15.98:0/468054932 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6f40c5a60 0x7fe6f41584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.968+0000 7fe6fc333640 1 --2- 172.21.15.98:0/468054932 >> v2:172.21.15.113:3300/0 conn(0x7fe6f40c4730 0x7fe6f40c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.968+0000 7fe6fc333640 1 --2- 172.21.15.98:0/468054932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f4151be0 0x7fe6f40c75a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.968+0000 7fe6fc333640 1 -- 172.21.15.98:0/468054932 >> 172.21.15.98:0/468054932 conn(0x7fe6f406c0f0 msgr2=0x7fe6f406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:40.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.968+0000 7fe6fc333640 1 -- 172.21.15.98:0/468054932 shutdown_connections 2023-11-28T17:46:40.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.968+0000 7fe6fc333640 1 -- 172.21.15.98:0/468054932 wait complete. 2023-11-28T17:46:40.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.970+0000 7fe6fc333640 1 Processor -- start 2023-11-28T17:46:40.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.970+0000 7fe6fc333640 1 -- start start 2023-11-28T17:46:40.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.971+0000 7fe6fc333640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6f40c4730 0x7fe6f4152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.971+0000 7fe6fc333640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f40c5a60 0x7fe6f415ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.971+0000 7fe6fc333640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f4151be0 0x7fe6f41ed2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.971+0000 7fe6fc333640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f415a9f0 con 0x7fe6f4151be0 2023-11-28T17:46:40.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.971+0000 7fe6fc333640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f415a150 con 0x7fe6f40c5a60 2023-11-28T17:46:40.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.971+0000 7fe6fc333640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f415a430 con 0x7fe6f40c4730 2023-11-28T17:46:40.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.972+0000 7fe6fa8d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f4151be0 0x7fe6f41ed2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.972+0000 7fe6fa8d0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f4151be0 0x7fe6f41ed2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36740/0 (socket says 172.21.15.98:36740) 2023-11-28T17:46:40.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.972+0000 7fe6fa8d0640 1 -- 172.21.15.98:0/1929270264 learned_addr learned my addr 172.21.15.98:0/1929270264 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:40.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.972+0000 7fe6fa0cf640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6f40c4730 0x7fe6f4152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.972+0000 7fe6fa8d0640 1 -- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6f40c4730 msgr2=0x7fe6f4152390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.972+0000 7fe6fa8d0640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe6f40c4730 0x7fe6f4152390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.972+0000 7fe6fa8d0640 1 -- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f40c5a60 msgr2=0x7fe6f415ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.972+0000 7fe6f98ce640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f40c5a60 0x7fe6f415ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.972+0000 7fe6fa8d0640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f40c5a60 0x7fe6f415ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.972+0000 7fe6fa8d0640 1 -- 172.21.15.98:0/1929270264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f00094d0 con 0x7fe6f4151be0 2023-11-28T17:46:40.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.972+0000 7fe6fa8d0640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f4151be0 0x7fe6f41ed2d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec007e60 tx=0x7fe6ec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:40.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.973+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1929270264 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec014620 con 0x7fe6f4151be0 2023-11-28T17:46:40.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.973+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1929270264 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ec00ee90 con 0x7fe6f4151be0 2023-11-28T17:46:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.973+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1929270264 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec0149e0 con 0x7fe6f4151be0 2023-11-28T17:46:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.973+0000 7fe6fc333640 1 -- 172.21.15.98:0/1929270264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f415a710 con 0x7fe6f4151be0 2023-11-28T17:46:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.973+0000 7fe6fc333640 1 -- 172.21.15.98:0/1929270264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f415b570 con 0x7fe6f4151be0 2023-11-28T17:46:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.974+0000 7fe6fc333640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f4159c10 0x7fe6f406a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.974+0000 7fe6fc333640 1 -- 172.21.15.98:0/1929270264 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f40744d0 con 0x7fe6f4159c10 2023-11-28T17:46:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.974+0000 7fe6fa0cf640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f4159c10 0x7fe6f406a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.975+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1929270264 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe6ec014b40 con 0x7fe6f4151be0 2023-11-28T17:46:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.975+0000 7fe6fa0cf640 1 -- 172.21.15.98:0/1929270264 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6f40c7e70 con 0x7fe6f4159c10 2023-11-28T17:46:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.975+0000 7fe6fa0cf640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f4159c10 0x7fe6f406a740 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe6f40c5330 tx=0x7fe6f0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.976+0000 7fe6eaffd640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe6d806b2a0 0x7fe6d806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.976+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1929270264 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe6ec08a540 con 0x7fe6f4151be0 2023-11-28T17:46:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.976+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1929270264 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f003aa00 con 0x7fe6f4159c10 2023-11-28T17:46:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.976+0000 7fe6f98ce640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe6d806b2a0 0x7fe6d806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.977+0000 7fe6f98ce640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe6d806b2a0 0x7fe6d806d760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4155520 tx=0x7fe6e4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.977+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1929270264 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fe6f003ab60 con 0x7fe6f4159c10 2023-11-28T17:46:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.977+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f4159c10 msgr2=0x7fe6f406a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.977+0000 7fe6eaffd640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f4159c10 0x7fe6f406a740 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe6f40c5330 tx=0x7fe6f0004730 comp rx=0 tx=0).stop 2023-11-28T17:46:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.977+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f4159c10 msgr2=0x7fe6f406a740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.977+0000 7fe6eaffd640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f4159c10 0x7fe6f406a740 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.984+0000 7fe6fc333640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f414c640 0x7fe6f41468a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:40.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.984+0000 7fe6fc333640 1 -- 172.21.15.98:0/1929270264 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f415a8a0 con 0x7fe6f414c640 2023-11-28T17:46:40.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.985+0000 7fe6fa0cf640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f414c640 0x7fe6f41468a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:40.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.985+0000 7fe6fa0cf640 1 -- 172.21.15.98:0/1929270264 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe6f4151fc0 con 0x7fe6f414c640 2023-11-28T17:46:40.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.986+0000 7fe6fa0cf640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f414c640 0x7fe6f41468a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00090b0 tx=0x7fe6f00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:40.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.986+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1929270264 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f004cad0 con 0x7fe6f414c640 2023-11-28T17:46:40.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.987+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1929270264 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4465 (secure 0 0 0) 0x7fe6f004cc30 con 0x7fe6f414c640 2023-11-28T17:46:40.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.987+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f414c640 msgr2=0x7fe6f41468a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:40.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.987+0000 7fe6eaffd640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f414c640 0x7fe6f41468a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00090b0 tx=0x7fe6f00046c0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.987+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f414c640 msgr2=0x7fe6f41468a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:40.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.987+0000 7fe6eaffd640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6f414c640 0x7fe6f41468a0 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00090b0 tx=0x7fe6f00046c0 comp rx=0 tx=0).stop 2023-11-28T17:46:40.989 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:46:40.989 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:46:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:46:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:46:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 60, 2023-11-28T17:46:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:46:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:46:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:46:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:46:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:46:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:46:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:46:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:46:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:46:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:46:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:46:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:46:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:46:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:46:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:46:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:46:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:46:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:46:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:46:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:46:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:46:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:46:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:46:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:46:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:46:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:46:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:46:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:46:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:46:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:46:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:46:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:46:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:46:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:46:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:46:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:46:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:46:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:46:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:46:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:46:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:46:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:46:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:46:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:46:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:46:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:46:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:46:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:46:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:46:41.002 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:46:41.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.987+0000 7fe6fc333640 1 -- 172.21.15.98:0/1929270264 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe6d806b2a0 msgr2=0x7fe6d806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.987+0000 7fe6fc333640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe6d806b2a0 0x7fe6d806d760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4155520 tx=0x7fe6e4008040 comp rx=0 tx=0).stop 2023-11-28T17:46:41.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.987+0000 7fe6fc333640 1 -- 172.21.15.98:0/1929270264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f4151be0 msgr2=0x7fe6f41ed2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.987+0000 7fe6fc333640 1 --2- 172.21.15.98:0/1929270264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6f4151be0 0x7fe6f41ed2d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec007e60 tx=0x7fe6ec00e4b0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.987+0000 7fe6fa8d0640 1 -- 172.21.15.98:0/1929270264 reap_dead start 2023-11-28T17:46:41.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.987+0000 7fe6fa8d0640 1 -- 172.21.15.98:0/1929270264 reap_dead start 2023-11-28T17:46:41.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.988+0000 7fe6fc333640 1 -- 172.21.15.98:0/1929270264 shutdown_connections 2023-11-28T17:46:41.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.988+0000 7fe6fc333640 1 -- 172.21.15.98:0/1929270264 >> 172.21.15.98:0/1929270264 conn(0x7fe6f406c0f0 msgr2=0x7fe6f40c9880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:41.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.988+0000 7fe6fc333640 1 -- 172.21.15.98:0/1929270264 shutdown_connections 2023-11-28T17:46:41.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:40.988+0000 7fe6fc333640 1 -- 172.21.15.98:0/1929270264 wait complete. 2023-11-28T17:46:41.006 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:46:41.006 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:46:41.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:46:41.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.216+0000 7f613eb73640 1 Processor -- start 2023-11-28T17:46:41.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.216+0000 7f613eb73640 1 -- start start 2023-11-28T17:46:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.218+0000 7f613eb73640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f61380c4380 0x7f61380c2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.218+0000 7f613eb73640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f613815a0f0 0x7f613815a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.218+0000 7f613eb73640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613815aaa0 con 0x7f61380c4760 2023-11-28T17:46:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.218+0000 7f613eb73640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61380c2a60 con 0x7f61380c4380 2023-11-28T17:46:41.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.218+0000 7f613eb73640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6138156980 con 0x7f613815a0f0 2023-11-28T17:46:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.218+0000 7f613c90f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f61380c4380 0x7f61380c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.218+0000 7f6137fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f61380c4760 0x7f61380c6020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51708/0 (socket says 172.21.15.98:51708) 2023-11-28T17:46:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.218+0000 7f6137fff640 1 -- 172.21.15.98:0/4192161829 learned_addr learned my addr 172.21.15.98:0/4192161829 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:46:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.218+0000 7f613d110640 1 --2- 172.21.15.98:0/4192161829 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f613815a0f0 0x7f613815a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.218+0000 7f613c90f640 1 -- 172.21.15.98:0/4192161829 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f613815a0f0 msgr2=0x7f613815a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.218+0000 7f613c90f640 1 --2- 172.21.15.98:0/4192161829 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f613815a0f0 0x7f613815a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.218+0000 7f613c90f640 1 -- 172.21.15.98:0/4192161829 >> v1:172.21.15.98:6789/0 conn(0x7f61380c4760 legacy=0x7f61380c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.219+0000 7f613c90f640 1 -- 172.21.15.98:0/4192161829 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138156c40 con 0x7f61380c4380 2023-11-28T17:46:41.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.219+0000 7f61377fe640 1 -- 172.21.15.98:0/4192161829 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572501820 0 0) 0x7f613815aaa0 con 0x7f61380c4760 2023-11-28T17:46:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.220+0000 7f613c90f640 1 --2- 172.21.15.98:0/4192161829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61380c4380 0x7f61380c2520 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f612c0097d0 tx=0x7f612c02b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb9049648d467222 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.220+0000 7f61377fe640 1 -- 172.21.15.98:0/4192161829 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612c031890 con 0x7f61380c4380 2023-11-28T17:46:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.220+0000 7f61377fe640 1 -- 172.21.15.98:0/4192161829 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612c0319f0 con 0x7f61380c4380 2023-11-28T17:46:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.220+0000 7f61377fe640 1 -- 172.21.15.98:0/4192161829 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612c031d80 con 0x7f61380c4380 2023-11-28T17:46:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.220+0000 7f613eb73640 1 -- 172.21.15.98:0/4192161829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61380c4380 msgr2=0x7f61380c2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.220+0000 7f613eb73640 1 --2- 172.21.15.98:0/4192161829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61380c4380 0x7f61380c2520 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f612c0097d0 tx=0x7f612c02b930 comp rx=0 tx=0).stop 2023-11-28T17:46:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.221+0000 7f613eb73640 1 -- 172.21.15.98:0/4192161829 shutdown_connections 2023-11-28T17:46:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.221+0000 7f613eb73640 1 --2- 172.21.15.98:0/4192161829 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f613815a0f0 0x7f613815a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.221+0000 7f613eb73640 1 --2- 172.21.15.98:0/4192161829 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61380c4380 0x7f61380c2520 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.221+0000 7f613eb73640 1 -- 172.21.15.98:0/4192161829 >> 172.21.15.98:0/4192161829 conn(0x7f613806c0f0 msgr2=0x7f613806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.221+0000 7f613eb73640 1 -- 172.21.15.98:0/4192161829 shutdown_connections 2023-11-28T17:46:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.221+0000 7f613eb73640 1 -- 172.21.15.98:0/4192161829 wait complete. 2023-11-28T17:46:41.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.222+0000 7f613eb73640 1 Processor -- start 2023-11-28T17:46:41.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.223+0000 7f613eb73640 1 -- start start 2023-11-28T17:46:41.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.223+0000 7f613eb73640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61380c4380 0x7f6138158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.223+0000 7f613eb73640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61380c4760 0x7f61381559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.223+0000 7f613eb73640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f613815a0f0 0x7f61380c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.224+0000 7f613eb73640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61381f3830 con 0x7f61380c4380 2023-11-28T17:46:41.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.224+0000 7f613eb73640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61381ef310 con 0x7f61380c4760 2023-11-28T17:46:41.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.224+0000 7f613eb73640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61381efb70 con 0x7f613815a0f0 2023-11-28T17:46:41.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.224+0000 7f613c90f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61380c4380 0x7f6138158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:41.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.224+0000 7f613c90f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61380c4380 0x7f6138158a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36766/0 (socket says 172.21.15.98:36766) 2023-11-28T17:46:41.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.224+0000 7f613c90f640 1 -- 172.21.15.98:0/2541775068 learned_addr learned my addr 172.21.15.98:0/2541775068 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:41.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.224+0000 7f613c90f640 1 -- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f613815a0f0 msgr2=0x7f61380c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.224+0000 7f613c90f640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f613815a0f0 0x7f61380c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.224+0000 7f613c90f640 1 -- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61380c4760 msgr2=0x7f61381559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.224+0000 7f613c90f640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f61380c4760 0x7f61381559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.224+0000 7f613c90f640 1 -- 172.21.15.98:0/2541775068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c009400 con 0x7f61380c4380 2023-11-28T17:46:41.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.224+0000 7f613c90f640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61380c4380 0x7f6138158a30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f612c0098f0 tx=0x7f612c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:41.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.225+0000 7f61357fa640 1 -- 172.21.15.98:0/2541775068 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612c03be00 con 0x7f61380c4380 2023-11-28T17:46:41.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.225+0000 7f61357fa640 1 -- 172.21.15.98:0/2541775068 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612c0453f0 con 0x7f61380c4380 2023-11-28T17:46:41.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.225+0000 7f61357fa640 1 -- 172.21.15.98:0/2541775068 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612c0457a0 con 0x7f61380c4380 2023-11-28T17:46:41.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.225+0000 7f613eb73640 1 -- 172.21.15.98:0/2541775068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6138159be0 con 0x7f61380c4380 2023-11-28T17:46:41.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.225+0000 7f613eb73640 1 -- 172.21.15.98:0/2541775068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61381ef520 con 0x7f61380c4380 2023-11-28T17:46:41.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.226+0000 7f613eb73640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61381ef6b0 0x7f61380ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.226+0000 7f613eb73640 1 -- 172.21.15.98:0/2541775068 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61380744d0 con 0x7f61381ef6b0 2023-11-28T17:46:41.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.226+0000 7f6137fff640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61381ef6b0 0x7f61380ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:41.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.227+0000 7f61357fa640 1 -- 172.21.15.98:0/2541775068 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f612c04e450 con 0x7f61380c4380 2023-11-28T17:46:41.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.227+0000 7f6137fff640 1 -- 172.21.15.98:0/2541775068 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61381f3340 con 0x7f61381ef6b0 2023-11-28T17:46:41.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.228+0000 7f6137fff640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61381ef6b0 0x7f61380ca450 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f61240011e0 tx=0x7f612400c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:41.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.228+0000 7f61357fa640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f611c06b2a0 0x7f611c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.228+0000 7f61357fa640 1 -- 172.21.15.98:0/2541775068 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f612c0ac460 con 0x7f61380c4380 2023-11-28T17:46:41.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.228+0000 7f613d110640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f611c06b2a0 0x7f611c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:41.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.228+0000 7f61357fa640 1 -- 172.21.15.98:0/2541775068 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61240074e0 con 0x7f61381ef6b0 2023-11-28T17:46:41.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.229+0000 7f61357fa640 1 -- 172.21.15.98:0/2541775068 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f6124007640 con 0x7f61381ef6b0 2023-11-28T17:46:41.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.229+0000 7f61357fa640 1 -- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61381ef6b0 msgr2=0x7f61380ca450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.229+0000 7f61357fa640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61381ef6b0 0x7f61380ca450 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f61240011e0 tx=0x7f612400c7f0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.229+0000 7f61357fa640 1 -- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61381ef6b0 msgr2=0x7f61380ca450 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:41.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.229+0000 7f61357fa640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61381ef6b0 0x7f61380ca450 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f61240011e0 tx=0x7f612400c7f0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.229+0000 7f613d110640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f611c06b2a0 0x7f611c06d760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f61300070b0 tx=0x7f6130007040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:41.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.236+0000 7f613eb73640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61381effe0 0x7f61380ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.236+0000 7f613eb73640 1 -- 172.21.15.98:0/2541775068 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61381574e0 con 0x7f61381effe0 2023-11-28T17:46:41.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.236+0000 7f6137fff640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61381effe0 0x7f61380ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:41.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.237+0000 7f6137fff640 1 -- 172.21.15.98:0/2541775068 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6138157200 con 0x7f61381effe0 2023-11-28T17:46:41.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.238+0000 7f6137fff640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61381effe0 0x7f61380ca990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f61380683d0 tx=0x7f6124001080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:41.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.238+0000 7f61357fa640 1 -- 172.21.15.98:0/2541775068 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612401f070 con 0x7f61381effe0 2023-11-28T17:46:41.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.239+0000 7f61357fa640 1 -- 172.21.15.98:0/2541775068 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4975 (secure 0 0 0) 0x7f6124013410 con 0x7f61381effe0 2023-11-28T17:46:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.239+0000 7f61357fa640 1 -- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61381effe0 msgr2=0x7f61380ca990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.239+0000 7f61357fa640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61381effe0 0x7f61380ca990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f61380683d0 tx=0x7f6124001080 comp rx=0 tx=0).stop 2023-11-28T17:46:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.239+0000 7f61357fa640 1 -- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61381effe0 msgr2=0x7f61380ca990 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.239+0000 7f61357fa640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f61381effe0 0x7f61380ca990 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f61380683d0 tx=0x7f6124001080 comp rx=0 tx=0).stop 2023-11-28T17:46:41.241 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:46:41.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:46:41.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:46:41.241 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:46:41.241 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 60, 2023-11-28T17:46:41.241 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:46:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:46:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:46:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:46:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2023-11-28T17:46:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:46:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:46:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:46:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:46:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:46:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:46:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:46:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:46:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:46:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:46:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:46:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:46:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:46:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:46:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:46:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:46:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:46:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:46:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:46:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:46:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:46:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:46:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:46:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:46:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:46:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:46:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:46:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:46:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:46:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:46:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:46:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:46:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:46:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:46:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:46:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:46:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:46:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:46:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:46:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:46:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:46:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:46:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:46:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:46:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 8 2023-11-28T17:46:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:46:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:46:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:46:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:46:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:46:41.258 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:46:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.239+0000 7f613eb73640 1 -- 172.21.15.98:0/2541775068 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f611c06b2a0 msgr2=0x7f611c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.239+0000 7f613eb73640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f611c06b2a0 0x7f611c06d760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f61300070b0 tx=0x7f6130007040 comp rx=0 tx=0).stop 2023-11-28T17:46:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.239+0000 7f613eb73640 1 -- 172.21.15.98:0/2541775068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61380c4380 msgr2=0x7f6138158a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.239+0000 7f613eb73640 1 --2- 172.21.15.98:0/2541775068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61380c4380 0x7f6138158a30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f612c0098f0 tx=0x7f612c0046c0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.239+0000 7f613d110640 1 -- 172.21.15.98:0/2541775068 reap_dead start 2023-11-28T17:46:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.239+0000 7f613d110640 1 -- 172.21.15.98:0/2541775068 reap_dead start 2023-11-28T17:46:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.239+0000 7f613eb73640 1 -- 172.21.15.98:0/2541775068 shutdown_connections 2023-11-28T17:46:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.239+0000 7f613eb73640 1 -- 172.21.15.98:0/2541775068 >> 172.21.15.98:0/2541775068 conn(0x7f613806c0f0 msgr2=0x7f613806dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.239+0000 7f613eb73640 1 -- 172.21.15.98:0/2541775068 shutdown_connections 2023-11-28T17:46:41.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.240+0000 7f613eb73640 1 -- 172.21.15.98:0/2541775068 wait complete. 2023-11-28T17:46:41.260 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:46:41.260 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-11-28T17:46:41.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-11-28T17:46:41.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.470+0000 7f9d45859640 1 Processor -- start 2023-11-28T17:46:41.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.470+0000 7f9d45859640 1 -- start start 2023-11-28T17:46:41.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.471+0000 7f9d45859640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9d400728a0 0x7f9d40158240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.471+0000 7f9d45859640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4015ac70 0x7f9d4015b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.471+0000 7f9d45859640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4006ade0 con 0x7f9d401518d0 2023-11-28T17:46:41.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.471+0000 7f9d45859640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d40151cb0 con 0x7f9d400728a0 2023-11-28T17:46:41.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.471+0000 7f9d45859640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4006b030 con 0x7f9d4015ac70 2023-11-28T17:46:41.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.471+0000 7f9d44857640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9d401518d0 0x7f9d40070bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51718/0 (socket says 172.21.15.98:51718) 2023-11-28T17:46:41.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.471+0000 7f9d44857640 1 -- 172.21.15.98:0/1033025430 learned_addr learned my addr 172.21.15.98:0/1033025430 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:46:41.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3ffff640 1 --2- 172.21.15.98:0/1033025430 >> v2:172.21.15.113:3300/0 conn(0x7f9d400728a0 0x7f9d40158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d45058640 1 --2- 172.21.15.98:0/1033025430 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4015ac70 0x7f9d4015b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 -- 172.21.15.98:0/1033025430 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051116570 0 0) 0x7f9d4006ade0 con 0x7f9d401518d0 2023-11-28T17:46:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 -- 172.21.15.98:0/1033025430 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d30003520 con 0x7f9d401518d0 2023-11-28T17:46:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 -- 172.21.15.98:0/1033025430 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1656580879 0 0) 0x7f9d30003520 con 0x7f9d401518d0 2023-11-28T17:46:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 -- 172.21.15.98:0/1033025430 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4015ac70 msgr2=0x7f9d4015b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 --2- 172.21.15.98:0/1033025430 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4015ac70 0x7f9d4015b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 -- 172.21.15.98:0/1033025430 >> v2:172.21.15.113:3300/0 conn(0x7f9d400728a0 msgr2=0x7f9d40158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 --2- 172.21.15.98:0/1033025430 >> v2:172.21.15.113:3300/0 conn(0x7f9d400728a0 0x7f9d40158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 -- 172.21.15.98:0/1033025430 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d400700e0 con 0x7f9d401518d0 2023-11-28T17:46:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 -- 172.21.15.98:0/1033025430 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f9d34001670 con 0x7f9d401518d0 2023-11-28T17:46:41.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 -- 172.21.15.98:0/1033025430 >> v1:172.21.15.98:6789/0 conn(0x7f9d401518d0 legacy=0x7f9d40070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 --2- 172.21.15.98:0/1033025430 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4015ac70 0x7f9d30003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 --2- 172.21.15.98:0/1033025430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d30003f40 0x7f9d300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 --2- 172.21.15.98:0/1033025430 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d30004d00 0x7f9d30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 -- 172.21.15.98:0/1033025430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d40151cb0 con 0x7f9d30003f40 2023-11-28T17:46:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 -- 172.21.15.98:0/1033025430 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4006b030 con 0x7f9d30004d00 2023-11-28T17:46:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.472+0000 7f9d3f7fe640 1 -- 172.21.15.98:0/1033025430 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3000ba30 con 0x7f9d4015ac70 2023-11-28T17:46:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.473+0000 7f9d3ffff640 1 --2- 172.21.15.98:0/1033025430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d30003f40 0x7f9d300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.473+0000 7f9d44857640 1 --2- 172.21.15.98:0/1033025430 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4015ac70 0x7f9d30003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.473+0000 7f9d45058640 1 --2- 172.21.15.98:0/1033025430 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d30004d00 0x7f9d30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.473+0000 7f9d3ffff640 1 -- 172.21.15.98:0/1033025430 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4015ac70 msgr2=0x7f9d30003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.473+0000 7f9d3ffff640 1 --2- 172.21.15.98:0/1033025430 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4015ac70 0x7f9d30003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.473+0000 7f9d3ffff640 1 -- 172.21.15.98:0/1033025430 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d30004d00 msgr2=0x7f9d30005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:41.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.473+0000 7f9d3ffff640 1 --2- 172.21.15.98:0/1033025430 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d30004d00 0x7f9d30005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.473+0000 7f9d3ffff640 1 -- 172.21.15.98:0/1033025430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d30003520 con 0x7f9d30003f40 2023-11-28T17:46:41.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.473+0000 7f9d45058640 1 --2- 172.21.15.98:0/1033025430 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d30004d00 0x7f9d30005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:46:41.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.473+0000 7f9d44857640 1 --2- 172.21.15.98:0/1033025430 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d4015ac70 0x7f9d30003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:46:41.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.474+0000 7f9d3ffff640 1 --2- 172.21.15.98:0/1033025430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d30003f40 0x7f9d300047c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c009ac0 tx=0x7f9d300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52534514686cd078 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:41.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.474+0000 7f9d3f7fe640 1 -- 172.21.15.98:0/1033025430 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c02c710 con 0x7f9d30003f40 2023-11-28T17:46:41.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.475+0000 7f9d3f7fe640 1 -- 172.21.15.98:0/1033025430 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2c02c870 con 0x7f9d30003f40 2023-11-28T17:46:41.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.475+0000 7f9d3f7fe640 1 -- 172.21.15.98:0/1033025430 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c02cc00 con 0x7f9d30003f40 2023-11-28T17:46:41.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.475+0000 7f9d45859640 1 -- 172.21.15.98:0/1033025430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d30003f40 msgr2=0x7f9d300047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.475+0000 7f9d45859640 1 --2- 172.21.15.98:0/1033025430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d30003f40 0x7f9d300047c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c009ac0 tx=0x7f9d300057f0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.475+0000 7f9d45058640 1 -- 172.21.15.98:0/1033025430 reap_dead start 2023-11-28T17:46:41.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.475+0000 7f9d45859640 1 -- 172.21.15.98:0/1033025430 shutdown_connections 2023-11-28T17:46:41.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.475+0000 7f9d45859640 1 -- 172.21.15.98:0/1033025430 >> 172.21.15.98:0/1033025430 conn(0x7f9d40066230 msgr2=0x7f9d4006a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:41.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.475+0000 7f9d45859640 1 -- 172.21.15.98:0/1033025430 shutdown_connections 2023-11-28T17:46:41.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.475+0000 7f9d45859640 1 -- 172.21.15.98:0/1033025430 wait complete. 2023-11-28T17:46:41.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.477+0000 7f9d45859640 1 Processor -- start 2023-11-28T17:46:41.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.477+0000 7f9d45859640 1 -- start start 2023-11-28T17:46:41.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.478+0000 7f9d45859640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d30003f40 0x7f9d401f1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.478+0000 7f9d45859640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d40151df0 0x7f9d4015b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.478+0000 7f9d45859640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d401edc80 0x7f9d401ee100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.478+0000 7f9d45859640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d401f19a0 con 0x7f9d401edc80 2023-11-28T17:46:41.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.478+0000 7f9d45859640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d40152510 con 0x7f9d30003f40 2023-11-28T17:46:41.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.478+0000 7f9d45859640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d401ed4e0 con 0x7f9d40151df0 2023-11-28T17:46:41.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.478+0000 7f9d45058640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d401edc80 0x7f9d401ee100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:41.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.478+0000 7f9d45058640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d401edc80 0x7f9d401ee100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36782/0 (socket says 172.21.15.98:36782) 2023-11-28T17:46:41.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.478+0000 7f9d45058640 1 -- 172.21.15.98:0/724090290 learned_addr learned my addr 172.21.15.98:0/724090290 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:41.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.479+0000 7f9d44857640 1 --2- 172.21.15.98:0/724090290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d30003f40 0x7f9d401f1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:41.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.479+0000 7f9d3ffff640 1 --2- 172.21.15.98:0/724090290 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d40151df0 0x7f9d4015b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:41.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.479+0000 7f9d45058640 1 -- 172.21.15.98:0/724090290 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d40151df0 msgr2=0x7f9d4015b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.479+0000 7f9d45058640 1 --2- 172.21.15.98:0/724090290 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d40151df0 0x7f9d4015b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.479+0000 7f9d45058640 1 -- 172.21.15.98:0/724090290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d30003f40 msgr2=0x7f9d401f1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.479+0000 7f9d45058640 1 --2- 172.21.15.98:0/724090290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d30003f40 0x7f9d401f1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.479+0000 7f9d45058640 1 -- 172.21.15.98:0/724090290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d401571f0 con 0x7f9d401edc80 2023-11-28T17:46:41.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.479+0000 7f9d45058640 1 --2- 172.21.15.98:0/724090290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d401edc80 0x7f9d401ee100 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9d400c4950 tx=0x7f9d3800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:41.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.479+0000 7f9d3d7fa640 1 -- 172.21.15.98:0/724090290 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d38017400 con 0x7f9d401edc80 2023-11-28T17:46:41.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.479+0000 7f9d3d7fa640 1 -- 172.21.15.98:0/724090290 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d38011040 con 0x7f9d401edc80 2023-11-28T17:46:41.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.479+0000 7f9d3d7fa640 1 -- 172.21.15.98:0/724090290 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d38017780 con 0x7f9d401edc80 2023-11-28T17:46:41.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.479+0000 7f9d45859640 1 -- 172.21.15.98:0/724090290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d401577d0 con 0x7f9d401edc80 2023-11-28T17:46:41.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.480+0000 7f9d45859640 1 -- 172.21.15.98:0/724090290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d401574d0 con 0x7f9d401edc80 2023-11-28T17:46:41.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.481+0000 7f9d45859640 1 -- 172.21.15.98:0/724090290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d401ed9a0 con 0x7f9d401edc80 2023-11-28T17:46:41.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.482+0000 7f9d3d7fa640 1 -- 172.21.15.98:0/724090290 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f9d380203b0 con 0x7f9d401edc80 2023-11-28T17:46:41.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.483+0000 7f9d3d7fa640 1 --2- 172.21.15.98:0/724090290 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9d2406b2d0 0x7f9d2406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:41.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.483+0000 7f9d3d7fa640 1 -- 172.21.15.98:0/724090290 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9d38089eb0 con 0x7f9d401edc80 2023-11-28T17:46:41.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.483+0000 7f9d44857640 1 --2- 172.21.15.98:0/724090290 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9d2406b2d0 0x7f9d2406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:41.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.484+0000 7f9d44857640 1 --2- 172.21.15.98:0/724090290 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9d2406b2d0 0x7f9d2406d790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9d34001860 tx=0x7f9d34009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:41.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.488+0000 7f9d3d7fa640 1 -- 172.21.15.98:0/724090290 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9d401ed9a0 con 0x7f9d401edc80 2023-11-28T17:46:41.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.746+0000 7f9d45859640 1 -- 172.21.15.98:0/724090290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f9d401ed650 con 0x7f9d401edc80 2023-11-28T17:46:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.747+0000 7f9d3d7fa640 1 -- 172.21.15.98:0/724090290 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f9d401ed650 con 0x7f9d401edc80 2023-11-28T17:46:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.749+0000 7f9d45859640 1 -- 172.21.15.98:0/724090290 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9d2406b2d0 msgr2=0x7f9d2406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.749+0000 7f9d45859640 1 --2- 172.21.15.98:0/724090290 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9d2406b2d0 0x7f9d2406d790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9d34001860 tx=0x7f9d34009040 comp rx=0 tx=0).stop 2023-11-28T17:46:41.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.749+0000 7f9d45859640 1 -- 172.21.15.98:0/724090290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d401edc80 msgr2=0x7f9d401ee100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.749+0000 7f9d45859640 1 --2- 172.21.15.98:0/724090290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d401edc80 0x7f9d401ee100 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9d400c4950 tx=0x7f9d3800f4a0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.749+0000 7f9d45859640 1 -- 172.21.15.98:0/724090290 shutdown_connections 2023-11-28T17:46:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.749+0000 7f9d45859640 1 --2- 172.21.15.98:0/724090290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d401edc80 0x7f9d401ee100 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.749+0000 7f9d45859640 1 --2- 172.21.15.98:0/724090290 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9d40151df0 0x7f9d4015b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.749+0000 7f9d45859640 1 --2- 172.21.15.98:0/724090290 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9d2406b2d0 0x7f9d2406d790 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.749+0000 7f9d45859640 1 --2- 172.21.15.98:0/724090290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d30003f40 0x7f9d401f1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.749+0000 7f9d45859640 1 -- 172.21.15.98:0/724090290 >> 172.21.15.98:0/724090290 conn(0x7f9d40066230 msgr2=0x7f9d400c2270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.750+0000 7f9d45859640 1 -- 172.21.15.98:0/724090290 shutdown_connections 2023-11-28T17:46:41.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:41.750+0000 7f9d45859640 1 -- 172.21.15.98:0/724090290 wait complete. 2023-11-28T17:46:41.772 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2023-11-28T17:46:46.324 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~115s 2023-11-28T17:46:51.774 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:46:51.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:46:51.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.995+0000 7fddbe141640 1 Processor -- start 2023-11-28T17:46:51.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.995+0000 7fddbe141640 1 -- start start 2023-11-28T17:46:51.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.997+0000 7fddbe141640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fddb80c5c00 0x7fddb80c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:51.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.997+0000 7fddbe141640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fddb80c4a20 0x7fddb80c6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:51.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.997+0000 7fddbe141640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb8157850 con 0x7fddb8150c00 2023-11-28T17:46:51.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.997+0000 7fddbe141640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb80c2e10 con 0x7fddb80c5c00 2023-11-28T17:46:51.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.997+0000 7fddbe141640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb8156980 con 0x7fddb80c4a20 2023-11-28T17:46:51.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.997+0000 7fddaffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fddb80c5c00 0x7fddb80c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:51.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.997+0000 7fddb77fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fddb8150c00 0x7fddb80c2700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60072/0 (socket says 172.21.15.98:60072) 2023-11-28T17:46:51.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.997+0000 7fddb77fe640 1 -- 172.21.15.98:0/3382943888 learned_addr learned my addr 172.21.15.98:0/3382943888 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:46:51.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.997+0000 7fddaffff640 1 -- 172.21.15.98:0/3382943888 >> v2:172.21.15.113:3300/0 conn(0x7fddb80c4a20 msgr2=0x7fddb80c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:51.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.997+0000 7fddaffff640 1 --2- 172.21.15.98:0/3382943888 >> v2:172.21.15.113:3300/0 conn(0x7fddb80c4a20 0x7fddb80c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:51.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.997+0000 7fddaffff640 1 -- 172.21.15.98:0/3382943888 >> v1:172.21.15.98:6789/0 conn(0x7fddb8150c00 legacy=0x7fddb80c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:51.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.997+0000 7fddaffff640 1 -- 172.21.15.98:0/3382943888 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb8156c40 con 0x7fddb80c5c00 2023-11-28T17:46:52.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.998+0000 7fddaffff640 1 --2- 172.21.15.98:0/3382943888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddb80c5c00 0x7fddb80c44e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdda00097a0 tx=0x7fdda002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a786edca78b2da1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:52.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.998+0000 7fddb6ffd640 1 -- 172.21.15.98:0/3382943888 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0031830 con 0x7fddb80c5c00 2023-11-28T17:46:52.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.998+0000 7fddb6ffd640 1 -- 172.21.15.98:0/3382943888 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda0031990 con 0x7fddb80c5c00 2023-11-28T17:46:52.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.998+0000 7fddb6ffd640 1 -- 172.21.15.98:0/3382943888 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0031d20 con 0x7fddb80c5c00 2023-11-28T17:46:52.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.999+0000 7fddbe141640 1 -- 172.21.15.98:0/3382943888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddb80c5c00 msgr2=0x7fddb80c44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.999+0000 7fddbe141640 1 --2- 172.21.15.98:0/3382943888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddb80c5c00 0x7fddb80c44e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdda00097a0 tx=0x7fdda002b8d0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.999+0000 7fddbe141640 1 -- 172.21.15.98:0/3382943888 shutdown_connections 2023-11-28T17:46:52.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.999+0000 7fddbe141640 1 --2- 172.21.15.98:0/3382943888 >> v2:172.21.15.113:3300/0 conn(0x7fddb80c4a20 0x7fddb80c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.999+0000 7fddbe141640 1 --2- 172.21.15.98:0/3382943888 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddb80c5c00 0x7fddb80c44e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.999+0000 7fddbe141640 1 -- 172.21.15.98:0/3382943888 >> 172.21.15.98:0/3382943888 conn(0x7fddb806a4d0 msgr2=0x7fddb806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:52.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.999+0000 7fddbe141640 1 -- 172.21.15.98:0/3382943888 shutdown_connections 2023-11-28T17:46:52.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:51.999+0000 7fddbe141640 1 -- 172.21.15.98:0/3382943888 wait complete. 2023-11-28T17:46:52.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.000+0000 7fddbe141640 1 Processor -- start 2023-11-28T17:46:52.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.001+0000 7fddbe141640 1 -- start start 2023-11-28T17:46:52.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.002+0000 7fddbe141640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddb80c4a20 0x7fddb81f03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.002+0000 7fddbe141640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddb80c5c00 0x7fddb80711a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.002+0000 7fddbe141640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddb8150c00 0x7fddb814c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.002+0000 7fddbe141640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb8072930 con 0x7fddb8150c00 2023-11-28T17:46:52.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.002+0000 7fddbe141640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb81f0900 con 0x7fddb80c5c00 2023-11-28T17:46:52.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.002+0000 7fddbe141640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb8070d10 con 0x7fddb80c4a20 2023-11-28T17:46:52.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.002+0000 7fddb7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddb8150c00 0x7fddb814c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.002+0000 7fddb7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddb8150c00 0x7fddb814c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50902/0 (socket says 172.21.15.98:50902) 2023-11-28T17:46:52.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.002+0000 7fddb7fff640 1 -- 172.21.15.98:0/2562253871 learned_addr learned my addr 172.21.15.98:0/2562253871 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:52.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.002+0000 7fddb7fff640 1 -- 172.21.15.98:0/2562253871 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddb80c4a20 msgr2=0x7fddb81f03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.002+0000 7fddb7fff640 1 --2- 172.21.15.98:0/2562253871 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddb80c4a20 0x7fddb81f03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.002+0000 7fddb7fff640 1 -- 172.21.15.98:0/2562253871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddb80c5c00 msgr2=0x7fddb80711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.002+0000 7fddaffff640 1 --2- 172.21.15.98:0/2562253871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddb80c5c00 0x7fddb80711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.002+0000 7fddb7fff640 1 --2- 172.21.15.98:0/2562253871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddb80c5c00 0x7fddb80711a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.002+0000 7fddb7fff640 1 -- 172.21.15.98:0/2562253871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda0009400 con 0x7fddb8150c00 2023-11-28T17:46:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.003+0000 7fddb7fff640 1 --2- 172.21.15.98:0/2562253871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddb8150c00 0x7fddb814c8e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fddb000b9a0 tx=0x7fddb000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.003+0000 7fddb4ff9640 1 -- 172.21.15.98:0/2562253871 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb000cb40 con 0x7fddb8150c00 2023-11-28T17:46:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.003+0000 7fddb4ff9640 1 -- 172.21.15.98:0/2562253871 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb000ccc0 con 0x7fddb8150c00 2023-11-28T17:46:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.003+0000 7fddb4ff9640 1 -- 172.21.15.98:0/2562253871 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0014680 con 0x7fddb8150c00 2023-11-28T17:46:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.003+0000 7fddbe141640 1 -- 172.21.15.98:0/2562253871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb80709e0 con 0x7fddb8150c00 2023-11-28T17:46:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.003+0000 7fddbe141640 1 -- 172.21.15.98:0/2562253871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb8072370 con 0x7fddb8150c00 2023-11-28T17:46:52.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.004+0000 7fddbe141640 1 -- 172.21.15.98:0/2562253871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb8067bf0 con 0x7fddb8150c00 2023-11-28T17:46:52.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.005+0000 7fddb4ff9640 1 -- 172.21.15.98:0/2562253871 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fddb001d3b0 con 0x7fddb8150c00 2023-11-28T17:46:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.006+0000 7fddb4ff9640 1 --2- 172.21.15.98:0/2562253871 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdd9806b2a0 0x7fdd9806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.006+0000 7fddb4ff9640 1 -- 172.21.15.98:0/2562253871 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fddb0089b40 con 0x7fddb8150c00 2023-11-28T17:46:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.007+0000 7fddb77fe640 1 --2- 172.21.15.98:0/2562253871 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdd9806b2a0 0x7fdd9806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.007+0000 7fddb77fe640 1 --2- 172.21.15.98:0/2562253871 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdd9806b2a0 0x7fdd9806d760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdda80015f0 tx=0x7fdda8007a40 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:52.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.010+0000 7fddb4ff9640 1 -- 172.21.15.98:0/2562253871 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fddb0054930 con 0x7fddb8150c00 2023-11-28T17:46:52.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.271+0000 7fddbe141640 1 -- 172.21.15.98:0/2562253871 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fddb8070840 con 0x7fddb8150c00 2023-11-28T17:46:52.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.271+0000 7fddb4ff9640 1 -- 172.21.15.98:0/2562253871 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7fddb8070840 con 0x7fddb8150c00 2023-11-28T17:46:52.273 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:46:52.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.274+0000 7fddbe141640 1 -- 172.21.15.98:0/2562253871 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdd9806b2a0 msgr2=0x7fdd9806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.274+0000 7fddbe141640 1 --2- 172.21.15.98:0/2562253871 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdd9806b2a0 0x7fdd9806d760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdda80015f0 tx=0x7fdda8007a40 comp rx=0 tx=0).stop 2023-11-28T17:46:52.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.274+0000 7fddbe141640 1 -- 172.21.15.98:0/2562253871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddb8150c00 msgr2=0x7fddb814c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.274+0000 7fddbe141640 1 --2- 172.21.15.98:0/2562253871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddb8150c00 0x7fddb814c8e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fddb000b9a0 tx=0x7fddb000be60 comp rx=0 tx=0).stop 2023-11-28T17:46:52.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.274+0000 7fddbe141640 1 -- 172.21.15.98:0/2562253871 shutdown_connections 2023-11-28T17:46:52.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.274+0000 7fddbe141640 1 --2- 172.21.15.98:0/2562253871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fddb8150c00 0x7fddb814c8e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.274+0000 7fddbe141640 1 --2- 172.21.15.98:0/2562253871 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdd9806b2a0 0x7fdd9806d760 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.274+0000 7fddbe141640 1 --2- 172.21.15.98:0/2562253871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fddb80c5c00 0x7fddb80711a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.274+0000 7fddbe141640 1 --2- 172.21.15.98:0/2562253871 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fddb80c4a20 0x7fddb81f03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.274+0000 7fddbe141640 1 -- 172.21.15.98:0/2562253871 >> 172.21.15.98:0/2562253871 conn(0x7fddb806a4d0 msgr2=0x7fddb80c66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:52.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.274+0000 7fddbe141640 1 -- 172.21.15.98:0/2562253871 shutdown_connections 2023-11-28T17:46:52.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.274+0000 7fddbe141640 1 -- 172.21.15.98:0/2562253871 wait complete. 2023-11-28T17:46:52.294 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:46:52.294 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-11-28T17:46:52.295 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:46:52.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:46:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.498+0000 7f81405be640 1 Processor -- start 2023-11-28T17:46:52.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.499+0000 7f81405be640 1 -- start start 2023-11-28T17:46:52.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.500+0000 7f81405be640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f8138153fd0 0x7f813815d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.500+0000 7f81405be640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81380b5220 0x7f81381645e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.500+0000 7f81405be640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813815a5d0 con 0x7f81380ca2b0 2023-11-28T17:46:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.500+0000 7f81405be640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8138159cc0 con 0x7f8138153fd0 2023-11-28T17:46:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.500+0000 7f81405be640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81380ca7b0 con 0x7f81380b5220 2023-11-28T17:46:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.500+0000 7f813db59640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f81380ca2b0 0x7f81380cf450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60078/0 (socket says 172.21.15.98:60078) 2023-11-28T17:46:52.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.500+0000 7f813db59640 1 -- 172.21.15.98:0/3822357496 learned_addr learned my addr 172.21.15.98:0/3822357496 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:46:52.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 -- 172.21.15.98:0/3822357496 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240203322 0 0) 0x7f813815a5d0 con 0x7f81380ca2b0 2023-11-28T17:46:52.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 -- 172.21.15.98:0/3822357496 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8120003520 con 0x7f81380ca2b0 2023-11-28T17:46:52.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813eb5b640 1 --2- 172.21.15.98:0/3822357496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81380b5220 0x7f81381645e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813e35a640 1 --2- 172.21.15.98:0/3822357496 >> v2:172.21.15.113:3300/0 conn(0x7f8138153fd0 0x7f813815d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 -- 172.21.15.98:0/3822357496 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2486129991 0 0) 0x7f8120003520 con 0x7f81380ca2b0 2023-11-28T17:46:52.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 -- 172.21.15.98:0/3822357496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81380b5220 msgr2=0x7f81381645e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:52.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 --2- 172.21.15.98:0/3822357496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81380b5220 0x7f81381645e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 -- 172.21.15.98:0/3822357496 >> v2:172.21.15.113:3300/0 conn(0x7f8138153fd0 msgr2=0x7f813815d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:52.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 --2- 172.21.15.98:0/3822357496 >> v2:172.21.15.113:3300/0 conn(0x7f8138153fd0 0x7f813815d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 -- 172.21.15.98:0/3822357496 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81380caa90 con 0x7f81380ca2b0 2023-11-28T17:46:52.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 -- 172.21.15.98:0/3822357496 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f812c001670 con 0x7f81380ca2b0 2023-11-28T17:46:52.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 -- 172.21.15.98:0/3822357496 >> v1:172.21.15.98:6789/0 conn(0x7f81380ca2b0 legacy=0x7f81380cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 --2- 172.21.15.98:0/3822357496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120003970 0x7f8120003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 --2- 172.21.15.98:0/3822357496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81200047c0 0x7f8120004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 --2- 172.21.15.98:0/3822357496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81380b5220 0x7f8120005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 -- 172.21.15.98:0/3822357496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8138159cc0 con 0x7f8120003970 2023-11-28T17:46:52.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 -- 172.21.15.98:0/3822357496 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81380ca7b0 con 0x7f81200047c0 2023-11-28T17:46:52.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813d358640 1 -- 172.21.15.98:0/3822357496 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812000bb10 con 0x7f81380b5220 2023-11-28T17:46:52.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.501+0000 7f813e35a640 1 --2- 172.21.15.98:0/3822357496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120003970 0x7f8120003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.502+0000 7f813e35a640 1 -- 172.21.15.98:0/3822357496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81380b5220 msgr2=0x7f8120005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:52.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.502+0000 7f813e35a640 1 --2- 172.21.15.98:0/3822357496 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81380b5220 0x7f8120005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.502+0000 7f813e35a640 1 -- 172.21.15.98:0/3822357496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81200047c0 msgr2=0x7f8120004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:52.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.502+0000 7f813e35a640 1 --2- 172.21.15.98:0/3822357496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81200047c0 0x7f8120004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.502+0000 7f813e35a640 1 -- 172.21.15.98:0/3822357496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8120003520 con 0x7f8120003970 2023-11-28T17:46:52.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.503+0000 7f813e35a640 1 --2- 172.21.15.98:0/3822357496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120003970 0x7f8120003d90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8128007330 tx=0x7f81200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d93a7701b61727 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:52.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.503+0000 7f813d358640 1 -- 172.21.15.98:0/3822357496 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812802ccb0 con 0x7f8120003970 2023-11-28T17:46:52.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.503+0000 7f813d358640 1 -- 172.21.15.98:0/3822357496 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812802ce10 con 0x7f8120003970 2023-11-28T17:46:52.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.503+0000 7f813d358640 1 -- 172.21.15.98:0/3822357496 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128034660 con 0x7f8120003970 2023-11-28T17:46:52.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.504+0000 7f81405be640 1 -- 172.21.15.98:0/3822357496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120003970 msgr2=0x7f8120003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.504+0000 7f81405be640 1 --2- 172.21.15.98:0/3822357496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8120003970 0x7f8120003d90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8128007330 tx=0x7f81200057f0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.504+0000 7f813eb5b640 1 -- 172.21.15.98:0/3822357496 reap_dead start 2023-11-28T17:46:52.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.504+0000 7f81405be640 1 -- 172.21.15.98:0/3822357496 shutdown_connections 2023-11-28T17:46:52.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.504+0000 7f81405be640 1 -- 172.21.15.98:0/3822357496 >> 172.21.15.98:0/3822357496 conn(0x7f813806af30 msgr2=0x7f813806b340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:52.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.504+0000 7f81405be640 1 -- 172.21.15.98:0/3822357496 shutdown_connections 2023-11-28T17:46:52.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.504+0000 7f81405be640 1 -- 172.21.15.98:0/3822357496 wait complete. 2023-11-28T17:46:52.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.505+0000 7f81405be640 1 Processor -- start 2023-11-28T17:46:52.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.506+0000 7f81405be640 1 -- start start 2023-11-28T17:46:52.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.506+0000 7f81405be640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8120003970 0x7f8138154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.506+0000 7f81405be640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81380d0190 0x7f8138150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.506+0000 7f81405be640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81380c49d0 0x7f8138150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.506+0000 7f81405be640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81381ef950 con 0x7f81380c49d0 2023-11-28T17:46:52.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.506+0000 7f81405be640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81381efb40 con 0x7f81380d0190 2023-11-28T17:46:52.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.506+0000 7f81405be640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81381efe20 con 0x7f8120003970 2023-11-28T17:46:52.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.507+0000 7f813eb5b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81380c49d0 0x7f8138150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.507+0000 7f813eb5b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81380c49d0 0x7f8138150f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50928/0 (socket says 172.21.15.98:50928) 2023-11-28T17:46:52.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.507+0000 7f813eb5b640 1 -- 172.21.15.98:0/2095086338 learned_addr learned my addr 172.21.15.98:0/2095086338 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:52.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.507+0000 7f813db59640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81380d0190 0x7f8138150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.507+0000 7f813e35a640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8120003970 0x7f8138154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.507+0000 7f813eb5b640 1 -- 172.21.15.98:0/2095086338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8120003970 msgr2=0x7f8138154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.507+0000 7f813eb5b640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8120003970 0x7f8138154f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.507+0000 7f813eb5b640 1 -- 172.21.15.98:0/2095086338 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81380d0190 msgr2=0x7f8138150a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.507+0000 7f813eb5b640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81380d0190 0x7f8138150a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.507+0000 7f813eb5b640 1 -- 172.21.15.98:0/2095086338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81381f0100 con 0x7f81380c49d0 2023-11-28T17:46:52.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.507+0000 7f813eb5b640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81380c49d0 0x7f8138150f50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f813806a590 tx=0x7f813000d8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:52.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.508+0000 7f8136ffd640 1 -- 172.21.15.98:0/2095086338 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130004030 con 0x7f81380c49d0 2023-11-28T17:46:52.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.508+0000 7f8136ffd640 1 -- 172.21.15.98:0/2095086338 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81300041b0 con 0x7f81380c49d0 2023-11-28T17:46:52.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.508+0000 7f8136ffd640 1 -- 172.21.15.98:0/2095086338 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81300126b0 con 0x7f81380c49d0 2023-11-28T17:46:52.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.508+0000 7f81405be640 1 -- 172.21.15.98:0/2095086338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81381f03e0 con 0x7f81380c49d0 2023-11-28T17:46:52.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.508+0000 7f81405be640 1 -- 172.21.15.98:0/2095086338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81380cc7c0 con 0x7f81380c49d0 2023-11-28T17:46:52.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.509+0000 7f81405be640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81381563b0 0x7f81381f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.509+0000 7f81405be640 1 -- 172.21.15.98:0/2095086338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8138074110 con 0x7f81381563b0 2023-11-28T17:46:52.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.509+0000 7f813e35a640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81381563b0 0x7f81381f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.509+0000 7f813e35a640 1 -- 172.21.15.98:0/2095086338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81381543b0 con 0x7f81381563b0 2023-11-28T17:46:52.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.510+0000 7f813e35a640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81381563b0 0x7f81381f2ec0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f81200057a0 tx=0x7f8128004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:52.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.510+0000 7f8136ffd640 1 -- 172.21.15.98:0/2095086338 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f8130012810 con 0x7f81380c49d0 2023-11-28T17:46:52.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.511+0000 7f8136ffd640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f812406b2d0 0x7f812406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.511+0000 7f8136ffd640 1 -- 172.21.15.98:0/2095086338 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8130089450 con 0x7f81380c49d0 2023-11-28T17:46:52.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.511+0000 7f813db59640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f812406b2d0 0x7f812406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.511+0000 7f8136ffd640 1 -- 172.21.15.98:0/2095086338 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812803e610 con 0x7f81381563b0 2023-11-28T17:46:52.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.511+0000 7f8136ffd640 1 -- 172.21.15.98:0/2095086338 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f81280071f0 con 0x7f81381563b0 2023-11-28T17:46:52.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.511+0000 7f8136ffd640 1 -- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81381563b0 msgr2=0x7f81381f2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.511+0000 7f8136ffd640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81381563b0 0x7f81381f2ec0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f81200057a0 tx=0x7f8128004ab0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.511+0000 7f8136ffd640 1 -- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81381563b0 msgr2=0x7f81381f2ec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:52.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.511+0000 7f8136ffd640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81381563b0 0x7f81381f2ec0 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f81200057a0 tx=0x7f8128004ab0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.512+0000 7f813db59640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f812406b2d0 0x7f812406d790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f812c001020 tx=0x7f812c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:52.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.518+0000 7f81405be640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81381f3400 0x7f81381f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.518+0000 7f81405be640 1 -- 172.21.15.98:0/2095086338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81381f0570 con 0x7f81381f3400 2023-11-28T17:46:52.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.519+0000 7f813e35a640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81381f3400 0x7f81381f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.519+0000 7f813e35a640 1 -- 172.21.15.98:0/2095086338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f813815a260 con 0x7f81381f3400 2023-11-28T17:46:52.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.519+0000 7f813e35a640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81381f3400 0x7f81381f3810 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8128037040 tx=0x7f812800b030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:52.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.520+0000 7f8136ffd640 1 -- 172.21.15.98:0/2095086338 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128048d90 con 0x7f81381f3400 2023-11-28T17:46:52.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.520+0000 7f8136ffd640 1 -- 172.21.15.98:0/2095086338 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4638 (secure 0 0 0) 0x7f8128052400 con 0x7f81381f3400 2023-11-28T17:46:52.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.520+0000 7f8136ffd640 1 -- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81381f3400 msgr2=0x7f81381f3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.520+0000 7f8136ffd640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81381f3400 0x7f81381f3810 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8128037040 tx=0x7f812800b030 comp rx=0 tx=0).stop 2023-11-28T17:46:52.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.520+0000 7f8136ffd640 1 -- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81381f3400 msgr2=0x7f81381f3810 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:52.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.520+0000 7f8136ffd640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81381f3400 0x7f81381f3810 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8128037040 tx=0x7f812800b030 comp rx=0 tx=0).stop 2023-11-28T17:46:52.522 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:46:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:46:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:46:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:46:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 60, 2023-11-28T17:46:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:46:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:46:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:46:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:46:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 12, 2023-11-28T17:46:52.523 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:46:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:46:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:52.524 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:46:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:46:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:52.525 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:46:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:46:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:46:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:46:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:46:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:46:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:52.526 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:46:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:46:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:46:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:46:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:46:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:46:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:46:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:46:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:52.527 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:46:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:46:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:46:52.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:46:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:46:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:46:52.529 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:46:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:46:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:46:52.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:46:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:46:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:46:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:52.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:46:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:46:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:46:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:46:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:52.532 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:46:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:46:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.534 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:46:52.534 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:46:52.534 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:52.534 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:52.534 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:52.534 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:52.534 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:52.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.534 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:46:52.534 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:46:52.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.534 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:52.535 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:52.535 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:46:52.535 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:52.535 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:52.535 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:46:52.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.535 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:52.535 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:52.535 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:46:52.535 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:52.535 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:52.535 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:46:52.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.536 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:52.536 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:52.536 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:46:52.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:52.536 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:52.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.536 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:46:52.536 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:46:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.521+0000 7f81405be640 1 -- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f812406b2d0 msgr2=0x7f812406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.521+0000 7f81405be640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f812406b2d0 0x7f812406d790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f812c001020 tx=0x7f812c009040 comp rx=0 tx=0).stop 2023-11-28T17:46:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.521+0000 7f81405be640 1 -- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81380c49d0 msgr2=0x7f8138150f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.521+0000 7f81405be640 1 --2- 172.21.15.98:0/2095086338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81380c49d0 0x7f8138150f50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f813806a590 tx=0x7f813000d8a0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.521+0000 7f813eb5b640 1 -- 172.21.15.98:0/2095086338 reap_dead start 2023-11-28T17:46:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.521+0000 7f813eb5b640 1 -- 172.21.15.98:0/2095086338 reap_dead start 2023-11-28T17:46:52.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.521+0000 7f81405be640 1 -- 172.21.15.98:0/2095086338 shutdown_connections 2023-11-28T17:46:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.521+0000 7f81405be640 1 -- 172.21.15.98:0/2095086338 >> 172.21.15.98:0/2095086338 conn(0x7f813806af30 msgr2=0x7f81380cbed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.521+0000 7f81405be640 1 -- 172.21.15.98:0/2095086338 shutdown_connections 2023-11-28T17:46:52.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.521+0000 7f81405be640 1 -- 172.21.15.98:0/2095086338 wait complete. 2023-11-28T17:46:52.539 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:46:52.539 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:46:52.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:46:52.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.747+0000 7fed3a021640 1 Processor -- start 2023-11-28T17:46:52.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.747+0000 7fed3a021640 1 -- start start 2023-11-28T17:46:52.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed3a021640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed340c9cb0 0x7fed340cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed3a021640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed340cac10 con 0x7fed34153ed0 2023-11-28T17:46:52.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed3a021640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed34159c10 con 0x7fed340cb360 2023-11-28T17:46:52.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed3a021640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed341542b0 con 0x7fed340c9cb0 2023-11-28T17:46:52.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed3901f640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fed34153ed0 0x7fed34071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60080/0 (socket says 172.21.15.98:60080) 2023-11-28T17:46:52.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed3901f640 1 -- 172.21.15.98:0/281883397 learned_addr learned my addr 172.21.15.98:0/281883397 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:46:52.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed2bfff640 1 -- 172.21.15.98:0/281883397 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329929096 0 0) 0x7fed340cac10 con 0x7fed34153ed0 2023-11-28T17:46:52.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed2bfff640 1 -- 172.21.15.98:0/281883397 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed20003520 con 0x7fed34153ed0 2023-11-28T17:46:52.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed3881e640 1 --2- 172.21.15.98:0/281883397 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed340c9cb0 0x7fed340cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed2bfff640 1 -- 172.21.15.98:0/281883397 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 643569105 0 0) 0x7fed20003520 con 0x7fed34153ed0 2023-11-28T17:46:52.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed2bfff640 1 -- 172.21.15.98:0/281883397 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed340c9cb0 msgr2=0x7fed340cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:52.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed2bfff640 1 --2- 172.21.15.98:0/281883397 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed340c9cb0 0x7fed340cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed2bfff640 1 -- 172.21.15.98:0/281883397 >> v1:172.21.15.113:6789/0 conn(0x7fed340cb360 legacy=0x7fed3415a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed2bfff640 1 -- 172.21.15.98:0/281883397 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed340cb140 con 0x7fed34153ed0 2023-11-28T17:46:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed2bfff640 1 -- 172.21.15.98:0/281883397 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7fed30001670 con 0x7fed34153ed0 2023-11-28T17:46:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed2bfff640 1 -- 172.21.15.98:0/281883397 >> v1:172.21.15.98:6789/0 conn(0x7fed34153ed0 legacy=0x7fed34071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed2bfff640 1 --2- 172.21.15.98:0/281883397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed200039a0 0x7fed20003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed2bfff640 1 --2- 172.21.15.98:0/281883397 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed340c9cb0 0x7fed200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed2bfff640 1 --2- 172.21.15.98:0/281883397 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed20004d00 0x7fed20005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed2bfff640 1 -- 172.21.15.98:0/281883397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2000bba0 con 0x7fed200039a0 2023-11-28T17:46:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed2bfff640 1 -- 172.21.15.98:0/281883397 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2000bce0 con 0x7fed20004d00 2023-11-28T17:46:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.750+0000 7fed2bfff640 1 -- 172.21.15.98:0/281883397 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed2000be70 con 0x7fed340c9cb0 2023-11-28T17:46:52.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.751+0000 7fed3901f640 1 --2- 172.21.15.98:0/281883397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed200039a0 0x7fed20003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.751+0000 7fed39820640 1 --2- 172.21.15.98:0/281883397 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed20004d00 0x7fed20005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.751+0000 7fed3881e640 1 --2- 172.21.15.98:0/281883397 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed340c9cb0 0x7fed200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.751+0000 7fed3901f640 1 -- 172.21.15.98:0/281883397 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed340c9cb0 msgr2=0x7fed200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:52.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.751+0000 7fed3901f640 1 --2- 172.21.15.98:0/281883397 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed340c9cb0 0x7fed200047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.751+0000 7fed3901f640 1 -- 172.21.15.98:0/281883397 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed20004d00 msgr2=0x7fed20005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:52.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.751+0000 7fed3901f640 1 --2- 172.21.15.98:0/281883397 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed20004d00 0x7fed20005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.751+0000 7fed3901f640 1 -- 172.21.15.98:0/281883397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed20003520 con 0x7fed200039a0 2023-11-28T17:46:52.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.751+0000 7fed3881e640 1 --2- 172.21.15.98:0/281883397 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed340c9cb0 0x7fed200047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:46:52.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.752+0000 7fed3901f640 1 --2- 172.21.15.98:0/281883397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed200039a0 0x7fed20003dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fed3000c240 tx=0x7fed200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4757178db03b872d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:52.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.752+0000 7fed2bfff640 1 -- 172.21.15.98:0/281883397 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed30005ad0 con 0x7fed200039a0 2023-11-28T17:46:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.752+0000 7fed2bfff640 1 -- 172.21.15.98:0/281883397 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed30005c30 con 0x7fed200039a0 2023-11-28T17:46:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.752+0000 7fed2bfff640 1 -- 172.21.15.98:0/281883397 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3002c870 con 0x7fed200039a0 2023-11-28T17:46:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.753+0000 7fed3a021640 1 -- 172.21.15.98:0/281883397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed200039a0 msgr2=0x7fed20003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.753+0000 7fed3a021640 1 --2- 172.21.15.98:0/281883397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed200039a0 0x7fed20003dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fed3000c240 tx=0x7fed200057f0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.753+0000 7fed39820640 1 -- 172.21.15.98:0/281883397 reap_dead start 2023-11-28T17:46:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.753+0000 7fed3a021640 1 -- 172.21.15.98:0/281883397 shutdown_connections 2023-11-28T17:46:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.753+0000 7fed3a021640 1 -- 172.21.15.98:0/281883397 >> 172.21.15.98:0/281883397 conn(0x7fed3406d380 msgr2=0x7fed3406acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.753+0000 7fed3a021640 1 -- 172.21.15.98:0/281883397 shutdown_connections 2023-11-28T17:46:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.753+0000 7fed3a021640 1 -- 172.21.15.98:0/281883397 wait complete. 2023-11-28T17:46:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.755+0000 7fed3a021640 1 Processor -- start 2023-11-28T17:46:52.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.755+0000 7fed3a021640 1 -- start start 2023-11-28T17:46:52.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.756+0000 7fed3a021640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed200039a0 0x7fed341f36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.756+0000 7fed3a021640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34157940 0x7fed341526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.756+0000 7fed3a021640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed341ef5d0 0x7fed340b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.756+0000 7fed3a021640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed341f3230 con 0x7fed200039a0 2023-11-28T17:46:52.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.756+0000 7fed3a021640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed341efa80 con 0x7fed34157940 2023-11-28T17:46:52.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.756+0000 7fed3a021640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3414f040 con 0x7fed341ef5d0 2023-11-28T17:46:52.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.756+0000 7fed3901f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed200039a0 0x7fed341f36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.756+0000 7fed3901f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed200039a0 0x7fed341f36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50958/0 (socket says 172.21.15.98:50958) 2023-11-28T17:46:52.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.756+0000 7fed3901f640 1 -- 172.21.15.98:0/549777599 learned_addr learned my addr 172.21.15.98:0/549777599 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:52.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.757+0000 7fed3901f640 1 -- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed341ef5d0 msgr2=0x7fed340b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.757+0000 7fed3901f640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fed341ef5d0 0x7fed340b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.757+0000 7fed3901f640 1 -- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34157940 msgr2=0x7fed341526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.757+0000 7fed3901f640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34157940 0x7fed341526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.757+0000 7fed3901f640 1 -- 172.21.15.98:0/549777599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed34153670 con 0x7fed200039a0 2023-11-28T17:46:52.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.757+0000 7fed3901f640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed200039a0 0x7fed341f36c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fed30001930 tx=0x7fed340b20b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:52.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.757+0000 7fed29ffb640 1 -- 172.21.15.98:0/549777599 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed30039070 con 0x7fed200039a0 2023-11-28T17:46:52.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.757+0000 7fed29ffb640 1 -- 172.21.15.98:0/549777599 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3002cc80 con 0x7fed200039a0 2023-11-28T17:46:52.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.757+0000 7fed29ffb640 1 -- 172.21.15.98:0/549777599 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3003c040 con 0x7fed200039a0 2023-11-28T17:46:52.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.757+0000 7fed3a021640 1 -- 172.21.15.98:0/549777599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed34157e60 con 0x7fed200039a0 2023-11-28T17:46:52.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.757+0000 7fed3a021640 1 -- 172.21.15.98:0/549777599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed341f2be0 con 0x7fed200039a0 2023-11-28T17:46:52.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.758+0000 7fed3a021640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34143c10 0x7fed34144050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.758+0000 7fed3a021640 1 -- 172.21.15.98:0/549777599 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed34144590 con 0x7fed34143c10 2023-11-28T17:46:52.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.759+0000 7fed3881e640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34143c10 0x7fed34144050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.759+0000 7fed29ffb640 1 -- 172.21.15.98:0/549777599 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fed30032070 con 0x7fed200039a0 2023-11-28T17:46:52.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.760+0000 7fed3881e640 1 -- 172.21.15.98:0/549777599 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed340c5810 con 0x7fed34143c10 2023-11-28T17:46:52.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.760+0000 7fed29ffb640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fed1806b2d0 0x7fed1806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.760+0000 7fed3881e640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34143c10 0x7fed34144050 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fed340b4630 tx=0x7fed2400e8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:52.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.760+0000 7fed29ffb640 1 -- 172.21.15.98:0/549777599 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fed300aaf60 con 0x7fed200039a0 2023-11-28T17:46:52.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.760+0000 7fed39820640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fed1806b2d0 0x7fed1806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.761+0000 7fed29ffb640 1 -- 172.21.15.98:0/549777599 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2400c070 con 0x7fed34143c10 2023-11-28T17:46:52.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.761+0000 7fed39820640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fed1806b2d0 0x7fed1806d790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fed340cb740 tx=0x7fed2c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:52.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.761+0000 7fed29ffb640 1 -- 172.21.15.98:0/549777599 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fed24013610 con 0x7fed34143c10 2023-11-28T17:46:52.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.761+0000 7fed29ffb640 1 -- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34143c10 msgr2=0x7fed34144050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.761+0000 7fed29ffb640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34143c10 0x7fed34144050 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fed340b4630 tx=0x7fed2400e8a0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.761+0000 7fed29ffb640 1 -- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34143c10 msgr2=0x7fed34144050 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:52.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.761+0000 7fed29ffb640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34143c10 0x7fed34144050 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fed340b4630 tx=0x7fed2400e8a0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.769+0000 7fed3a021640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34148d10 0x7fed34149120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:52.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.769+0000 7fed3a021640 1 -- 172.21.15.98:0/549777599 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed34068170 con 0x7fed34148d10 2023-11-28T17:46:52.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.769+0000 7fed3881e640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34148d10 0x7fed34149120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:52.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.770+0000 7fed3881e640 1 -- 172.21.15.98:0/549777599 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fed3415df20 con 0x7fed34148d10 2023-11-28T17:46:52.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.770+0000 7fed3881e640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34148d10 0x7fed34149120 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fed24018000 tx=0x7fed24004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:52.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.771+0000 7fed29ffb640 1 -- 172.21.15.98:0/549777599 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed24013d20 con 0x7fed34148d10 2023-11-28T17:46:52.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.771+0000 7fed29ffb640 1 -- 172.21.15.98:0/549777599 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4466 (secure 0 0 0) 0x7fed24013e80 con 0x7fed34148d10 2023-11-28T17:46:52.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.771+0000 7fed29ffb640 1 -- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34148d10 msgr2=0x7fed34149120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.771+0000 7fed29ffb640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34148d10 0x7fed34149120 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fed24018000 tx=0x7fed24004730 comp rx=0 tx=0).stop 2023-11-28T17:46:52.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.771+0000 7fed29ffb640 1 -- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34148d10 msgr2=0x7fed34149120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:52.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.771+0000 7fed29ffb640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed34148d10 0x7fed34149120 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.774 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:46:52.774 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:46:52.774 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:46:52.774 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:46:52.774 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 60, 2023-11-28T17:46:52.774 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:46:52.775 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:46:52.775 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:46:52.775 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:46:52.775 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:52.775 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 12, 2023-11-28T17:46:52.775 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:52.775 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:46:52.775 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:46:52.775 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:52.775 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:52.775 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:52.775 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:52.775 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:52.776 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:52.776 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:52.776 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:52.776 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:52.776 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:52.776 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:52.776 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:46:52.776 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:46:52.776 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:52.776 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:52.776 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:52.776 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:52.777 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:52.777 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:52.777 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:52.777 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:52.777 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:52.777 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:52.777 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:52.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.777 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:46:52.777 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:46:52.777 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:46:52.777 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:46:52.777 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:46:52.778 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:46:52.778 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:52.778 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:52.778 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:46:52.778 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:46:52.778 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:46:52.778 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:46:52.778 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:46:52.778 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:46:52.778 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:46:52.778 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:52.778 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:46:52.779 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:52.779 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:52.779 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:52.779 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:52.779 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:52.779 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:52.779 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:52.779 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:52.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:52.779 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:52.779 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:52.779 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:46:52.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.780 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:46:52.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.780 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:46:52.780 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:46:52.780 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:52.780 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:52.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:52.780 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:46:52.780 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:52.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:52.781 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:46:52.781 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:52.781 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:52.781 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:52.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.781 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:46:52.781 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:46:52.781 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:52.781 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:52.781 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:52.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.782 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:46:52.782 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:46:52.782 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:52.782 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:52.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:52.782 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:46:52.782 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:52.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:52.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:46:52.783 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:52.783 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:52.783 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:52.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:46:52.783 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:46:52.783 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:52.783 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:52.783 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:52.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.784 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:46:52.784 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:46:52.784 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:52.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:52.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:52.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:46:52.784 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:52.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:52.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:46:52.785 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:52.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:52.785 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:52.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:46:52.785 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:46:52.785 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:52.785 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:52.785 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:52.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:52.785 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:52.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.786 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:46:52.786 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:46:52.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.786 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:52.786 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:52.786 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:46:52.786 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:52.786 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:52.786 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:46:52.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:52.786 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:52.786 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:52.787 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:46:52.787 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:52.787 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:52.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:52.787 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:46:52.787 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:46:52.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.772+0000 7fed3a021640 1 -- 172.21.15.98:0/549777599 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fed1806b2d0 msgr2=0x7fed1806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.772+0000 7fed3a021640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fed1806b2d0 0x7fed1806d790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fed340cb740 tx=0x7fed2c009040 comp rx=0 tx=0).stop 2023-11-28T17:46:52.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.772+0000 7fed3a021640 1 -- 172.21.15.98:0/549777599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed200039a0 msgr2=0x7fed341f36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:52.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.772+0000 7fed3a021640 1 --2- 172.21.15.98:0/549777599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed200039a0 0x7fed341f36c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fed30001930 tx=0x7fed340b20b0 comp rx=0 tx=0).stop 2023-11-28T17:46:52.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.772+0000 7fed39820640 1 -- 172.21.15.98:0/549777599 reap_dead start 2023-11-28T17:46:52.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.772+0000 7fed39820640 1 -- 172.21.15.98:0/549777599 reap_dead start 2023-11-28T17:46:52.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.772+0000 7fed3a021640 1 -- 172.21.15.98:0/549777599 shutdown_connections 2023-11-28T17:46:52.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.772+0000 7fed3a021640 1 -- 172.21.15.98:0/549777599 >> 172.21.15.98:0/549777599 conn(0x7fed3406d380 msgr2=0x7fed3415a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:52.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.772+0000 7fed3a021640 1 -- 172.21.15.98:0/549777599 shutdown_connections 2023-11-28T17:46:52.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.772+0000 7fed3a021640 1 -- 172.21.15.98:0/549777599 wait complete. 2023-11-28T17:46:52.789 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:46:52.790 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:46:52.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:46:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.997+0000 7f7d75e94640 1 Processor -- start 2023-11-28T17:46:52.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.997+0000 7f7d75e94640 1 -- start start 2023-11-28T17:46:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.999+0000 7f7d75e94640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d70151be0 0x7f7d700c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.999+0000 7f7d75e94640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7d701584d0 0x7f7d7006dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.999+0000 7f7d75e94640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d700c8150 con 0x7f7d700c5a60 2023-11-28T17:46:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.999+0000 7f7d75e94640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d700c8850 con 0x7f7d701584d0 2023-11-28T17:46:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.999+0000 7f7d75e94640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d700c5f50 con 0x7f7d70151be0 2023-11-28T17:46:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.999+0000 7f7d6ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7d701584d0 0x7f7d7006dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.999+0000 7f7d6ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7d701584d0 0x7f7d7006dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50968/0 (socket says 172.21.15.98:50968) 2023-11-28T17:46:53.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.999+0000 7f7d6ffff640 1 -- 172.21.15.98:0/4109651911 learned_addr learned my addr 172.21.15.98:0/4109651911 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:53.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:52.999+0000 7f7d6f7fe640 1 --2- 172.21.15.98:0/4109651911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d70151be0 0x7f7d700c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:53.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.000+0000 7f7d6ffff640 1 -- 172.21.15.98:0/4109651911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d70151be0 msgr2=0x7f7d700c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.000+0000 7f7d6ffff640 1 --2- 172.21.15.98:0/4109651911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d70151be0 0x7f7d700c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.000+0000 7f7d6ffff640 1 -- 172.21.15.98:0/4109651911 >> v1:172.21.15.98:6789/0 conn(0x7f7d700c5a60 legacy=0x7f7d700c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.000+0000 7f7d6ffff640 1 -- 172.21.15.98:0/4109651911 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d700c7b90 con 0x7f7d701584d0 2023-11-28T17:46:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.000+0000 7f7d6e7fc640 1 -- 172.21.15.98:0/4109651911 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144383219 0 0) 0x7f7d700c8150 con 0x7f7d700c5a60 2023-11-28T17:46:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.001+0000 7f7d6ffff640 1 --2- 172.21.15.98:0/4109651911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d701584d0 0x7f7d7006dd80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7d68009e50 tx=0x7f7d6802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d2c3aef31183d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.001+0000 7f7d6e7fc640 1 -- 172.21.15.98:0/4109651911 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6802cb00 con 0x7f7d701584d0 2023-11-28T17:46:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.001+0000 7f7d6e7fc640 1 -- 172.21.15.98:0/4109651911 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6802cc60 con 0x7f7d701584d0 2023-11-28T17:46:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.001+0000 7f7d6e7fc640 1 -- 172.21.15.98:0/4109651911 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d680343f0 con 0x7f7d701584d0 2023-11-28T17:46:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.001+0000 7f7d75e94640 1 -- 172.21.15.98:0/4109651911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d701584d0 msgr2=0x7f7d7006dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.001+0000 7f7d75e94640 1 --2- 172.21.15.98:0/4109651911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d701584d0 0x7f7d7006dd80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7d68009e50 tx=0x7f7d6802bf60 comp rx=0 tx=0).stop 2023-11-28T17:46:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.002+0000 7f7d75e94640 1 -- 172.21.15.98:0/4109651911 shutdown_connections 2023-11-28T17:46:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.002+0000 7f7d75e94640 1 --2- 172.21.15.98:0/4109651911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d701584d0 0x7f7d7006dd80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.002+0000 7f7d75e94640 1 --2- 172.21.15.98:0/4109651911 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d70151be0 0x7f7d700c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.002+0000 7f7d75e94640 1 -- 172.21.15.98:0/4109651911 >> 172.21.15.98:0/4109651911 conn(0x7f7d7006c0f0 msgr2=0x7f7d7006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:53.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.002+0000 7f7d75e94640 1 -- 172.21.15.98:0/4109651911 shutdown_connections 2023-11-28T17:46:53.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.002+0000 7f7d75e94640 1 -- 172.21.15.98:0/4109651911 wait complete. 2023-11-28T17:46:53.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.003+0000 7f7d75e94640 1 Processor -- start 2023-11-28T17:46:53.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.004+0000 7f7d75e94640 1 -- start start 2023-11-28T17:46:53.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.004+0000 7f7d75e94640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d700c5a60 0x7f7d701f3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:53.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.004+0000 7f7d75e94640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d70151be0 0x7f7d701f0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:53.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.004+0000 7f7d75e94640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d701584d0 0x7f7d7014fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:53.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.004+0000 7f7d75e94640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d701efd40 con 0x7f7d70151be0 2023-11-28T17:46:53.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.004+0000 7f7d75e94640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d701f3890 con 0x7f7d701584d0 2023-11-28T17:46:53.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.004+0000 7f7d75e94640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7015bad0 con 0x7f7d700c5a60 2023-11-28T17:46:53.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.005+0000 7f7d6effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d70151be0 0x7f7d701f0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:53.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.005+0000 7f7d6effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d70151be0 0x7f7d701f0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50976/0 (socket says 172.21.15.98:50976) 2023-11-28T17:46:53.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.005+0000 7f7d6effd640 1 -- 172.21.15.98:0/2118512355 learned_addr learned my addr 172.21.15.98:0/2118512355 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:53.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.005+0000 7f7d6f7fe640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d700c5a60 0x7f7d701f3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:53.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.005+0000 7f7d6ffff640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d701584d0 0x7f7d7014fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:53.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.005+0000 7f7d6effd640 1 -- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d700c5a60 msgr2=0x7f7d701f3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:53.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.005+0000 7f7d6effd640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d700c5a60 0x7f7d701f3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.005+0000 7f7d6effd640 1 -- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d701584d0 msgr2=0x7f7d7014fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:53.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.005+0000 7f7d6effd640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d701584d0 0x7f7d7014fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.005+0000 7f7d6effd640 1 -- 172.21.15.98:0/2118512355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d68009b30 con 0x7f7d70151be0 2023-11-28T17:46:53.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.005+0000 7f7d6effd640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d70151be0 0x7f7d701f0840 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7d6400c3e0 tx=0x7f7d6400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:53.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.006+0000 7f7d57fff640 1 -- 172.21.15.98:0/2118512355 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d640035f0 con 0x7f7d70151be0 2023-11-28T17:46:53.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.006+0000 7f7d57fff640 1 -- 172.21.15.98:0/2118512355 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d64003750 con 0x7f7d70151be0 2023-11-28T17:46:53.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.006+0000 7f7d57fff640 1 -- 172.21.15.98:0/2118512355 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d64003b20 con 0x7f7d70151be0 2023-11-28T17:46:53.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.006+0000 7f7d75e94640 1 -- 172.21.15.98:0/2118512355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d7015bd90 con 0x7f7d70151be0 2023-11-28T17:46:53.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.006+0000 7f7d75e94640 1 -- 172.21.15.98:0/2118512355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d701ef960 con 0x7f7d70151be0 2023-11-28T17:46:53.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.007+0000 7f7d75e94640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d701f0060 0x7f7d7014eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:53.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.007+0000 7f7d75e94640 1 -- 172.21.15.98:0/2118512355 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d700744d0 con 0x7f7d701f0060 2023-11-28T17:46:53.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.007+0000 7f7d6f7fe640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d701f0060 0x7f7d7014eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:53.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.008+0000 7f7d57fff640 1 -- 172.21.15.98:0/2118512355 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f7d64003c80 con 0x7f7d70151be0 2023-11-28T17:46:53.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.008+0000 7f7d6f7fe640 1 -- 172.21.15.98:0/2118512355 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d70150e90 con 0x7f7d701f0060 2023-11-28T17:46:53.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.009+0000 7f7d6f7fe640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d701f0060 0x7f7d7014eab0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000f4a0 tx=0x7f7d6000f960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:53.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.009+0000 7f7d57fff640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7d5c06b2a0 0x7f7d5c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:53.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.009+0000 7f7d57fff640 1 -- 172.21.15.98:0/2118512355 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7d64089320 con 0x7f7d70151be0 2023-11-28T17:46:53.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.009+0000 7f7d6ffff640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7d5c06b2a0 0x7f7d5c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:53.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.009+0000 7f7d57fff640 1 -- 172.21.15.98:0/2118512355 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d60004020 con 0x7f7d701f0060 2023-11-28T17:46:53.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.010+0000 7f7d57fff640 1 -- 172.21.15.98:0/2118512355 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f7d60004180 con 0x7f7d701f0060 2023-11-28T17:46:53.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.010+0000 7f7d57fff640 1 -- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d701f0060 msgr2=0x7f7d7014eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:53.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.010+0000 7f7d57fff640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d701f0060 0x7f7d7014eab0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000f4a0 tx=0x7f7d6000f960 comp rx=0 tx=0).stop 2023-11-28T17:46:53.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.010+0000 7f7d6ffff640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7d5c06b2a0 0x7f7d5c06d760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7d68035000 tx=0x7f7d68030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:53.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.010+0000 7f7d57fff640 1 -- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d701f0060 msgr2=0x7f7d7014eab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:53.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.010+0000 7f7d57fff640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d701f0060 0x7f7d7014eab0 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000f4a0 tx=0x7f7d6000f960 comp rx=0 tx=0).stop 2023-11-28T17:46:53.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.018+0000 7f7d75e94640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d70152390 0x7f7d7014eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:53.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.018+0000 7f7d75e94640 1 -- 172.21.15.98:0/2118512355 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d701efb50 con 0x7f7d70152390 2023-11-28T17:46:53.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.018+0000 7f7d6f7fe640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d70152390 0x7f7d7014eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:53.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.018+0000 7f7d6f7fe640 1 -- 172.21.15.98:0/2118512355 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7d70158940 con 0x7f7d70152390 2023-11-28T17:46:53.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.019+0000 7f7d6f7fe640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d70152390 0x7f7d7014eff0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000f4a0 tx=0x7f7d600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:53.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.020+0000 7f7d57fff640 1 -- 172.21.15.98:0/2118512355 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d60014e50 con 0x7f7d70152390 2023-11-28T17:46:53.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.020+0000 7f7d57fff640 1 -- 172.21.15.98:0/2118512355 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4976 (secure 0 0 0) 0x7f7d60003610 con 0x7f7d70152390 2023-11-28T17:46:53.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.020+0000 7f7d57fff640 1 -- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d70152390 msgr2=0x7f7d7014eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:53.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.020+0000 7f7d57fff640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d70152390 0x7f7d7014eff0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000f4a0 tx=0x7f7d600046c0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.020+0000 7f7d57fff640 1 -- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d70152390 msgr2=0x7f7d7014eff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:46:53.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.020+0000 7f7d57fff640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7d70152390 0x7f7d7014eff0 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000f4a0 tx=0x7f7d600046c0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.023 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:46:53.023 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:46:53.023 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:46:53.023 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:46:53.023 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 60, 2023-11-28T17:46:53.023 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:46:53.023 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:46:53.023 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:46:53.024 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:46:53.024 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:53.024 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 12, 2023-11-28T17:46:53.024 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:53.024 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:46:53.024 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:46:53.024 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:53.024 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:53.024 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:53.024 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:53.024 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:53.024 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:53.024 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:53.025 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:53.025 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:53.025 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:53.025 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:53.025 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:46:53.025 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:46:53.025 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:53.025 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:53.025 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:53.025 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:53.025 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:53.025 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:53.026 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:53.026 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:53.026 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:53.026 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:53.026 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:53.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:53.026 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:46:53.026 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:46:53.026 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:46:53.026 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:46:53.026 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:46:53.026 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:46:53.027 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:53.027 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:46:53.027 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:46:53.027 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:46:53.027 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:46:53.027 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:46:53.027 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:46:53.027 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:46:53.027 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:46:53.027 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:46:53.027 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:46:53.027 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:46:53.028 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:46:53.028 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:46:53.028 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:46:53.028 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:46:53.028 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:46:53.028 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:46:53.028 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:46:53.028 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:46:53.028 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:46:53.028 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:53.028 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:46:53.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:53.028 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:46:53.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:53.029 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:46:53.029 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:46:53.029 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:53.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:53.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:53.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:53.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:46:53.029 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:53.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:53.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:53.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:53.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:46:53.030 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:53.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:53.030 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:53.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:53.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:46:53.030 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:46:53.030 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:53.030 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:53.030 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:53.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:53.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:53.031 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:46:53.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:46:53.031 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:53.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:53.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:53.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:53.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:46:53.031 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:53.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:53.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:53.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:53.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:46:53.032 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:53.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:53.032 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:53.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:53.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:46:53.032 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:46:53.032 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:53.032 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:53.032 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:53.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:53.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:53.032 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:46:53.032 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:46:53.033 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:46:53.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:46:53.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:53.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:46:53.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:46:53.033 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:53.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:53.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:53.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:46:53.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:46:53.033 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:46:53.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:53.033 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:53.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:53.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:46:53.034 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:46:53.034 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:46:53.034 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:46:53.034 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:46:53.034 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:53.034 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:53.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:53.034 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:46:53.034 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:46:53.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:53.035 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:53.035 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:53.035 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:46:53.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:53.035 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:53.035 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:46:53.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:53.035 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:53.035 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:53.035 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:46:53.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:53.035 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:53.035 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:46:53.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:53.036 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:53.036 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:53.036 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 8 2023-11-28T17:46:53.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:53.036 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:53.036 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:46:53.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:53.036 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:53.036 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:53.036 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:46:53.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:53.036 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:46:53.037 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:46:53.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:46:53.037 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:46:53.037 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:46:53.037 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:46:53.037 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:46:53.037 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:46:53.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:46:53.037 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:46:53.037 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:46:53.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.021+0000 7f7d75e94640 1 -- 172.21.15.98:0/2118512355 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7d5c06b2a0 msgr2=0x7f7d5c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:53.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.021+0000 7f7d75e94640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7d5c06b2a0 0x7f7d5c06d760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7d68035000 tx=0x7f7d68030000 comp rx=0 tx=0).stop 2023-11-28T17:46:53.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.021+0000 7f7d75e94640 1 -- 172.21.15.98:0/2118512355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d70151be0 msgr2=0x7f7d701f0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:53.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.021+0000 7f7d75e94640 1 --2- 172.21.15.98:0/2118512355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d70151be0 0x7f7d701f0840 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7d6400c3e0 tx=0x7f7d6400c8a0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.021+0000 7f7d6ffff640 1 -- 172.21.15.98:0/2118512355 reap_dead start 2023-11-28T17:46:53.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.021+0000 7f7d6ffff640 1 -- 172.21.15.98:0/2118512355 reap_dead start 2023-11-28T17:46:53.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.021+0000 7f7d75e94640 1 -- 172.21.15.98:0/2118512355 shutdown_connections 2023-11-28T17:46:53.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.021+0000 7f7d75e94640 1 -- 172.21.15.98:0/2118512355 >> 172.21.15.98:0/2118512355 conn(0x7f7d7006c0f0 msgr2=0x7f7d700c27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:53.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.021+0000 7f7d75e94640 1 -- 172.21.15.98:0/2118512355 shutdown_connections 2023-11-28T17:46:53.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.021+0000 7f7d75e94640 1 -- 172.21.15.98:0/2118512355 wait complete. 2023-11-28T17:46:53.040 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:46:53.040 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2023-11-28T17:46:53.040 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-11-28T17:46:53.040 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2023-11-28T17:46:53.040 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-11-28T17:46:53.041 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-28T17:46:53.041 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:46:53.079 INFO:tasks.ceph.mon.c:Stopped 2023-11-28T17:46:53.079 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-11-28T17:46:53.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:46:53.142 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~121s 2023-11-28T17:46:53.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.280+0000 7f7101215640 1 Processor -- start 2023-11-28T17:46:53.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.280+0000 7f7101215640 1 -- start start 2023-11-28T17:46:53.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.281+0000 7f7101215640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70fc0c29b0 0x7f70fc1518a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:53.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.281+0000 7f7101215640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70fc0c4570 con 0x7f70fc06f800 2023-11-28T17:46:53.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.281+0000 7f7101215640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70fc157be0 con 0x7f70fc0c25d0 2023-11-28T17:46:53.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.281+0000 7f7101215640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70fc1579d0 con 0x7f70fc0c29b0 2023-11-28T17:46:53.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.281+0000 7f70f259c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70fc0c29b0 0x7f70fc1518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:53.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.281+0000 7f70f259c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70fc0c29b0 0x7f70fc1518a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51056/0 (socket says 172.21.15.98:51056) 2023-11-28T17:46:53.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.281+0000 7f70f259c640 1 -- 172.21.15.98:0/3606519549 learned_addr learned my addr 172.21.15.98:0/3606519549 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:53.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.282+0000 7f70f259c640 1 -- 172.21.15.98:0/3606519549 >> v1:172.21.15.113:6789/0 conn(0x7f70fc0c25d0 legacy=0x7f70fc06e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:53.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.282+0000 7f70f259c640 1 -- 172.21.15.98:0/3606519549 >> v1:172.21.15.98:6789/0 conn(0x7f70fc06f800 legacy=0x7f70fc0c8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:53.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.282+0000 7f70f259c640 1 -- 172.21.15.98:0/3606519549 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc157ec0 con 0x7f70fc0c29b0 2023-11-28T17:46:53.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.283+0000 7f70f259c640 1 --2- 172.21.15.98:0/3606519549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc0c29b0 0x7f70fc1518a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009870 tx=0x7f70e402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b92fb0c6ef4b785 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:53.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.283+0000 7f70fa59c640 1 -- 172.21.15.98:0/3606519549 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e4030860 con 0x7f70fc0c29b0 2023-11-28T17:46:53.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.283+0000 7f70fa59c640 1 -- 172.21.15.98:0/3606519549 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e40309c0 con 0x7f70fc0c29b0 2023-11-28T17:46:53.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.283+0000 7f70fa59c640 1 -- 172.21.15.98:0/3606519549 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e4030d50 con 0x7f70fc0c29b0 2023-11-28T17:46:53.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.284+0000 7f7101215640 1 -- 172.21.15.98:0/3606519549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc0c29b0 msgr2=0x7f70fc1518a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:53.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.284+0000 7f7101215640 1 --2- 172.21.15.98:0/3606519549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc0c29b0 0x7f70fc1518a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009870 tx=0x7f70e402b9a0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.284+0000 7f7101215640 1 -- 172.21.15.98:0/3606519549 shutdown_connections 2023-11-28T17:46:53.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.284+0000 7f7101215640 1 --2- 172.21.15.98:0/3606519549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc0c29b0 0x7f70fc1518a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.284+0000 7f7101215640 1 -- 172.21.15.98:0/3606519549 >> 172.21.15.98:0/3606519549 conn(0x7f70fc066990 msgr2=0x7f70fc0700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:53.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.284+0000 7f7101215640 1 -- 172.21.15.98:0/3606519549 shutdown_connections 2023-11-28T17:46:53.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.284+0000 7f7101215640 1 -- 172.21.15.98:0/3606519549 wait complete. 2023-11-28T17:46:53.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.286+0000 7f7101215640 1 Processor -- start 2023-11-28T17:46:53.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.286+0000 7f7101215640 1 -- start start 2023-11-28T17:46:53.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f7101215640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70fc06f800 0x7f70fc1ef8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:53.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f7101215640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc0c25d0 0x7f70fc148c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:53.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f7101215640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f70fc0c29b0 0x7f70fc149190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f7101215640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc0c7560 con 0x7f70fc0c25d0 2023-11-28T17:46:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f7101215640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc1496d0 con 0x7f70fc06f800 2023-11-28T17:46:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f7101215640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc0ca710 con 0x7f70fc0c29b0 2023-11-28T17:46:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f70fb59e640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f70fc0c29b0 msgr2=0x7f70fc149190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:46:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f70f259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc0c25d0 0x7f70fc148c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f70fb59e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f70fc0c29b0 0x7f70fc149190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f70f259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc0c25d0 0x7f70fc148c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51058/0 (socket says 172.21.15.98:51058) 2023-11-28T17:46:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f70f259c640 1 -- 172.21.15.98:0/3201363606 learned_addr learned my addr 172.21.15.98:0/3201363606 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f70fad9d640 1 --2- 172.21.15.98:0/3201363606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70fc06f800 0x7f70fc1ef8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f70f259c640 1 -- 172.21.15.98:0/3201363606 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f70fc0c29b0 msgr2=0x7f70fc149190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:46:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f70f259c640 1 --2- 172.21.15.98:0/3201363606 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f70fc0c29b0 0x7f70fc149190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f70f259c640 1 -- 172.21.15.98:0/3201363606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70fc06f800 msgr2=0x7f70fc1ef8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:53.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f70f259c640 1 --2- 172.21.15.98:0/3201363606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70fc06f800 0x7f70fc1ef8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.287+0000 7f70f259c640 1 -- 172.21.15.98:0/3201363606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e40094d0 con 0x7f70fc0c25d0 2023-11-28T17:46:53.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.288+0000 7f70f259c640 1 --2- 172.21.15.98:0/3201363606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc0c25d0 0x7f70fc148c50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f70e4000c00 tx=0x7f70e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:53.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.288+0000 7f70f3fff640 1 -- 172.21.15.98:0/3201363606 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e4030810 con 0x7f70fc0c25d0 2023-11-28T17:46:53.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.288+0000 7f70f3fff640 1 -- 172.21.15.98:0/3201363606 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e4043400 con 0x7f70fc0c25d0 2023-11-28T17:46:53.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.288+0000 7f70f3fff640 1 -- 172.21.15.98:0/3201363606 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e40437b0 con 0x7f70fc0c25d0 2023-11-28T17:46:53.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.288+0000 7f7101215640 1 -- 172.21.15.98:0/3201363606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70fc0ca430 con 0x7f70fc0c25d0 2023-11-28T17:46:53.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.288+0000 7f7101215640 1 -- 172.21.15.98:0/3201363606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70fc0c5270 con 0x7f70fc0c25d0 2023-11-28T17:46:53.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.289+0000 7f7101215640 1 -- 172.21.15.98:0/3201363606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70fc075bb0 con 0x7f70fc0c25d0 2023-11-28T17:46:53.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.290+0000 7f70f3fff640 1 -- 172.21.15.98:0/3201363606 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f70e4043910 con 0x7f70fc0c25d0 2023-11-28T17:46:53.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.291+0000 7f70f3fff640 1 --2- 172.21.15.98:0/3201363606 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f70dc06b250 0x7f70dc06d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:53.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.291+0000 7f70f3fff640 1 -- 172.21.15.98:0/3201363606 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f70e403e070 con 0x7f70fc0c25d0 2023-11-28T17:46:53.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.291+0000 7f70fad9d640 1 --2- 172.21.15.98:0/3201363606 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f70dc06b250 0x7f70dc06d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:53.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.292+0000 7f70fad9d640 1 --2- 172.21.15.98:0/3201363606 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f70dc06b250 0x7f70dc06d710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f70ec005520 tx=0x7f70ec009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.295+0000 7f70f3fff640 1 -- 172.21.15.98:0/3201363606 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f70e4077e50 con 0x7f70fc0c25d0 2023-11-28T17:46:53.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.555+0000 7f7101215640 1 -- 172.21.15.98:0/3201363606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f70fc074f60 con 0x7f70fc0c25d0 2023-11-28T17:46:53.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.557+0000 7f70f3fff640 1 -- 172.21.15.98:0/3201363606 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f70e4034620 con 0x7f70fc0c25d0 2023-11-28T17:46:53.559 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:46:53.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.559+0000 7f7101215640 1 -- 172.21.15.98:0/3201363606 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f70dc06b250 msgr2=0x7f70dc06d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:53.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.559+0000 7f7101215640 1 --2- 172.21.15.98:0/3201363606 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f70dc06b250 0x7f70dc06d710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f70ec005520 tx=0x7f70ec009040 comp rx=0 tx=0).stop 2023-11-28T17:46:53.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.560+0000 7f7101215640 1 -- 172.21.15.98:0/3201363606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc0c25d0 msgr2=0x7f70fc148c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:53.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.560+0000 7f7101215640 1 --2- 172.21.15.98:0/3201363606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc0c25d0 0x7f70fc148c50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f70e4000c00 tx=0x7f70e4004730 comp rx=0 tx=0).stop 2023-11-28T17:46:53.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.560+0000 7f7101215640 1 -- 172.21.15.98:0/3201363606 shutdown_connections 2023-11-28T17:46:53.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.560+0000 7f7101215640 1 --2- 172.21.15.98:0/3201363606 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f70fc0c29b0 0x7f70fc149190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.560+0000 7f7101215640 1 --2- 172.21.15.98:0/3201363606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70fc0c25d0 0x7f70fc148c50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.560+0000 7f7101215640 1 --2- 172.21.15.98:0/3201363606 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f70dc06b250 0x7f70dc06d710 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.560+0000 7f7101215640 1 --2- 172.21.15.98:0/3201363606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70fc06f800 0x7f70fc1ef8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:53.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.560+0000 7f7101215640 1 -- 172.21.15.98:0/3201363606 >> 172.21.15.98:0/3201363606 conn(0x7f70fc066990 msgr2=0x7f70fc150870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:53.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.560+0000 7f7101215640 1 -- 172.21.15.98:0/3201363606 shutdown_connections 2023-11-28T17:46:53.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:53.560+0000 7f7101215640 1 -- 172.21.15.98:0/3201363606 wait complete. 2023-11-28T17:46:56.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:46:56.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.808+0000 7f2855ff4640 1 Processor -- start 2023-11-28T17:46:56.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.808+0000 7f2855ff4640 1 -- start start 2023-11-28T17:46:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.810+0000 7f2855ff4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f28500ca9b0 0x7f285006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.810+0000 7f2855ff4640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28500c7cf0 0x7f2850159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.810+0000 7f2855ff4640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285006c830 con 0x7f2850153e90 2023-11-28T17:46:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.810+0000 7f2855ff4640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28500c9cc0 con 0x7f28500ca9b0 2023-11-28T17:46:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.810+0000 7f2855ff4640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28500c90d0 con 0x7f28500c7cf0 2023-11-28T17:46:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.810+0000 7f28557f3640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28500c7cf0 msgr2=0x7f2850159dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:46:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.810+0000 7f28557f3640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28500c7cf0 0x7f2850159dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.810+0000 7f284ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f28500ca9b0 0x7f285006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.810+0000 7f284ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f28500ca9b0 0x7f285006bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51068/0 (socket says 172.21.15.98:51068) 2023-11-28T17:46:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.810+0000 7f284ffff640 1 -- 172.21.15.98:0/2776226422 learned_addr learned my addr 172.21.15.98:0/2776226422 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.810+0000 7f284ffff640 1 -- 172.21.15.98:0/2776226422 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28500c7cf0 msgr2=0x7f2850159dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:46:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.810+0000 7f284ffff640 1 --2- 172.21.15.98:0/2776226422 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28500c7cf0 0x7f2850159dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.810+0000 7f284ffff640 1 -- 172.21.15.98:0/2776226422 >> v1:172.21.15.113:6789/0 conn(0x7f2850153e90 legacy=0x7f28500701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:46:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.810+0000 7f284ffff640 1 -- 172.21.15.98:0/2776226422 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28500c8160 con 0x7f28500ca9b0 2023-11-28T17:46:56.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.812+0000 7f284ffff640 1 --2- 172.21.15.98:0/2776226422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28500ca9b0 0x7f285006bf50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f283c009870 tx=0x7f283c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df076c5b449b9a36 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:56.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.812+0000 7f284f7fe640 1 -- 172.21.15.98:0/2776226422 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c031870 con 0x7f28500ca9b0 2023-11-28T17:46:56.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.812+0000 7f284f7fe640 1 -- 172.21.15.98:0/2776226422 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283c0319d0 con 0x7f28500ca9b0 2023-11-28T17:46:56.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.812+0000 7f284f7fe640 1 -- 172.21.15.98:0/2776226422 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c031d60 con 0x7f28500ca9b0 2023-11-28T17:46:56.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.812+0000 7f2855ff4640 1 -- 172.21.15.98:0/2776226422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28500ca9b0 msgr2=0x7f285006bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:56.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.812+0000 7f2855ff4640 1 --2- 172.21.15.98:0/2776226422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28500ca9b0 0x7f285006bf50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f283c009870 tx=0x7f283c02b910 comp rx=0 tx=0).stop 2023-11-28T17:46:56.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.813+0000 7f2855ff4640 1 -- 172.21.15.98:0/2776226422 shutdown_connections 2023-11-28T17:46:56.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.813+0000 7f2855ff4640 1 --2- 172.21.15.98:0/2776226422 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28500c7cf0 0x7f2850159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:56.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.813+0000 7f2855ff4640 1 --2- 172.21.15.98:0/2776226422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28500ca9b0 0x7f285006bf50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:56.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.813+0000 7f2855ff4640 1 -- 172.21.15.98:0/2776226422 >> 172.21.15.98:0/2776226422 conn(0x7f28500722b0 msgr2=0x7f2850072690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:56.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.813+0000 7f2855ff4640 1 -- 172.21.15.98:0/2776226422 shutdown_connections 2023-11-28T17:46:56.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.813+0000 7f2855ff4640 1 -- 172.21.15.98:0/2776226422 wait complete. 2023-11-28T17:46:56.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.815+0000 7f2855ff4640 1 Processor -- start 2023-11-28T17:46:56.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.815+0000 7f2855ff4640 1 -- start start 2023-11-28T17:46:56.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f2855ff4640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28500c7cf0 0x7f2850158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:56.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f2855ff4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28500ca9b0 0x7f2850159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:56.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f2855ff4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2850153e90 0x7f2850150730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:56.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f2855ff4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28501ef2f0 con 0x7f2850153e90 2023-11-28T17:46:56.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f2855ff4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28500ad8c0 con 0x7f28500ca9b0 2023-11-28T17:46:56.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f2855ff4640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28500b3580 con 0x7f28500c7cf0 2023-11-28T17:46:56.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f2854ff2640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28500c7cf0 msgr2=0x7f2850158b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:46:56.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f2854ff2640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28500c7cf0 0x7f2850158b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:46:56.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f28557f3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2850153e90 0x7f2850150730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:56.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f28557f3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2850153e90 0x7f2850150730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51070/0 (socket says 172.21.15.98:51070) 2023-11-28T17:46:56.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f28557f3640 1 -- 172.21.15.98:0/1749894921 learned_addr learned my addr 172.21.15.98:0/1749894921 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:46:56.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f284ffff640 1 --2- 172.21.15.98:0/1749894921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28500ca9b0 0x7f2850159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:56.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f28557f3640 1 -- 172.21.15.98:0/1749894921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28500c7cf0 msgr2=0x7f2850158b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:46:56.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f28557f3640 1 --2- 172.21.15.98:0/1749894921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28500c7cf0 0x7f2850158b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:56.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f28557f3640 1 -- 172.21.15.98:0/1749894921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28500ca9b0 msgr2=0x7f2850159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:56.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f28557f3640 1 --2- 172.21.15.98:0/1749894921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28500ca9b0 0x7f2850159050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:56.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.816+0000 7f28557f3640 1 -- 172.21.15.98:0/1749894921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283c0094d0 con 0x7f2850153e90 2023-11-28T17:46:56.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.817+0000 7f28557f3640 1 --2- 172.21.15.98:0/1749894921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2850153e90 0x7f2850150730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f284800b990 tx=0x7f284800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:56.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.817+0000 7f284d7fa640 1 -- 172.21.15.98:0/1749894921 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284800cb20 con 0x7f2850153e90 2023-11-28T17:46:56.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.817+0000 7f284d7fa640 1 -- 172.21.15.98:0/1749894921 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284800cc80 con 0x7f2850153e90 2023-11-28T17:46:56.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.817+0000 7f284d7fa640 1 -- 172.21.15.98:0/1749894921 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28480156b0 con 0x7f2850153e90 2023-11-28T17:46:56.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.817+0000 7f2855ff4640 1 -- 172.21.15.98:0/1749894921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28500b32e0 con 0x7f2850153e90 2023-11-28T17:46:56.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.817+0000 7f2855ff4640 1 -- 172.21.15.98:0/1749894921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28500adad0 con 0x7f2850153e90 2023-11-28T17:46:56.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.818+0000 7f2855ff4640 1 -- 172.21.15.98:0/1749894921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2850074db0 con 0x7f2850153e90 2023-11-28T17:46:56.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.819+0000 7f284d7fa640 1 -- 172.21.15.98:0/1749894921 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2848012070 con 0x7f2850153e90 2023-11-28T17:46:56.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.820+0000 7f284d7fa640 1 --2- 172.21.15.98:0/1749894921 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f283406b250 0x7f283406d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:46:56.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.820+0000 7f284d7fa640 1 -- 172.21.15.98:0/1749894921 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2848089d70 con 0x7f2850153e90 2023-11-28T17:46:56.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.820+0000 7f2854ff2640 1 --2- 172.21.15.98:0/1749894921 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f283406b250 0x7f283406d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:46:56.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.821+0000 7f2854ff2640 1 --2- 172.21.15.98:0/1749894921 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f283406b250 0x7f283406d710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2844003680 tx=0x7f2844007a10 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:46:56.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:56.825+0000 7f284d7fa640 1 -- 172.21.15.98:0/1749894921 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2848054a30 con 0x7f2850153e90 2023-11-28T17:46:57.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:57.083+0000 7f2855ff4640 1 -- 172.21.15.98:0/1749894921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f28500c8370 con 0x7f2850153e90 2023-11-28T17:46:57.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:57.084+0000 7f284d7fa640 1 -- 172.21.15.98:0/1749894921 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f284801a070 con 0x7f2850153e90 2023-11-28T17:46:57.085 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:46:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:57.086+0000 7f2855ff4640 1 -- 172.21.15.98:0/1749894921 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f283406b250 msgr2=0x7f283406d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:57.086+0000 7f2855ff4640 1 --2- 172.21.15.98:0/1749894921 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f283406b250 0x7f283406d710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2844003680 tx=0x7f2844007a10 comp rx=0 tx=0).stop 2023-11-28T17:46:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:57.086+0000 7f2855ff4640 1 -- 172.21.15.98:0/1749894921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2850153e90 msgr2=0x7f2850150730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:46:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:57.086+0000 7f2855ff4640 1 --2- 172.21.15.98:0/1749894921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2850153e90 0x7f2850150730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f284800b990 tx=0x7f284800be50 comp rx=0 tx=0).stop 2023-11-28T17:46:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:57.087+0000 7f2855ff4640 1 -- 172.21.15.98:0/1749894921 shutdown_connections 2023-11-28T17:46:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:57.087+0000 7f2855ff4640 1 --2- 172.21.15.98:0/1749894921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2850153e90 0x7f2850150730 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:57.087+0000 7f2855ff4640 1 --2- 172.21.15.98:0/1749894921 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f283406b250 0x7f283406d710 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:57.087+0000 7f2855ff4640 1 --2- 172.21.15.98:0/1749894921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28500ca9b0 0x7f2850159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:57.087+0000 7f2855ff4640 1 --2- 172.21.15.98:0/1749894921 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28500c7cf0 0x7f2850158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:46:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:57.087+0000 7f2855ff4640 1 -- 172.21.15.98:0/1749894921 >> 172.21.15.98:0/1749894921 conn(0x7f28500722b0 msgr2=0x7f28500cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:46:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:57.087+0000 7f2855ff4640 1 -- 172.21.15.98:0/1749894921 shutdown_connections 2023-11-28T17:46:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:46:57.087+0000 7f2855ff4640 1 -- 172.21.15.98:0/1749894921 wait complete. 2023-11-28T17:46:59.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~128s 2023-11-28T17:47:00.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:47:00.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:00.372+0000 7fb8e0333640 1 Processor -- start 2023-11-28T17:47:00.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:00.372+0000 7fb8e0333640 1 -- start start 2023-11-28T17:47:00.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:00.374+0000 7fb8e0333640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8151d80 0x7fb8d8157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:00.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:00.374+0000 7fb8e0333640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d815b150 con 0x7fb8d80c4910 2023-11-28T17:47:00.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:00.374+0000 7fb8e0333640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d806e1f0 con 0x7fb8d815bc10 2023-11-28T17:47:00.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:00.374+0000 7fb8e0333640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d806e4d0 con 0x7fb8d8151d80 2023-11-28T17:47:00.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:00.374+0000 7fb8de0cf640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8151d80 msgr2=0x7fb8d8157b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:00.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:00.374+0000 7fb8de0cf640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8151d80 0x7fb8d8157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:00.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:00.374+0000 7fb8de8d0640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb8d80c4910 0x7fb8d81624e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51072/0 (socket says 172.21.15.98:51072) 2023-11-28T17:47:00.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:00.374+0000 7fb8de8d0640 1 -- 172.21.15.98:0/2053808499 learned_addr learned my addr 172.21.15.98:0/2053808499 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:47:00.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:00.575+0000 7fb8de0cf640 1 -- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8151d80 msgr2=0x7fb8d8157b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:00.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:00.575+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8151d80 0x7fb8d8157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:47:00.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:00.976+0000 7fb8de0cf640 1 -- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8151d80 msgr2=0x7fb8d8157b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:00.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:00.976+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8151d80 0x7fb8d8157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:47:01.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:01.777+0000 7fb8de0cf640 1 -- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8151d80 msgr2=0x7fb8d8157b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:01.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:01.777+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8151d80 0x7fb8d8157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:47:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.374+0000 7fb8cffff640 1 -- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8151d80 msgr2=0x7fb8d8157b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.374+0000 7fb8cffff640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8151d80 0x7fb8d8157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.374+0000 7fb8cffff640 1 -- 172.21.15.98:0/2053808499 >> v1:172.21.15.113:6789/0 conn(0x7fb8d815bc10 legacy=0x7fb8d80c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.374+0000 7fb8cffff640 1 -- 172.21.15.98:0/2053808499 >> v1:172.21.15.98:6789/0 conn(0x7fb8d80c4910 legacy=0x7fb8d81624e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.374+0000 7fb8cffff640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 0x7fb8bc005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.374+0000 7fb8cffff640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 0x7fb8bc008710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.374+0000 7fb8cffff640 1 -- 172.21.15.98:0/2053808499 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d806e4d0 con 0x7fb8bc003710 2023-11-28T17:47:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.374+0000 7fb8cffff640 1 -- 172.21.15.98:0/2053808499 --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8bc008dc0 con 0x7fb8bc006270 2023-11-28T17:47:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.375+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 0x7fb8bc005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.375+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 0x7fb8bc008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.375+0000 7fb8de0cf640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 msgr2=0x7fb8bc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:47:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.375+0000 7fb8de0cf640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 msgr2=0x7fb8bc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.375+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 0x7fb8bc005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.376+0000 7fb8dd8ce640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 msgr2=0x7fb8bc008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:47:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.376+0000 7fb8dd8ce640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 msgr2=0x7fb8bc008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.376+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 0x7fb8bc008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.376+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 0x7fb8bc005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.376+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 0x7fb8bc008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:03.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.577+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 0x7fb8bc005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:03.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.577+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 0x7fb8bc008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.578+0000 7fb8dd8ce640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 msgr2=0x7fb8bc008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:47:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.578+0000 7fb8dd8ce640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 msgr2=0x7fb8bc008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.578+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 0x7fb8bc008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.578+0000 7fb8de0cf640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 msgr2=0x7fb8bc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:47:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.578+0000 7fb8de0cf640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 msgr2=0x7fb8bc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.578+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 0x7fb8bc005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:03.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.578+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 0x7fb8bc008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:47:03.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.578+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 0x7fb8bc005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:47:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.979+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 0x7fb8bc005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:03.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.979+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 0x7fb8bc008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.980+0000 7fb8dd8ce640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 msgr2=0x7fb8bc008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:47:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.980+0000 7fb8dd8ce640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 msgr2=0x7fb8bc008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.980+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 0x7fb8bc008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.980+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 0x7fb8bc008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:47:03.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.980+0000 7fb8de0cf640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 msgr2=0x7fb8bc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:47:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.980+0000 7fb8de0cf640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 msgr2=0x7fb8bc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.980+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 0x7fb8bc005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:03.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:03.980+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 0x7fb8bc005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:47:04.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:04.782+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 0x7fb8bc008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:04.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:04.782+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 0x7fb8bc005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:04.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:04.782+0000 7fb8dd8ce640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 msgr2=0x7fb8bc008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:47:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:04.782+0000 7fb8dd8ce640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 msgr2=0x7fb8bc008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:04.782+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 0x7fb8bc008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:04.782+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 0x7fb8bc008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:47:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:04.782+0000 7fb8de0cf640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 msgr2=0x7fb8bc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:47:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:04.782+0000 7fb8de0cf640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 msgr2=0x7fb8bc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:04.782+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 0x7fb8bc005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:04.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:04.782+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 0x7fb8bc005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:47:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.374+0000 7fb8cffff640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 msgr2=0x7fb8bc008710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.374+0000 7fb8cffff640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.113:3300/0 conn(0x7fb8bc006270 0x7fb8bc008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.374+0000 7fb8cffff640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 msgr2=0x7fb8bc005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:06.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.374+0000 7fb8cffff640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc003710 0x7fb8bc005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.375+0000 7fb8de8d0640 1 -- 172.21.15.98:0/2053808499 reap_dead start 2023-11-28T17:47:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.375+0000 7fb8cffff640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8bc007e20 0x7fb8bc008200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.375+0000 7fb8cffff640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 0x7fb8bc00d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.375+0000 7fb8cffff640 1 -- 172.21.15.98:0/2053808499 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d806e7b0 con 0x7fb8bc003b80 2023-11-28T17:47:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.375+0000 7fb8cffff640 1 -- 172.21.15.98:0/2053808499 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d806e4d0 con 0x7fb8bc008740 2023-11-28T17:47:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.375+0000 7fb8cffff640 1 -- 172.21.15.98:0/2053808499 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc008dc0 con 0x7fb8bc007e20 2023-11-28T17:47:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.375+0000 7fb8dd8ce640 1 -- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8bc007e20 msgr2=0x7fb8bc008200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.375+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8bc007e20 0x7fb8bc008200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:06.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.375+0000 7fb8de8d0640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 0x7fb8bc00d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:06.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.376+0000 7fb8de8d0640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 msgr2=0x7fb8bc00d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:47:06.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.376+0000 7fb8de8d0640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 msgr2=0x7fb8bc00d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:06.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.376+0000 7fb8de8d0640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 0x7fb8bc00d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:06.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.376+0000 7fb8de8d0640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 0x7fb8bc00d190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:06.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~135s 2023-11-28T17:47:06.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.576+0000 7fb8dd8ce640 1 -- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8bc007e20 msgr2=0x7fb8bc008200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:06.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.576+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8bc007e20 0x7fb8bc008200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:47:06.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.577+0000 7fb8de8d0640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 0x7fb8bc00d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:06.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.578+0000 7fb8de8d0640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 msgr2=0x7fb8bc00d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:47:06.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.578+0000 7fb8de8d0640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 msgr2=0x7fb8bc00d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:06.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.578+0000 7fb8de8d0640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 0x7fb8bc00d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:06.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.578+0000 7fb8de8d0640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 0x7fb8bc00d190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:47:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.977+0000 7fb8dd8ce640 1 -- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8bc007e20 msgr2=0x7fb8bc008200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.977+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8bc007e20 0x7fb8bc008200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:47:06.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.979+0000 7fb8de8d0640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 0x7fb8bc00d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:06.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.980+0000 7fb8de8d0640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 msgr2=0x7fb8bc00d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:47:06.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.980+0000 7fb8de8d0640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 msgr2=0x7fb8bc00d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.980+0000 7fb8de8d0640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 0x7fb8bc00d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:06.980+0000 7fb8de8d0640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 0x7fb8bc00d190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:47:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.467+0000 7fb8dd0cd640 1 -- 172.21.15.98:0/2053808499 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604437911 0 0) 0x7fb8d806e7b0 con 0x7fb8bc003b80 2023-11-28T17:47:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.467+0000 7fb8dd0cd640 1 -- 172.21.15.98:0/2053808499 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8c4003520 con 0x7fb8bc003b80 2023-11-28T17:47:07.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.468+0000 7fb8dd0cd640 1 -- 172.21.15.98:0/2053808499 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1018841194 0 0) 0x7fb8c4003520 con 0x7fb8bc003b80 2023-11-28T17:47:07.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.468+0000 7fb8dd0cd640 1 -- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8bc007e20 msgr2=0x7fb8bc008200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:07.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.468+0000 7fb8dd0cd640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8bc007e20 0x7fb8bc008200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.468+0000 7fb8dd0cd640 1 -- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 msgr2=0x7fb8bc00d190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:07.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.468+0000 7fb8dd0cd640 1 --2- 172.21.15.98:0/2053808499 >> v2:172.21.15.98:3300/0 conn(0x7fb8bc008740 0x7fb8bc00d190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.468+0000 7fb8dd0cd640 1 -- 172.21.15.98:0/2053808499 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc00e950 con 0x7fb8bc003b80 2023-11-28T17:47:07.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.468+0000 7fb8dd0cd640 1 -- 172.21.15.98:0/2053808499 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7fb8d40098d0 con 0x7fb8bc003b80 2023-11-28T17:47:07.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.468+0000 7fb8dd0cd640 1 -- 172.21.15.98:0/2053808499 >> v1:172.21.15.98:6789/0 conn(0x7fb8bc003b80 legacy=0x7fb8bc006700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.469+0000 7fb8dd0cd640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8bc007e20 0x7fb8c4003990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.469+0000 7fb8dd0cd640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8c4003fc0 0x7fb8c40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.469+0000 7fb8dd0cd640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8c4004d00 0x7fb8c4005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.469+0000 7fb8dd0cd640 1 -- 172.21.15.98:0/2053808499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d806e4d0 con 0x7fb8c4004d00 2023-11-28T17:47:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.469+0000 7fb8dd0cd640 1 -- 172.21.15.98:0/2053808499 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc008dc0 con 0x7fb8c4003fc0 2023-11-28T17:47:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.469+0000 7fb8dd0cd640 1 -- 172.21.15.98:0/2053808499 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c400bc90 con 0x7fb8bc007e20 2023-11-28T17:47:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.469+0000 7fb8de0cf640 1 -- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8bc007e20 msgr2=0x7fb8c4003990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.469+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8bc007e20 0x7fb8c4003990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.470+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8c4003fc0 0x7fb8c40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.470+0000 7fb8de8d0640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8c4004d00 0x7fb8c4005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:07.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.471+0000 7fb8dd8ce640 1 -- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8bc007e20 msgr2=0x7fb8c4003990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:07.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.471+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8bc007e20 0x7fb8c4003990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.471+0000 7fb8dd8ce640 1 -- 172.21.15.98:0/2053808499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8c4004d00 msgr2=0x7fb8c4005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:07.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.471+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8c4004d00 0x7fb8c4005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.471+0000 7fb8dd8ce640 1 -- 172.21.15.98:0/2053808499 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c4003520 con 0x7fb8c4003fc0 2023-11-28T17:47:07.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.471+0000 7fb8de8d0640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8c4004d00 0x7fb8c4005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:47:07.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.474+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8c4003fc0 0x7fb8c40047c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80017c0 tx=0x7fb8c40057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=537e2329ae7811a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:07.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.475+0000 7fb8dd0cd640 1 -- 172.21.15.98:0/2053808499 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c802cd20 con 0x7fb8c4003fc0 2023-11-28T17:47:07.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.475+0000 7fb8dd0cd640 1 -- 172.21.15.98:0/2053808499 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c80050d0 con 0x7fb8c4003fc0 2023-11-28T17:47:07.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.475+0000 7fb8dd0cd640 1 -- 172.21.15.98:0/2053808499 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c802cea0 con 0x7fb8c4003fc0 2023-11-28T17:47:07.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.476+0000 7fb8e0333640 1 -- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8c4003fc0 msgr2=0x7fb8c40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:07.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.476+0000 7fb8e0333640 1 --2- 172.21.15.98:0/2053808499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8c4003fc0 0x7fb8c40047c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80017c0 tx=0x7fb8c40057f0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.476+0000 7fb8de8d0640 1 -- 172.21.15.98:0/2053808499 reap_dead start 2023-11-28T17:47:07.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.476+0000 7fb8e0333640 1 -- 172.21.15.98:0/2053808499 shutdown_connections 2023-11-28T17:47:07.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.476+0000 7fb8e0333640 1 -- 172.21.15.98:0/2053808499 >> 172.21.15.98:0/2053808499 conn(0x7fb8d806a4d0 msgr2=0x7fb8d806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:07.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.476+0000 7fb8e0333640 1 -- 172.21.15.98:0/2053808499 shutdown_connections 2023-11-28T17:47:07.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.476+0000 7fb8e0333640 1 -- 172.21.15.98:0/2053808499 wait complete. 2023-11-28T17:47:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.480+0000 7fb8e0333640 1 Processor -- start 2023-11-28T17:47:07.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.480+0000 7fb8e0333640 1 -- start start 2023-11-28T17:47:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.482+0000 7fb8e0333640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8c4003fc0 0x7fb8d81ec180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.482+0000 7fb8e0333640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8d8141200 0x7fb8d8156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.482+0000 7fb8e0333640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8157390 0x7fb8d80c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.482+0000 7fb8e0333640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d814b710 con 0x7fb8c4003fc0 2023-11-28T17:47:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.482+0000 7fb8e0333640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d814b150 con 0x7fb8d8141200 2023-11-28T17:47:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.482+0000 7fb8e0333640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d814b430 con 0x7fb8d8157390 2023-11-28T17:47:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.482+0000 7fb8de8d0640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8157390 msgr2=0x7fb8d80c58f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.482+0000 7fb8de8d0640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8157390 0x7fb8d80c58f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.482+0000 7fb8de0cf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8c4003fc0 0x7fb8d81ec180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:07.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.482+0000 7fb8de0cf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8c4003fc0 0x7fb8d81ec180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57634/0 (socket says 172.21.15.98:57634) 2023-11-28T17:47:07.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.482+0000 7fb8de0cf640 1 -- 172.21.15.98:0/600678559 learned_addr learned my addr 172.21.15.98:0/600678559 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:07.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.483+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/600678559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8d8141200 0x7fb8d8156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:07.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.483+0000 7fb8de0cf640 1 -- 172.21.15.98:0/600678559 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8157390 msgr2=0x7fb8d80c58f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:47:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.483+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/600678559 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8157390 0x7fb8d80c58f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.483+0000 7fb8de0cf640 1 -- 172.21.15.98:0/600678559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8d8141200 msgr2=0x7fb8d8156e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.483+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/600678559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8d8141200 0x7fb8d8156e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.483+0000 7fb8de0cf640 1 -- 172.21.15.98:0/600678559 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d81eb980 con 0x7fb8c4003fc0 2023-11-28T17:47:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.483+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/600678559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8d8141200 0x7fb8d8156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:47:07.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.483+0000 7fb8de0cf640 1 --2- 172.21.15.98:0/600678559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8c4003fc0 0x7fb8d81ec180 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400be30 tx=0x7fb8d400a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:07.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.484+0000 7fb8ceffd640 1 -- 172.21.15.98:0/600678559 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d40167f0 con 0x7fb8c4003fc0 2023-11-28T17:47:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.484+0000 7fb8ceffd640 1 -- 172.21.15.98:0/600678559 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d4011040 con 0x7fb8c4003fc0 2023-11-28T17:47:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.484+0000 7fb8ceffd640 1 -- 172.21.15.98:0/600678559 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4016bb0 con 0x7fb8c4003fc0 2023-11-28T17:47:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.484+0000 7fb8e0333640 1 -- 172.21.15.98:0/600678559 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d81ebc60 con 0x7fb8c4003fc0 2023-11-28T17:47:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.484+0000 7fb8e0333640 1 -- 172.21.15.98:0/600678559 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d814d6b0 con 0x7fb8c4003fc0 2023-11-28T17:47:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.485+0000 7fb8e0333640 1 -- 172.21.15.98:0/600678559 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d8067bf0 con 0x7fb8c4003fc0 2023-11-28T17:47:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.486+0000 7fb8ceffd640 1 -- 172.21.15.98:0/600678559 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb8d4016d10 con 0x7fb8c4003fc0 2023-11-28T17:47:07.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.488+0000 7fb8ceffd640 1 --2- 172.21.15.98:0/600678559 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb8bc06c030 0x7fb8bc06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.488+0000 7fb8ceffd640 1 -- 172.21.15.98:0/600678559 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb8d408ae40 con 0x7fb8c4003fc0 2023-11-28T17:47:07.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.488+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/600678559 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb8bc06c030 0x7fb8bc06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:07.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.489+0000 7fb8dd8ce640 1 --2- 172.21.15.98:0/600678559 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb8bc06c030 0x7fb8bc06e4f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8035000 tx=0x7fb8c8030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:07.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.493+0000 7fb8ceffd640 1 -- 172.21.15.98:0/600678559 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8d4055d80 con 0x7fb8c4003fc0 2023-11-28T17:47:07.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.752+0000 7fb8e0333640 1 -- 172.21.15.98:0/600678559 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb8d80c6860 con 0x7fb8c4003fc0 2023-11-28T17:47:07.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.753+0000 7fb8ceffd640 1 -- 172.21.15.98:0/600678559 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7fb8d400ae60 con 0x7fb8c4003fc0 2023-11-28T17:47:07.754 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":64,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:47:07.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.755+0000 7fb8e0333640 1 -- 172.21.15.98:0/600678559 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb8bc06c030 msgr2=0x7fb8bc06e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:07.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.755+0000 7fb8e0333640 1 --2- 172.21.15.98:0/600678559 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb8bc06c030 0x7fb8bc06e4f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8035000 tx=0x7fb8c8030000 comp rx=0 tx=0).stop 2023-11-28T17:47:07.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.755+0000 7fb8e0333640 1 -- 172.21.15.98:0/600678559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8c4003fc0 msgr2=0x7fb8d81ec180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:07.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.755+0000 7fb8e0333640 1 --2- 172.21.15.98:0/600678559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8c4003fc0 0x7fb8d81ec180 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400be30 tx=0x7fb8d400a7b0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.755+0000 7fb8e0333640 1 -- 172.21.15.98:0/600678559 shutdown_connections 2023-11-28T17:47:07.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.755+0000 7fb8e0333640 1 --2- 172.21.15.98:0/600678559 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8d8157390 0x7fb8d80c58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.755+0000 7fb8e0333640 1 --2- 172.21.15.98:0/600678559 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb8bc06c030 0x7fb8bc06e4f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.755+0000 7fb8e0333640 1 --2- 172.21.15.98:0/600678559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8d8141200 0x7fb8d8156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.755+0000 7fb8e0333640 1 --2- 172.21.15.98:0/600678559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8c4003fc0 0x7fb8d81ec180 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.755+0000 7fb8e0333640 1 -- 172.21.15.98:0/600678559 >> 172.21.15.98:0/600678559 conn(0x7fb8d806a4d0 msgr2=0x7fb8d814c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:07.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.755+0000 7fb8e0333640 1 -- 172.21.15.98:0/600678559 shutdown_connections 2023-11-28T17:47:07.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.755+0000 7fb8e0333640 1 -- 172.21.15.98:0/600678559 wait complete. 2023-11-28T17:47:07.775 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-11-28T17:47:07.775 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-11-28T17:47:07.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:47:07.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.977+0000 7f1cfc7a9640 1 Processor -- start 2023-11-28T17:47:07.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.978+0000 7f1cfc7a9640 1 -- start start 2023-11-28T17:47:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.979+0000 7f1cfc7a9640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cf4070a20 0x7f1cf4154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:07.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.979+0000 7f1cfc7a9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1cf40c36a0 0x7f1cf4155310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:07.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.979+0000 7f1cfc7a9640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf4158640 con 0x7f1cf4070640 2023-11-28T17:47:07.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.979+0000 7f1cfc7a9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf40c7d60 con 0x7f1cf40c36a0 2023-11-28T17:47:07.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.979+0000 7f1cfc7a9640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf40ce300 con 0x7f1cf4070a20 2023-11-28T17:47:07.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.979+0000 7f1cf9d44640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cf4070a20 msgr2=0x7f1cf4154dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:07.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.980+0000 7f1cf9d44640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cf4070a20 0x7f1cf4154dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:07.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.980+0000 7f1cfad46640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1cf40c36a0 0x7f1cf4155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:07.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.980+0000 7f1cfad46640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1cf40c36a0 0x7f1cf4155310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57644/0 (socket says 172.21.15.98:57644) 2023-11-28T17:47:07.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.980+0000 7f1cfad46640 1 -- 172.21.15.98:0/605001049 learned_addr learned my addr 172.21.15.98:0/605001049 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:07.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.980+0000 7f1cfad46640 1 -- 172.21.15.98:0/605001049 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cf4070a20 msgr2=0x7f1cf4154dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:07.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.980+0000 7f1cfad46640 1 --2- 172.21.15.98:0/605001049 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cf4070a20 0x7f1cf4154dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.980+0000 7f1cfad46640 1 -- 172.21.15.98:0/605001049 >> v1:172.21.15.113:6789/0 conn(0x7f1cf4070640 legacy=0x7f1cf40c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:07.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.980+0000 7f1cfad46640 1 -- 172.21.15.98:0/605001049 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf40c7f50 con 0x7f1cf40c36a0 2023-11-28T17:47:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.981+0000 7f1cfad46640 1 --2- 172.21.15.98:0/605001049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf40c36a0 0x7f1cf4155310 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1cec009870 tx=0x7f1cec02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=302308f3d7b62735 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.981+0000 7f1cf9543640 1 -- 172.21.15.98:0/605001049 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec02f850 con 0x7f1cf40c36a0 2023-11-28T17:47:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.982+0000 7f1cf9543640 1 -- 172.21.15.98:0/605001049 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec02f9b0 con 0x7f1cf40c36a0 2023-11-28T17:47:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.982+0000 7f1cf9543640 1 -- 172.21.15.98:0/605001049 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec02fd40 con 0x7f1cf40c36a0 2023-11-28T17:47:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.982+0000 7f1cfc7a9640 1 -- 172.21.15.98:0/605001049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf40c36a0 msgr2=0x7f1cf4155310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.982+0000 7f1cfc7a9640 1 --2- 172.21.15.98:0/605001049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf40c36a0 0x7f1cf4155310 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1cec009870 tx=0x7f1cec02ba30 comp rx=0 tx=0).stop 2023-11-28T17:47:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.982+0000 7f1cfc7a9640 1 -- 172.21.15.98:0/605001049 shutdown_connections 2023-11-28T17:47:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.982+0000 7f1cfc7a9640 1 --2- 172.21.15.98:0/605001049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf40c36a0 0x7f1cf4155310 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.982+0000 7f1cfc7a9640 1 --2- 172.21.15.98:0/605001049 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cf4070a20 0x7f1cf4154dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.982+0000 7f1cfc7a9640 1 -- 172.21.15.98:0/605001049 >> 172.21.15.98:0/605001049 conn(0x7f1cf406e620 msgr2=0x7f1cf406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.982+0000 7f1cfc7a9640 1 -- 172.21.15.98:0/605001049 shutdown_connections 2023-11-28T17:47:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.983+0000 7f1cfc7a9640 1 -- 172.21.15.98:0/605001049 wait complete. 2023-11-28T17:47:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.984+0000 7f1cfc7a9640 1 Processor -- start 2023-11-28T17:47:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.984+0000 7f1cfc7a9640 1 -- start start 2023-11-28T17:47:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.985+0000 7f1cfc7a9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cf4070640 0x7f1cf406d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.985+0000 7f1cfc7a9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf4070a20 0x7f1cf40b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.985+0000 7f1cfc7a9640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cf40c36a0 0x7f1cf41e2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.985+0000 7f1cfc7a9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf41e2780 con 0x7f1cf4070a20 2023-11-28T17:47:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.985+0000 7f1cfc7a9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf40b1990 con 0x7f1cf4070640 2023-11-28T17:47:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.985+0000 7f1cfc7a9640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf406bd60 con 0x7f1cf40c36a0 2023-11-28T17:47:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.985+0000 7f1cfad46640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cf40c36a0 msgr2=0x7f1cf41e2c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.985+0000 7f1cfad46640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cf40c36a0 0x7f1cf41e2c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.985+0000 7f1cf9d44640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf4070a20 0x7f1cf40b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.985+0000 7f1cf9d44640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf4070a20 0x7f1cf40b1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57656/0 (socket says 172.21.15.98:57656) 2023-11-28T17:47:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.985+0000 7f1cf9d44640 1 -- 172.21.15.98:0/3870566040 learned_addr learned my addr 172.21.15.98:0/3870566040 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.986+0000 7f1cfa545640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cf4070640 0x7f1cf406d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.986+0000 7f1cf9d44640 1 -- 172.21.15.98:0/3870566040 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cf40c36a0 msgr2=0x7f1cf41e2c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:47:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.986+0000 7f1cf9d44640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1cf40c36a0 0x7f1cf41e2c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.986+0000 7f1cf9d44640 1 -- 172.21.15.98:0/3870566040 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cf4070640 msgr2=0x7f1cf406d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.986+0000 7f1cf9d44640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cf4070640 0x7f1cf406d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.986+0000 7f1cf9d44640 1 -- 172.21.15.98:0/3870566040 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec0094d0 con 0x7f1cf4070a20 2023-11-28T17:47:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.986+0000 7f1cfa545640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1cf4070640 0x7f1cf406d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:47:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.986+0000 7f1cf9d44640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf4070a20 0x7f1cf40b1f00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000b550 tx=0x7f1ce000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.986+0000 7f1cf2ffd640 1 -- 172.21.15.98:0/3870566040 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0003dc0 con 0x7f1cf4070a20 2023-11-28T17:47:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.986+0000 7f1cf2ffd640 1 -- 172.21.15.98:0/3870566040 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce0005070 con 0x7f1cf4070a20 2023-11-28T17:47:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.986+0000 7f1cfc7a9640 1 -- 172.21.15.98:0/3870566040 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf406c040 con 0x7f1cf4070a20 2023-11-28T17:47:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.986+0000 7f1cf2ffd640 1 -- 172.21.15.98:0/3870566040 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0005420 con 0x7f1cf4070a20 2023-11-28T17:47:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.987+0000 7f1cfc7a9640 1 -- 172.21.15.98:0/3870566040 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf41e2410 con 0x7f1cf4070a20 2023-11-28T17:47:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.987+0000 7f1cfc7a9640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf406c390 0x7f1cf41e3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.987+0000 7f1cfc7a9640 1 -- 172.21.15.98:0/3870566040 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf406af60 con 0x7f1cf406c390 2023-11-28T17:47:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.988+0000 7f1cfa545640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf406c390 0x7f1cf41e3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.988+0000 7f1cfa545640 1 -- 172.21.15.98:0/3870566040 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cf41582d0 con 0x7f1cf406c390 2023-11-28T17:47:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.988+0000 7f1cfa545640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf406c390 0x7f1cf41e3150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ce800d810 tx=0x7f1ce800dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.988+0000 7f1cf2ffd640 1 -- 172.21.15.98:0/3870566040 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f1ce0005580 con 0x7f1cf4070a20 2023-11-28T17:47:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.989+0000 7f1cf2ffd640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1cd806b250 0x7f1cd806d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.989+0000 7f1cf2ffd640 1 -- 172.21.15.98:0/3870566040 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1ce0089c60 con 0x7f1cf4070a20 2023-11-28T17:47:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.990+0000 7f1cfad46640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1cd806b250 0x7f1cd806d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.990+0000 7f1cf2ffd640 1 -- 172.21.15.98:0/3870566040 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8016070 con 0x7f1cf406c390 2023-11-28T17:47:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.990+0000 7f1cf2ffd640 1 -- 172.21.15.98:0/3870566040 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f1ce8003ba0 con 0x7f1cf406c390 2023-11-28T17:47:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.990+0000 7f1cf2ffd640 1 -- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf406c390 msgr2=0x7f1cf41e3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.990+0000 7f1cf2ffd640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf406c390 0x7f1cf41e3150 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ce800d810 tx=0x7f1ce800dcd0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.990+0000 7f1cf2ffd640 1 -- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf406c390 msgr2=0x7f1cf41e3150 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.990+0000 7f1cf2ffd640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf406c390 0x7f1cf41e3150 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ce800d810 tx=0x7f1ce800dcd0 comp rx=0 tx=0).stop 2023-11-28T17:47:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.990+0000 7f1cfad46640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1cd806b250 0x7f1cd806d710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1cec036000 tx=0x7f1cec039040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.997+0000 7f1cfc7a9640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf41f2d40 0x7f1cf41f3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.997+0000 7f1cfc7a9640 1 -- 172.21.15.98:0/3870566040 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf4157ff0 con 0x7f1cf41f2d40 2023-11-28T17:47:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.997+0000 7f1cfa545640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf41f2d40 0x7f1cf41f3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.997+0000 7f1cfa545640 1 -- 172.21.15.98:0/3870566040 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1cf4157ca0 con 0x7f1cf41f2d40 2023-11-28T17:47:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.998+0000 7f1cfa545640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf41f2d40 0x7f1cf41f3150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1ce800d810 tx=0x7f1ce80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.998+0000 7f1cf2ffd640 1 -- 172.21.15.98:0/3870566040 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8021070 con 0x7f1cf41f2d40 2023-11-28T17:47:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.998+0000 7f1cf2ffd640 1 -- 172.21.15.98:0/3870566040 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4966 (secure 0 0 0) 0x7f1ce8005070 con 0x7f1cf41f2d40 2023-11-28T17:47:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.998+0000 7f1cf2ffd640 1 -- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf41f2d40 msgr2=0x7f1cf41f3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.998+0000 7f1cf2ffd640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf41f2d40 0x7f1cf41f3150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1ce800d810 tx=0x7f1ce80014a0 comp rx=0 tx=0).stop 2023-11-28T17:47:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.998+0000 7f1cf2ffd640 1 -- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf41f2d40 msgr2=0x7f1cf41f3150 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.998+0000 7f1cf2ffd640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf41f2d40 0x7f1cf41f3150 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1ce800d810 tx=0x7f1ce80014a0 comp rx=0 tx=0).stop 2023-11-28T17:47:08.000 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:47:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:47:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:47:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:47:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2023-11-28T17:47:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:47:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:47:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: 1 2023-11-28T17:47:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:47:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:47:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:47:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:47:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:47:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:47:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:47:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:47:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:47:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:47:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:47:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:47:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:47:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:47:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:47:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:47:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:47:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:47:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:47:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:47:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:47:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:47:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:47:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:47:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:47:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:47:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:47:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:47:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:47:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:47:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:47:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:47:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:47:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:47:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:47:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:47:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:47:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:47:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:47:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:47:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:47:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:47:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:47:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:47:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:47:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 6 2023-11-28T17:47:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:47:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:47:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:47:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:47:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:47:08.026 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:47:08.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.999+0000 7f1cfc7a9640 1 -- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1cd806b250 msgr2=0x7f1cd806d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:08.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.999+0000 7f1cfc7a9640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1cd806b250 0x7f1cd806d710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1cec036000 tx=0x7f1cec039040 comp rx=0 tx=0).stop 2023-11-28T17:47:08.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.999+0000 7f1cfc7a9640 1 -- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf4070a20 msgr2=0x7f1cf40b1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:08.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.999+0000 7f1cfc7a9640 1 --2- 172.21.15.98:0/3870566040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cf4070a20 0x7f1cf40b1f00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000b550 tx=0x7f1ce000ba10 comp rx=0 tx=0).stop 2023-11-28T17:47:08.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.999+0000 7f1cfad46640 1 -- 172.21.15.98:0/3870566040 reap_dead start 2023-11-28T17:47:08.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.999+0000 7f1cfad46640 1 -- 172.21.15.98:0/3870566040 reap_dead start 2023-11-28T17:47:08.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.999+0000 7f1cfc7a9640 1 -- 172.21.15.98:0/3870566040 shutdown_connections 2023-11-28T17:47:08.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.999+0000 7f1cfc7a9640 1 -- 172.21.15.98:0/3870566040 >> 172.21.15.98:0/3870566040 conn(0x7f1cf406e620 msgr2=0x7f1cf4157420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:08.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.999+0000 7f1cfc7a9640 1 -- 172.21.15.98:0/3870566040 shutdown_connections 2023-11-28T17:47:08.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:07.999+0000 7f1cfc7a9640 1 -- 172.21.15.98:0/3870566040 wait complete. 2023-11-28T17:47:08.029 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:47:08.029 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-11-28T17:47:08.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:47:08.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.237+0000 7ff8f9659640 1 Processor -- start 2023-11-28T17:47:08.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.237+0000 7ff8f9659640 1 -- start start 2023-11-28T17:47:08.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.239+0000 7ff8f9659640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ff8f40c24e0 0x7ff8f40c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.239+0000 7ff8f9659640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff8f406b160 0x7ff8f40c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.239+0000 7ff8f9659640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f40c6300 con 0x7ff8f4157ab0 2023-11-28T17:47:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.239+0000 7ff8f9659640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f40c2e00 con 0x7ff8f406b160 2023-11-28T17:47:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.239+0000 7ff8f9659640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f40c7430 con 0x7ff8f40c24e0 2023-11-28T17:47:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.239+0000 7ff8ebfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff8f406b160 0x7ff8f40c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.239+0000 7ff8ebfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff8f406b160 0x7ff8f40c28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57686/0 (socket says 172.21.15.98:57686) 2023-11-28T17:47:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.239+0000 7ff8ebfff640 1 -- 172.21.15.98:0/1623360383 learned_addr learned my addr 172.21.15.98:0/1623360383 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:08.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.239+0000 7ff8ebfff640 1 -- 172.21.15.98:0/1623360383 >> v2:172.21.15.113:3300/0 conn(0x7ff8f40c24e0 msgr2=0x7ff8f40c6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:08.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.239+0000 7ff8f3fff640 1 --2- 172.21.15.98:0/1623360383 >> v2:172.21.15.113:3300/0 conn(0x7ff8f40c24e0 0x7ff8f40c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:08.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.239+0000 7ff8ebfff640 1 --2- 172.21.15.98:0/1623360383 >> v2:172.21.15.113:3300/0 conn(0x7ff8f40c24e0 0x7ff8f40c6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:08.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.239+0000 7ff8ebfff640 1 -- 172.21.15.98:0/1623360383 >> v1:172.21.15.98:6789/0 conn(0x7ff8f4157ab0 legacy=0x7ff8f415b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:08.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.239+0000 7ff8ebfff640 1 -- 172.21.15.98:0/1623360383 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f40c7710 con 0x7ff8f406b160 2023-11-28T17:47:08.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.239+0000 7ff8f3fff640 1 --2- 172.21.15.98:0/1623360383 >> v2:172.21.15.113:3300/0 conn(0x7ff8f40c24e0 0x7ff8f40c6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:47:08.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.239+0000 7ff8f37fe640 1 -- 172.21.15.98:0/1623360383 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052502506 0 0) 0x7ff8f40c6300 con 0x7ff8f4157ab0 2023-11-28T17:47:08.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.241+0000 7ff8ebfff640 1 --2- 172.21.15.98:0/1623360383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f406b160 0x7ff8f40c28c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4068150 tx=0x7ff8dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb163b70f4f32b6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.241+0000 7ff8f37fe640 1 -- 172.21.15.98:0/1623360383 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc031870 con 0x7ff8f406b160 2023-11-28T17:47:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.241+0000 7ff8f37fe640 1 -- 172.21.15.98:0/1623360383 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8dc0319d0 con 0x7ff8f406b160 2023-11-28T17:47:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.241+0000 7ff8f37fe640 1 -- 172.21.15.98:0/1623360383 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc031d60 con 0x7ff8f406b160 2023-11-28T17:47:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.241+0000 7ff8f9659640 1 -- 172.21.15.98:0/1623360383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f406b160 msgr2=0x7ff8f40c28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.241+0000 7ff8f9659640 1 --2- 172.21.15.98:0/1623360383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f406b160 0x7ff8f40c28c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4068150 tx=0x7ff8dc02b910 comp rx=0 tx=0).stop 2023-11-28T17:47:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.241+0000 7ff8f9659640 1 -- 172.21.15.98:0/1623360383 shutdown_connections 2023-11-28T17:47:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.241+0000 7ff8f9659640 1 --2- 172.21.15.98:0/1623360383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f406b160 0x7ff8f40c28c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.241+0000 7ff8f9659640 1 --2- 172.21.15.98:0/1623360383 >> v2:172.21.15.113:3300/0 conn(0x7ff8f40c24e0 0x7ff8f40c6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.242+0000 7ff8f9659640 1 -- 172.21.15.98:0/1623360383 >> 172.21.15.98:0/1623360383 conn(0x7ff8f40cafe0 msgr2=0x7ff8f40cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.242+0000 7ff8f9659640 1 -- 172.21.15.98:0/1623360383 shutdown_connections 2023-11-28T17:47:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.242+0000 7ff8f9659640 1 -- 172.21.15.98:0/1623360383 wait complete. 2023-11-28T17:47:08.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.243+0000 7ff8f9659640 1 Processor -- start 2023-11-28T17:47:08.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.243+0000 7ff8f9659640 1 -- start start 2023-11-28T17:47:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.244+0000 7ff8f9659640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f406b160 0x7ff8f4158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.244+0000 7ff8f9659640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8f40c24e0 0x7ff8f41ef620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.244+0000 7ff8f9659640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f4157ab0 0x7ff8f41efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.244+0000 7ff8f9659640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f415bdf0 con 0x7ff8f4157ab0 2023-11-28T17:47:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.244+0000 7ff8f9659640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f40714e0 con 0x7ff8f406b160 2023-11-28T17:47:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.244+0000 7ff8f9659640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f41514d0 con 0x7ff8f40c24e0 2023-11-28T17:47:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.244+0000 7ff8ebfff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8f40c24e0 msgr2=0x7ff8f41ef620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.244+0000 7ff8ebfff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8f40c24e0 0x7ff8f41ef620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.244+0000 7ff8f8e58640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f4157ab0 0x7ff8f41efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.245+0000 7ff8f8e58640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f4157ab0 0x7ff8f41efb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57696/0 (socket says 172.21.15.98:57696) 2023-11-28T17:47:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.245+0000 7ff8f8e58640 1 -- 172.21.15.98:0/531713904 learned_addr learned my addr 172.21.15.98:0/531713904 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.245+0000 7ff8f3fff640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f406b160 0x7ff8f4158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.245+0000 7ff8f8e58640 1 -- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8f40c24e0 msgr2=0x7ff8f41ef620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:47:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.245+0000 7ff8f8e58640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ff8f40c24e0 0x7ff8f41ef620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.245+0000 7ff8f8e58640 1 -- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f406b160 msgr2=0x7ff8f4158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.245+0000 7ff8f8e58640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f406b160 0x7ff8f4158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:08.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.245+0000 7ff8f8e58640 1 -- 172.21.15.98:0/531713904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc0094d0 con 0x7ff8f4157ab0 2023-11-28T17:47:08.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.245+0000 7ff8f3fff640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f406b160 0x7ff8f4158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:47:08.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.245+0000 7ff8f8e58640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f4157ab0 0x7ff8f41efb60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00c710 tx=0x7ff8ec00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:08.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.245+0000 7ff8f17fa640 1 -- 172.21.15.98:0/531713904 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec003da0 con 0x7ff8f4157ab0 2023-11-28T17:47:08.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.245+0000 7ff8f17fa640 1 -- 172.21.15.98:0/531713904 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ec00ce20 con 0x7ff8f4157ab0 2023-11-28T17:47:08.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.245+0000 7ff8f17fa640 1 -- 172.21.15.98:0/531713904 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec005290 con 0x7ff8f4157ab0 2023-11-28T17:47:08.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.245+0000 7ff8f9659640 1 -- 172.21.15.98:0/531713904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f41517b0 con 0x7ff8f4157ab0 2023-11-28T17:47:08.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.246+0000 7ff8f9659640 1 -- 172.21.15.98:0/531713904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f415b780 con 0x7ff8f4157ab0 2023-11-28T17:47:08.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.246+0000 7ff8f9659640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f4158bd0 0x7ff8f414a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:08.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.246+0000 7ff8f9659640 1 -- 172.21.15.98:0/531713904 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f4074310 con 0x7ff8f4158bd0 2023-11-28T17:47:08.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.247+0000 7ff8f3fff640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f4158bd0 0x7ff8f414a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:08.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.248+0000 7ff8f17fa640 1 -- 172.21.15.98:0/531713904 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff8ec0053f0 con 0x7ff8f4157ab0 2023-11-28T17:47:08.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.248+0000 7ff8f3fff640 1 -- 172.21.15.98:0/531713904 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8f4157f60 con 0x7ff8f4158bd0 2023-11-28T17:47:08.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.248+0000 7ff8f3fff640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f4158bd0 0x7ff8f414a420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff8e400d790 tx=0x7ff8e400dc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:08.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.249+0000 7ff8f17fa640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff8d406b250 0x7ff8d406d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.249+0000 7ff8f17fa640 1 -- 172.21.15.98:0/531713904 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff8ec089560 con 0x7ff8f4157ab0 2023-11-28T17:47:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.249+0000 7ff8ebfff640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff8d406b250 0x7ff8d406d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.249+0000 7ff8f17fa640 1 -- 172.21.15.98:0/531713904 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4008b10 con 0x7ff8f4158bd0 2023-11-28T17:47:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.249+0000 7ff8f17fa640 1 -- 172.21.15.98:0/531713904 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7ff8e400c040 con 0x7ff8f4158bd0 2023-11-28T17:47:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.249+0000 7ff8f17fa640 1 -- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f4158bd0 msgr2=0x7ff8f414a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.249+0000 7ff8f17fa640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f4158bd0 0x7ff8f414a420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff8e400d790 tx=0x7ff8e400dc50 comp rx=0 tx=0).stop 2023-11-28T17:47:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.249+0000 7ff8f17fa640 1 -- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f4158bd0 msgr2=0x7ff8f414a420 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.249+0000 7ff8f17fa640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f4158bd0 0x7ff8f414a420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.250+0000 7ff8ebfff640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff8d406b250 0x7ff8d406d710 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc036000 tx=0x7ff8dc008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:08.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.256+0000 7ff8f9659640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f414f870 0x7ff8f414a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:08.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.256+0000 7ff8f9659640 1 -- 172.21.15.98:0/531713904 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f414aea0 con 0x7ff8f414f870 2023-11-28T17:47:08.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.257+0000 7ff8f3fff640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f414f870 0x7ff8f414a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:08.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.258+0000 7ff8f3fff640 1 -- 172.21.15.98:0/531713904 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff8f40c68a0 con 0x7ff8f414f870 2023-11-28T17:47:08.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.258+0000 7ff8f3fff640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f414f870 0x7ff8f414a960 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff8e400d790 tx=0x7ff8e4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:08.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.259+0000 7ff8f17fa640 1 -- 172.21.15.98:0/531713904 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4020040 con 0x7ff8f414f870 2023-11-28T17:47:08.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.259+0000 7ff8f17fa640 1 -- 172.21.15.98:0/531713904 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4624 (secure 0 0 0) 0x7ff8e4012e60 con 0x7ff8f414f870 2023-11-28T17:47:08.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.259+0000 7ff8f17fa640 1 -- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f414f870 msgr2=0x7ff8f414a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.259+0000 7ff8f17fa640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f414f870 0x7ff8f414a960 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff8e400d790 tx=0x7ff8e4004730 comp rx=0 tx=0).stop 2023-11-28T17:47:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.259+0000 7ff8f17fa640 1 -- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f414f870 msgr2=0x7ff8f414a960 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.259+0000 7ff8f17fa640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8f414f870 0x7ff8f414a960 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff8e400d790 tx=0x7ff8e4004730 comp rx=0 tx=0).stop 2023-11-28T17:47:08.261 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:47:08.261 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:47:08.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:47:08.262 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:47:08.262 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 64, 2023-11-28T17:47:08.262 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:47:08.262 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:47:08.262 INFO:teuthology.orchestra.run.smithi098.stdout: 1 2023-11-28T17:47:08.262 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:08.262 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:47:08.262 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:08.262 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:47:08.262 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:47:08.263 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:08.263 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:08.263 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:08.263 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:08.263 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:08.263 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:08.263 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:08.263 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:08.263 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:08.263 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:08.264 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:08.264 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:47:08.264 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:47:08.264 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:08.264 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:08.264 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:08.264 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:08.264 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:08.264 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:08.264 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:08.264 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:08.264 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:08.265 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:08.265 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:08.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.265 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:47:08.265 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:47:08.265 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:47:08.265 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:47:08.265 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:47:08.265 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:47:08.265 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:08.265 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:08.265 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:47:08.266 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:47:08.266 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:47:08.266 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:47:08.266 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:47:08.266 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:47:08.266 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:47:08.266 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:08.266 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:47:08.266 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:08.266 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:08.267 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:08.267 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:08.267 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:08.267 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:08.267 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:08.267 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:08.267 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:08.267 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:08.267 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:08.267 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:47:08.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.268 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:47:08.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.268 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:47:08.268 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:47:08.268 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:08.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:08.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:08.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:47:08.269 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:08.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:08.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:47:08.270 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:08.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.270 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:08.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:47:08.271 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:47:08.271 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:08.271 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:08.271 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:08.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.272 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:47:08.272 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:47:08.272 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:08.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:08.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:08.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:47:08.273 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:08.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:08.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:47:08.274 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:08.274 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.274 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:08.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:47:08.274 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:47:08.275 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:08.275 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:08.275 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:08.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.275 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:47:08.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:47:08.276 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:08.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:08.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:08.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:47:08.277 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:08.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:08.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:47:08.278 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:08.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.278 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:08.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:47:08.279 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:47:08.279 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:08.279 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:08.279 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:08.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.279 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:08.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.279 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:47:08.279 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:47:08.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.279 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:08.279 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:08.279 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:47:08.280 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.280 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:08.280 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:47:08.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.280 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:08.280 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:08.280 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:47:08.280 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.280 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:08.280 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:47:08.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:08.280 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:08.281 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:08.281 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:47:08.281 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:08.281 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:08.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:08.281 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:47:08.281 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:47:08.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.260+0000 7ff8f9659640 1 -- 172.21.15.98:0/531713904 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff8d406b250 msgr2=0x7ff8d406d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:08.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.260+0000 7ff8f9659640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ff8d406b250 0x7ff8d406d710 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc036000 tx=0x7ff8dc008030 comp rx=0 tx=0).stop 2023-11-28T17:47:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.260+0000 7ff8f9659640 1 -- 172.21.15.98:0/531713904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f4157ab0 msgr2=0x7ff8f41efb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.260+0000 7ff8f9659640 1 --2- 172.21.15.98:0/531713904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8f4157ab0 0x7ff8f41efb60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00c710 tx=0x7ff8ec00cbd0 comp rx=0 tx=0).stop 2023-11-28T17:47:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.260+0000 7ff8f8e58640 1 -- 172.21.15.98:0/531713904 reap_dead start 2023-11-28T17:47:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.260+0000 7ff8f8e58640 1 -- 172.21.15.98:0/531713904 reap_dead start 2023-11-28T17:47:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.260+0000 7ff8f9659640 1 -- 172.21.15.98:0/531713904 shutdown_connections 2023-11-28T17:47:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.260+0000 7ff8f9659640 1 -- 172.21.15.98:0/531713904 >> 172.21.15.98:0/531713904 conn(0x7ff8f40cafe0 msgr2=0x7ff8f40c3630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.260+0000 7ff8f9659640 1 -- 172.21.15.98:0/531713904 shutdown_connections 2023-11-28T17:47:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:08.260+0000 7ff8f9659640 1 -- 172.21.15.98:0/531713904 wait complete. 2023-11-28T17:47:08.283 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:47:08.283 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-11-28T17:47:09.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:47:09.363 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:47:13.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~142s 2023-11-28T17:47:19.994 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~148s 2023-11-28T17:47:26.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~155s 2023-11-28T17:47:28.284 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-11-28T17:47:28.285 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2023-11-28T17:47:28.285 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-28T17:47:28.285 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-11-28T17:47:28.294 INFO:tasks.ceph.mon.c:Started 2023-11-28T17:47:28.295 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:47:28.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:47:28.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.511+0000 7fba9785c640 1 Processor -- start 2023-11-28T17:47:28.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.512+0000 7fba9785c640 1 -- start start 2023-11-28T17:47:28.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.513+0000 7fba9785c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fba900c2ae0 0x7fba9015bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:28.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.513+0000 7fba9785c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba9006eb70 con 0x7fba9006f2d0 2023-11-28T17:47:28.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.513+0000 7fba9785c640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba9006e5b0 con 0x7fba900c73a0 2023-11-28T17:47:28.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.513+0000 7fba9785c640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba9015b890 con 0x7fba900c2ae0 2023-11-28T17:47:28.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.513+0000 7fba955f8640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7fba900c73a0 0x7fba900c5270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.98:41342/0 (socket says 172.21.15.98:41342) 2023-11-28T17:47:28.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.513+0000 7fba955f8640 1 -- 172.21.15.98:0/692186695 learned_addr learned my addr 172.21.15.98:0/692186695 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:47:28.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.514+0000 7fba94df7640 1 --2- 172.21.15.98:0/692186695 >> v2:172.21.15.113:3300/0 conn(0x7fba900c2ae0 0x7fba9015bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:28.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.514+0000 7fba83fff640 1 -- 172.21.15.98:0/692186695 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652583102 0 0) 0x7fba9006e5b0 con 0x7fba900c73a0 2023-11-28T17:47:28.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.514+0000 7fba83fff640 1 -- 172.21.15.98:0/692186695 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba7c003520 con 0x7fba900c73a0 2023-11-28T17:47:28.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.514+0000 7fba83fff640 1 -- 172.21.15.98:0/692186695 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875714678 0 0) 0x7fba9006eb70 con 0x7fba9006f2d0 2023-11-28T17:47:28.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.514+0000 7fba83fff640 1 -- 172.21.15.98:0/692186695 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba9006e5b0 con 0x7fba9006f2d0 2023-11-28T17:47:28.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.514+0000 7fba94df7640 1 -- 172.21.15.98:0/692186695 >> v1:172.21.15.113:6789/0 conn(0x7fba900c73a0 legacy=0x7fba900c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:28.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.514+0000 7fba94df7640 1 -- 172.21.15.98:0/692186695 >> v1:172.21.15.98:6789/0 conn(0x7fba9006f2d0 legacy=0x7fba90162430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:28.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.514+0000 7fba94df7640 1 -- 172.21.15.98:0/692186695 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9006e890 con 0x7fba900c2ae0 2023-11-28T17:47:28.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.515+0000 7fba94df7640 1 --2- 172.21.15.98:0/692186695 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba900c2ae0 0x7fba9015bcb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fba84009d90 tx=0x7fba8402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=575bd46d3e41121 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:28.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.516+0000 7fba83fff640 1 -- 172.21.15.98:0/692186695 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8402cd00 con 0x7fba900c2ae0 2023-11-28T17:47:28.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.516+0000 7fba83fff640 1 -- 172.21.15.98:0/692186695 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8402ce60 con 0x7fba900c2ae0 2023-11-28T17:47:28.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.516+0000 7fba83fff640 1 -- 172.21.15.98:0/692186695 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba84034660 con 0x7fba900c2ae0 2023-11-28T17:47:28.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.516+0000 7fba9785c640 1 -- 172.21.15.98:0/692186695 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba900c2ae0 msgr2=0x7fba9015bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:28.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.516+0000 7fba9785c640 1 --2- 172.21.15.98:0/692186695 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba900c2ae0 0x7fba9015bcb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fba84009d90 tx=0x7fba8402bf50 comp rx=0 tx=0).stop 2023-11-28T17:47:28.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.517+0000 7fba9785c640 1 -- 172.21.15.98:0/692186695 shutdown_connections 2023-11-28T17:47:28.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.517+0000 7fba9785c640 1 --2- 172.21.15.98:0/692186695 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba900c2ae0 0x7fba9015bcb0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:28.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.517+0000 7fba9785c640 1 -- 172.21.15.98:0/692186695 >> 172.21.15.98:0/692186695 conn(0x7fba9006a4d0 msgr2=0x7fba9006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:28.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.517+0000 7fba9785c640 1 -- 172.21.15.98:0/692186695 shutdown_connections 2023-11-28T17:47:28.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.517+0000 7fba9785c640 1 -- 172.21.15.98:0/692186695 wait complete. 2023-11-28T17:47:28.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.519+0000 7fba9785c640 1 Processor -- start 2023-11-28T17:47:28.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.519+0000 7fba9785c640 1 -- start start 2023-11-28T17:47:28.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba9785c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba9006f2d0 0x7fba90072130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:28.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba9785c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba900c2ae0 0x7fba901e1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:28.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba9785c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba900c73a0 0x7fba901f05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:28.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba9785c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba901f1290 con 0x7fba900c73a0 2023-11-28T17:47:28.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba9785c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba90158660 con 0x7fba9006f2d0 2023-11-28T17:47:28.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba9785c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba901e0c70 con 0x7fba900c2ae0 2023-11-28T17:47:28.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba95df9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba900c73a0 0x7fba901f05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:28.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba94df7640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba900c2ae0 0x7fba901e1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:28.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba95df9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba900c73a0 0x7fba901f05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39408/0 (socket says 172.21.15.98:39408) 2023-11-28T17:47:28.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba95df9640 1 -- 172.21.15.98:0/2028518211 learned_addr learned my addr 172.21.15.98:0/2028518211 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:28.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba955f8640 1 --2- 172.21.15.98:0/2028518211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba9006f2d0 0x7fba90072130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:28.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba94df7640 1 -- 172.21.15.98:0/2028518211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba900c2ae0 msgr2=0x7fba901e1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-11-28T17:47:28.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba94df7640 1 -- 172.21.15.98:0/2028518211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba900c2ae0 msgr2=0x7fba901e1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:47:28.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba94df7640 1 --2- 172.21.15.98:0/2028518211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba900c2ae0 0x7fba901e1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:28.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba94df7640 1 --2- 172.21.15.98:0/2028518211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba900c2ae0 0x7fba901e1740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:28.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba95df9640 1 -- 172.21.15.98:0/2028518211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba900c2ae0 msgr2=0x7fba901e1740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:47:28.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba95df9640 1 --2- 172.21.15.98:0/2028518211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba900c2ae0 0x7fba901e1740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:28.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba95df9640 1 -- 172.21.15.98:0/2028518211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba9006f2d0 msgr2=0x7fba90072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:28.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba95df9640 1 --2- 172.21.15.98:0/2028518211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba9006f2d0 0x7fba90072130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:28.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.520+0000 7fba95df9640 1 -- 172.21.15.98:0/2028518211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba84009a70 con 0x7fba900c73a0 2023-11-28T17:47:28.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.521+0000 7fba95df9640 1 --2- 172.21.15.98:0/2028518211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba900c73a0 0x7fba901f05a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fba8800c400 tx=0x7fba8800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:28.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.521+0000 7fba81ffb640 1 -- 172.21.15.98:0/2028518211 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba880052d0 con 0x7fba900c73a0 2023-11-28T17:47:28.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.521+0000 7fba81ffb640 1 -- 172.21.15.98:0/2028518211 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba88005430 con 0x7fba900c73a0 2023-11-28T17:47:28.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.521+0000 7fba9785c640 1 -- 172.21.15.98:0/2028518211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba901efd80 con 0x7fba900c73a0 2023-11-28T17:47:28.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.521+0000 7fba81ffb640 1 -- 172.21.15.98:0/2028518211 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba880037f0 con 0x7fba900c73a0 2023-11-28T17:47:28.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.521+0000 7fba9785c640 1 -- 172.21.15.98:0/2028518211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba901e4560 con 0x7fba900c73a0 2023-11-28T17:47:28.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.522+0000 7fba9785c640 1 -- 172.21.15.98:0/2028518211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba901587f0 con 0x7fba900c73a0 2023-11-28T17:47:28.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.523+0000 7fba81ffb640 1 -- 172.21.15.98:0/2028518211 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fba88003950 con 0x7fba900c73a0 2023-11-28T17:47:28.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.524+0000 7fba81ffb640 1 --2- 172.21.15.98:0/2028518211 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fba7406b2a0 0x7fba7406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:28.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.524+0000 7fba81ffb640 1 -- 172.21.15.98:0/2028518211 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fba88089260 con 0x7fba900c73a0 2023-11-28T17:47:28.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.525+0000 7fba955f8640 1 --2- 172.21.15.98:0/2028518211 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fba7406b2a0 0x7fba7406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:28.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.525+0000 7fba955f8640 1 --2- 172.21.15.98:0/2028518211 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fba7406b2a0 0x7fba7406d760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fba8c000f60 tx=0x7fba8c007d30 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:28.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.529+0000 7fba81ffb640 1 -- 172.21.15.98:0/2028518211 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba901587f0 con 0x7fba900c73a0 2023-11-28T17:47:28.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.783+0000 7fba9785c640 1 -- 172.21.15.98:0/2028518211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fba901587a0 con 0x7fba900c73a0 2023-11-28T17:47:28.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.784+0000 7fba81ffb640 1 -- 172.21.15.98:0/2028518211 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7fba88053fe0 con 0x7fba900c73a0 2023-11-28T17:47:28.785 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":64,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:47:28.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.786+0000 7fba9785c640 1 -- 172.21.15.98:0/2028518211 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fba7406b2a0 msgr2=0x7fba7406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:28.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.786+0000 7fba9785c640 1 --2- 172.21.15.98:0/2028518211 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fba7406b2a0 0x7fba7406d760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fba8c000f60 tx=0x7fba8c007d30 comp rx=0 tx=0).stop 2023-11-28T17:47:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.786+0000 7fba9785c640 1 -- 172.21.15.98:0/2028518211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba900c73a0 msgr2=0x7fba901f05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.786+0000 7fba9785c640 1 --2- 172.21.15.98:0/2028518211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba900c73a0 0x7fba901f05a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fba8800c400 tx=0x7fba8800c8c0 comp rx=0 tx=0).stop 2023-11-28T17:47:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.786+0000 7fba9785c640 1 -- 172.21.15.98:0/2028518211 shutdown_connections 2023-11-28T17:47:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.786+0000 7fba9785c640 1 --2- 172.21.15.98:0/2028518211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba900c73a0 0x7fba901f05a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.786+0000 7fba9785c640 1 --2- 172.21.15.98:0/2028518211 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fba900c2ae0 0x7fba901e1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.786+0000 7fba9785c640 1 --2- 172.21.15.98:0/2028518211 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fba7406b2a0 0x7fba7406d760 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.786+0000 7fba9785c640 1 --2- 172.21.15.98:0/2028518211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba9006f2d0 0x7fba90072130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.786+0000 7fba9785c640 1 -- 172.21.15.98:0/2028518211 >> 172.21.15.98:0/2028518211 conn(0x7fba9006a4d0 msgr2=0x7fba900cb350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.787+0000 7fba9785c640 1 -- 172.21.15.98:0/2028518211 shutdown_connections 2023-11-28T17:47:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:28.787+0000 7fba9785c640 1 -- 172.21.15.98:0/2028518211 wait complete. 2023-11-28T17:47:31.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:47:32.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.025+0000 7f0ccbfff640 1 Processor -- start 2023-11-28T17:47:32.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.025+0000 7f0ccbfff640 1 -- start start 2023-11-28T17:47:32.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.027+0000 7f0ccbfff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc1518d0 0x7f0ccc158240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:32.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.027+0000 7f0ccbfff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ccc06ade0 con 0x7f0ccc15ac70 2023-11-28T17:47:32.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.027+0000 7f0ccbfff640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ccc151cb0 con 0x7f0ccc0728a0 2023-11-28T17:47:32.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.027+0000 7f0ccbfff640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc06b030 con 0x7f0ccc1518d0 2023-11-28T17:47:32.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.028+0000 7f0ccb7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0ccc15ac70 0x7f0ccc15b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53448/0 (socket says 172.21.15.98:53448) 2023-11-28T17:47:32.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.028+0000 7f0ccb7fe640 1 -- 172.21.15.98:0/3148209235 learned_addr learned my addr 172.21.15.98:0/3148209235 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:47:32.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.028+0000 7f0ccaffd640 1 --2- 172.21.15.98:0/3148209235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc1518d0 0x7f0ccc158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:32.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154958804 0 0) 0x7f0ccc06ade0 con 0x7f0ccc15ac70 2023-11-28T17:47:32.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb8003520 con 0x7f0ccc15ac70 2023-11-28T17:47:32.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0ccaffd640 1 -- 172.21.15.98:0/3148209235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc1518d0 msgr2=0x7f0ccc158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:47:32.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0ccaffd640 1 -- 172.21.15.98:0/3148209235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc1518d0 msgr2=0x7f0ccc158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:32.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0ccaffd640 1 --2- 172.21.15.98:0/3148209235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc1518d0 0x7f0ccc158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:32.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0ccaffd640 1 --2- 172.21.15.98:0/3148209235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc1518d0 0x7f0ccc158240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:32.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220493500 0 0) 0x7f0ccc151cb0 con 0x7f0ccc0728a0 2023-11-28T17:47:32.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ccc06ade0 con 0x7f0ccc0728a0 2023-11-28T17:47:32.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1768946025 0 0) 0x7f0cb8003520 con 0x7f0ccc15ac70 2023-11-28T17:47:32.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc1518d0 msgr2=0x7f0ccc158240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:32.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 --2- 172.21.15.98:0/3148209235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc1518d0 0x7f0ccc158240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:32.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 >> v1:172.21.15.113:6789/0 conn(0x7f0ccc0728a0 legacy=0x7f0ccc070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:32.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc0700e0 con 0x7f0ccc15ac70 2023-11-28T17:47:32.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f0cc40019f0 con 0x7f0ccc15ac70 2023-11-28T17:47:32.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 >> v1:172.21.15.98:6789/0 conn(0x7f0ccc15ac70 legacy=0x7f0ccc15b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:32.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 --2- 172.21.15.98:0/3148209235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cb8003b30 0x7f0cb8003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:32.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 --2- 172.21.15.98:0/3148209235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb80047c0 0x7f0cb8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:32.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 --2- 172.21.15.98:0/3148209235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc1518d0 0x7f0cb800fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:32.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc06b030 con 0x7f0cb80047c0 2023-11-28T17:47:32.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb80054e0 con 0x7f0cb8003b30 2023-11-28T17:47:32.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb8005650 con 0x7f0ccc1518d0 2023-11-28T17:47:32.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.029+0000 7f0cc27fc640 1 --2- 172.21.15.98:0/3148209235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb80047c0 0x7f0cb8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:32.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.030+0000 7f0ccaffd640 1 --2- 172.21.15.98:0/3148209235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cb8003b30 0x7f0cb8003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:32.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.030+0000 7f0cc27fc640 1 -- 172.21.15.98:0/3148209235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc1518d0 msgr2=0x7f0cb800fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:32.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.030+0000 7f0cc27fc640 1 --2- 172.21.15.98:0/3148209235 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc1518d0 0x7f0cb800fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:32.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.030+0000 7f0cc27fc640 1 -- 172.21.15.98:0/3148209235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cb8003b30 msgr2=0x7f0cb8003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:32.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.030+0000 7f0cc27fc640 1 --2- 172.21.15.98:0/3148209235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cb8003b30 0x7f0cb8003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:32.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.030+0000 7f0cc27fc640 1 -- 172.21.15.98:0/3148209235 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb8003520 con 0x7f0cb80047c0 2023-11-28T17:47:32.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.031+0000 7f0cc27fc640 1 --2- 172.21.15.98:0/3148209235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb80047c0 0x7f0cb8004c40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4001900 tx=0x7f0cb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9cd4a45425cdaf6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:32.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.031+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb40052d0 con 0x7f0cb80047c0 2023-11-28T17:47:32.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.031+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb4005430 con 0x7f0cb80047c0 2023-11-28T17:47:32.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.031+0000 7f0cca7fc640 1 -- 172.21.15.98:0/3148209235 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb4003810 con 0x7f0cb80047c0 2023-11-28T17:47:32.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.032+0000 7f0ccbfff640 1 -- 172.21.15.98:0/3148209235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb80047c0 msgr2=0x7f0cb8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:32.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.032+0000 7f0ccbfff640 1 --2- 172.21.15.98:0/3148209235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cb80047c0 0x7f0cb8004c40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4001900 tx=0x7f0cb80057f0 comp rx=0 tx=0).stop 2023-11-28T17:47:32.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.032+0000 7f0ccb7fe640 1 -- 172.21.15.98:0/3148209235 reap_dead start 2023-11-28T17:47:32.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.032+0000 7f0ccbfff640 1 -- 172.21.15.98:0/3148209235 shutdown_connections 2023-11-28T17:47:32.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.032+0000 7f0ccbfff640 1 -- 172.21.15.98:0/3148209235 >> 172.21.15.98:0/3148209235 conn(0x7f0ccc066230 msgr2=0x7f0ccc06a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:32.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.032+0000 7f0ccbfff640 1 -- 172.21.15.98:0/3148209235 shutdown_connections 2023-11-28T17:47:32.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.032+0000 7f0ccbfff640 1 -- 172.21.15.98:0/3148209235 wait complete. 2023-11-28T17:47:32.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.034+0000 7f0ccbfff640 1 Processor -- start 2023-11-28T17:47:32.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.034+0000 7f0ccbfff640 1 -- start start 2023-11-28T17:47:32.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0ccbfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cb80047c0 0x7f0ccc14fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:32.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0ccbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ccc1eb5a0 0x7f0ccc0c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:32.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0ccbfff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc0c6a00 0x7f0ccc14a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:32.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0ccbfff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0c5940 con 0x7f0ccc1eb5a0 2023-11-28T17:47:32.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0ccbfff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc14cb20 con 0x7f0cb80047c0 2023-11-28T17:47:32.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0ccbfff640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0c55f0 con 0x7f0ccc0c6a00 2023-11-28T17:47:32.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0cc27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ccc1eb5a0 0x7f0ccc0c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:32.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0cc27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ccc1eb5a0 0x7f0ccc0c5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37266/0 (socket says 172.21.15.98:37266) 2023-11-28T17:47:32.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0cc27fc640 1 -- 172.21.15.98:0/2753697265 learned_addr learned my addr 172.21.15.98:0/2753697265 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:32.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0ccaffd640 1 --2- 172.21.15.98:0/2753697265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cb80047c0 0x7f0ccc14fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:32.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0cc27fc640 1 -- 172.21.15.98:0/2753697265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc0c6a00 msgr2=0x7f0ccc14a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:32.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0cc27fc640 1 --2- 172.21.15.98:0/2753697265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc0c6a00 0x7f0ccc14a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:32.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0cc27fc640 1 -- 172.21.15.98:0/2753697265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cb80047c0 msgr2=0x7f0ccc14fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:32.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0cc27fc640 1 --2- 172.21.15.98:0/2753697265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cb80047c0 0x7f0ccc14fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:32.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0cc27fc640 1 -- 172.21.15.98:0/2753697265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc0c1700 con 0x7f0ccc1eb5a0 2023-11-28T17:47:32.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.035+0000 7f0ccaffd640 1 --2- 172.21.15.98:0/2753697265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cb80047c0 0x7f0ccc14fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:47:32.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.036+0000 7f0cc27fc640 1 --2- 172.21.15.98:0/2753697265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ccc1eb5a0 0x7f0ccc0c5d60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4001900 tx=0x7f0cb40012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:32.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.036+0000 7f0cc8ff9640 1 -- 172.21.15.98:0/2753697265 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb4007450 con 0x7f0ccc1eb5a0 2023-11-28T17:47:32.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.036+0000 7f0cc8ff9640 1 -- 172.21.15.98:0/2753697265 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb40075b0 con 0x7f0ccc1eb5a0 2023-11-28T17:47:32.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.036+0000 7f0cc8ff9640 1 -- 172.21.15.98:0/2753697265 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb4007720 con 0x7f0ccc1eb5a0 2023-11-28T17:47:32.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.036+0000 7f0ccbfff640 1 -- 172.21.15.98:0/2753697265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ccc1ebb00 con 0x7f0ccc1eb5a0 2023-11-28T17:47:32.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.036+0000 7f0ccbfff640 1 -- 172.21.15.98:0/2753697265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ccc1ec0c0 con 0x7f0ccc1eb5a0 2023-11-28T17:47:32.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.037+0000 7f0ccbfff640 1 -- 172.21.15.98:0/2753697265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ccc0c5a80 con 0x7f0ccc1eb5a0 2023-11-28T17:47:32.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.038+0000 7f0cc8ff9640 1 -- 172.21.15.98:0/2753697265 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0cb40037f0 con 0x7f0ccc1eb5a0 2023-11-28T17:47:32.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.039+0000 7f0cc8ff9640 1 --2- 172.21.15.98:0/2753697265 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0cac06b2d0 0x7f0cac06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.039+0000 7f0cc8ff9640 1 -- 172.21.15.98:0/2753697265 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0cb40abae0 con 0x7f0ccc1eb5a0 2023-11-28T17:47:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.039+0000 7f0ccaffd640 1 --2- 172.21.15.98:0/2753697265 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0cac06b2d0 0x7f0cac06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:32.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.040+0000 7f0ccaffd640 1 --2- 172.21.15.98:0/2753697265 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0cac06b2d0 0x7f0cac06d790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc006fd0 tx=0x7f0cbc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:32.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.043+0000 7f0cc8ff9640 1 -- 172.21.15.98:0/2753697265 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ccc0c5a80 con 0x7f0ccc1eb5a0 2023-11-28T17:47:32.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.306+0000 7f0ccbfff640 1 -- 172.21.15.98:0/2753697265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0ccc0c5a80 con 0x7f0ccc1eb5a0 2023-11-28T17:47:32.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.307+0000 7f0cc8ff9640 1 -- 172.21.15.98:0/2753697265 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7f0cb4037620 con 0x7f0ccc1eb5a0 2023-11-28T17:47:32.308 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":64,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:47:32.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.310+0000 7f0ccbfff640 1 -- 172.21.15.98:0/2753697265 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0cac06b2d0 msgr2=0x7f0cac06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:32.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.310+0000 7f0ccbfff640 1 --2- 172.21.15.98:0/2753697265 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0cac06b2d0 0x7f0cac06d790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc006fd0 tx=0x7f0cbc008040 comp rx=0 tx=0).stop 2023-11-28T17:47:32.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.310+0000 7f0ccbfff640 1 -- 172.21.15.98:0/2753697265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ccc1eb5a0 msgr2=0x7f0ccc0c5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:32.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.310+0000 7f0ccbfff640 1 --2- 172.21.15.98:0/2753697265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ccc1eb5a0 0x7f0ccc0c5d60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4001900 tx=0x7f0cb40012d0 comp rx=0 tx=0).stop 2023-11-28T17:47:32.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.310+0000 7f0ccbfff640 1 -- 172.21.15.98:0/2753697265 shutdown_connections 2023-11-28T17:47:32.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.310+0000 7f0ccbfff640 1 --2- 172.21.15.98:0/2753697265 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0ccc0c6a00 0x7f0ccc14a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:32.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.310+0000 7f0ccbfff640 1 --2- 172.21.15.98:0/2753697265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ccc1eb5a0 0x7f0ccc0c5d60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:32.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.310+0000 7f0ccbfff640 1 --2- 172.21.15.98:0/2753697265 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0cac06b2d0 0x7f0cac06d790 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:32.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.310+0000 7f0ccbfff640 1 --2- 172.21.15.98:0/2753697265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cb80047c0 0x7f0ccc14fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:32.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.310+0000 7f0ccbfff640 1 -- 172.21.15.98:0/2753697265 >> 172.21.15.98:0/2753697265 conn(0x7f0ccc066230 msgr2=0x7f0ccc0ca670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:32.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.310+0000 7f0ccbfff640 1 -- 172.21.15.98:0/2753697265 shutdown_connections 2023-11-28T17:47:32.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:32.310+0000 7f0ccbfff640 1 -- 172.21.15.98:0/2753697265 wait complete. 2023-11-28T17:47:33.520 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~162s 2023-11-28T17:47:35.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:47:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.548+0000 7f2f2e70f640 1 Processor -- start 2023-11-28T17:47:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.549+0000 7f2f2e70f640 1 -- start start 2023-11-28T17:47:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.550+0000 7f2f2e70f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 0x7f2f280cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.550+0000 7f2f2e70f640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 0x7f2f2815d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.550+0000 7f2f2e70f640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2806b010 con 0x7f2f28153f60 2023-11-28T17:47:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.550+0000 7f2f2e70f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2815dd60 con 0x7f2f2806bd40 2023-11-28T17:47:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.550+0000 7f2f2e70f640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2815e040 con 0x7f2f2815d240 2023-11-28T17:47:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.551+0000 7f2f1ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 0x7f2f280cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.551+0000 7f2f1ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 0x7f2f280cb9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37282/0 (socket says 172.21.15.98:37282) 2023-11-28T17:47:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.551+0000 7f2f1ffff640 1 -- 172.21.15.98:0/1864090109 learned_addr learned my addr 172.21.15.98:0/1864090109 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.551+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 0x7f2f2815d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.551+0000 7f2f1ffff640 1 -- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 msgr2=0x7f2f280cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:47:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.551+0000 7f2f1ffff640 1 -- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 msgr2=0x7f2f280cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.551+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 0x7f2f280cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.551+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 0x7f2f280cb9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.551+0000 7f2f2ccac640 1 -- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 msgr2=0x7f2f2815d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:47:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.551+0000 7f2f2ccac640 1 -- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 msgr2=0x7f2f2815d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.551+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 0x7f2f2815d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.551+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 0x7f2f2815d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.752+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 0x7f2f280cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.752+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 0x7f2f2815d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.753+0000 7f2f2ccac640 1 -- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 msgr2=0x7f2f2815d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:47:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.753+0000 7f2f2ccac640 1 -- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 msgr2=0x7f2f2815d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.753+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 0x7f2f2815d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.753+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 0x7f2f2815d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:47:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.753+0000 7f2f1ffff640 1 -- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 msgr2=0x7f2f280cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:47:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.753+0000 7f2f1ffff640 1 -- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 msgr2=0x7f2f280cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.753+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 0x7f2f280cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:35.753+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 0x7f2f280cb9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:47:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.154+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 0x7f2f280cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.154+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 0x7f2f2815d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.154+0000 7f2f2ccac640 1 -- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 msgr2=0x7f2f2815d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:47:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.154+0000 7f2f2ccac640 1 -- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 msgr2=0x7f2f2815d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.154+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 0x7f2f2815d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.154+0000 7f2f1ffff640 1 -- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 msgr2=0x7f2f280cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:47:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.154+0000 7f2f1ffff640 1 -- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 msgr2=0x7f2f280cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.154+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 0x7f2f2815d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:47:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.154+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 0x7f2f280cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.155+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 0x7f2f280cb9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:47:36.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.956+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 0x7f2f280cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.956+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 0x7f2f2815d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.957+0000 7f2f2ccac640 1 -- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 msgr2=0x7f2f2815d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:47:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.957+0000 7f2f2ccac640 1 -- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 msgr2=0x7f2f2815d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.957+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 0x7f2f2815d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.957+0000 7f2f1ffff640 1 -- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 msgr2=0x7f2f280cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:47:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.957+0000 7f2f1ffff640 1 -- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 msgr2=0x7f2f280cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.957+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 0x7f2f280cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.957+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 0x7f2f2815d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:47:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:36.957+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 0x7f2f280cb9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:47:38.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.550+0000 7f2f267fc640 1 -- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 msgr2=0x7f2f2815d650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:38.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.550+0000 7f2f267fc640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f2815d240 0x7f2f2815d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:38.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.550+0000 7f2f267fc640 1 -- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 msgr2=0x7f2f280cb9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:38.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.550+0000 7f2f267fc640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.98:3300/0 conn(0x7f2f2806bd40 0x7f2f280cb9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:38.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.550+0000 7f2f267fc640 1 -- 172.21.15.98:0/1864090109 >> v1:172.21.15.113:6789/0 conn(0x7f2f28153f60 legacy=0x7f2f280c4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:38.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.551+0000 7f2f267fc640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.113:3300/0 conn(0x7f2f080073e0 0x7f2f08009880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:38.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.551+0000 7f2f267fc640 1 -- 172.21.15.98:0/1864090109 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2806b260 con 0x7f2f080036e0 2023-11-28T17:47:38.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.551+0000 7f2f267fc640 1 -- 172.21.15.98:0/1864090109 --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2815dd60 con 0x7f2f080073e0 2023-11-28T17:47:38.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.551+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.113:3300/0 conn(0x7f2f080073e0 0x7f2f08009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:38.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.552+0000 7f2f1ffff640 1 -- 172.21.15.98:0/1864090109 >> v2:172.21.15.113:3300/0 conn(0x7f2f080073e0 msgr2=0x7f2f08009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:47:38.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.552+0000 7f2f1ffff640 1 -- 172.21.15.98:0/1864090109 >> v2:172.21.15.113:3300/0 conn(0x7f2f080073e0 msgr2=0x7f2f08009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:38.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.552+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.113:3300/0 conn(0x7f2f080073e0 0x7f2f08009880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:38.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.552+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.113:3300/0 conn(0x7f2f080073e0 0x7f2f08009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:38.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.599+0000 7f2f277fe640 1 -- 172.21.15.98:0/1864090109 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218999474 0 0) 0x7f2f2806b260 con 0x7f2f080036e0 2023-11-28T17:47:38.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.599+0000 7f2f277fe640 1 -- 172.21.15.98:0/1864090109 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f14003520 con 0x7f2f080036e0 2023-11-28T17:47:38.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.600+0000 7f2f277fe640 1 -- 172.21.15.98:0/1864090109 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1790973035 0 0) 0x7f2f14003520 con 0x7f2f080036e0 2023-11-28T17:47:38.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.600+0000 7f2f277fe640 1 -- 172.21.15.98:0/1864090109 >> v2:172.21.15.113:3300/0 conn(0x7f2f080073e0 msgr2=0x7f2f08009880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:38.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.600+0000 7f2f277fe640 1 --2- 172.21.15.98:0/1864090109 >> v2:172.21.15.113:3300/0 conn(0x7f2f080073e0 0x7f2f08009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:38.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.600+0000 7f2f277fe640 1 -- 172.21.15.98:0/1864090109 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0800afe0 con 0x7f2f080036e0 2023-11-28T17:47:38.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.600+0000 7f2f277fe640 1 -- 172.21.15.98:0/1864090109 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f2f180016f0 con 0x7f2f080036e0 2023-11-28T17:47:38.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.600+0000 7f2f277fe640 1 -- 172.21.15.98:0/1864090109 >> v1:172.21.15.98:6789/0 conn(0x7f2f080036e0 legacy=0x7f2f08005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:38.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.600+0000 7f2f2ccac640 1 -- 172.21.15.98:0/1864090109 reap_dead start 2023-11-28T17:47:38.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.600+0000 7f2f277fe640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f14003970 0x7f2f14003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:38.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.600+0000 7f2f277fe640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f140047c0 0x7f2f14004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:38.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.600+0000 7f2f277fe640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f140052a0 0x7f2f1400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:38.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.600+0000 7f2f277fe640 1 -- 172.21.15.98:0/1864090109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2815dd60 con 0x7f2f140052a0 2023-11-28T17:47:38.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.600+0000 7f2f277fe640 1 -- 172.21.15.98:0/1864090109 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1400c1f0 con 0x7f2f140047c0 2023-11-28T17:47:38.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.600+0000 7f2f277fe640 1 -- 172.21.15.98:0/1864090109 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1400c3b0 con 0x7f2f14003970 2023-11-28T17:47:38.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.601+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f140052a0 0x7f2f1400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:38.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.601+0000 7f2f27fff640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f14003970 0x7f2f14003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:38.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.601+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f140047c0 0x7f2f14004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:38.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.602+0000 7f2f27fff640 1 -- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f140047c0 msgr2=0x7f2f14004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:38.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.602+0000 7f2f27fff640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f140047c0 0x7f2f14004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:38.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.602+0000 7f2f27fff640 1 -- 172.21.15.98:0/1864090109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f140052a0 msgr2=0x7f2f1400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:38.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.602+0000 7f2f27fff640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f140052a0 0x7f2f1400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:38.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.602+0000 7f2f27fff640 1 -- 172.21.15.98:0/1864090109 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f14003520 con 0x7f2f14003970 2023-11-28T17:47:38.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.602+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f140052a0 0x7f2f1400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:47:38.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.604+0000 7f2f27fff640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f14003970 0x7f2f14003d70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2f18007fb0 tx=0x7f2f140057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44dd6debb74295cf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.604+0000 7f2f277fe640 1 -- 172.21.15.98:0/1864090109 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1802c9c0 con 0x7f2f14003970 2023-11-28T17:47:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.605+0000 7f2f277fe640 1 -- 172.21.15.98:0/1864090109 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1802cb20 con 0x7f2f14003970 2023-11-28T17:47:38.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.605+0000 7f2f277fe640 1 -- 172.21.15.98:0/1864090109 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f18034440 con 0x7f2f14003970 2023-11-28T17:47:38.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.606+0000 7f2f2e70f640 1 -- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f14003970 msgr2=0x7f2f14003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:38.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.606+0000 7f2f2e70f640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f14003970 0x7f2f14003d70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2f18007fb0 tx=0x7f2f140057f0 comp rx=0 tx=0).stop 2023-11-28T17:47:38.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.606+0000 7f2f2e70f640 1 -- 172.21.15.98:0/1864090109 shutdown_connections 2023-11-28T17:47:38.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.606+0000 7f2f2e70f640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f140052a0 0x7f2f1400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:38.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.606+0000 7f2f2e70f640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f140047c0 0x7f2f14004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:38.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.606+0000 7f2f2e70f640 1 --2- 172.21.15.98:0/1864090109 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f14003970 0x7f2f14003d70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:38.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.606+0000 7f2f2e70f640 1 -- 172.21.15.98:0/1864090109 >> 172.21.15.98:0/1864090109 conn(0x7f2f28066410 msgr2=0x7f2f28066820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:38.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.606+0000 7f2f2e70f640 1 -- 172.21.15.98:0/1864090109 shutdown_connections 2023-11-28T17:47:38.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.606+0000 7f2f2e70f640 1 -- 172.21.15.98:0/1864090109 wait complete. 2023-11-28T17:47:38.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.610+0000 7f2f2e70f640 1 Processor -- start 2023-11-28T17:47:38.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.611+0000 7f2f2e70f640 1 -- start start 2023-11-28T17:47:38.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.613+0000 7f2f2e70f640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f14003970 0x7f2f281f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:38.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.613+0000 7f2f2e70f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f140047c0 0x7f2f2815c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:38.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.613+0000 7f2f2e70f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f140052a0 0x7f2f280b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.613+0000 7f2f2e70f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f281f3bf0 con 0x7f2f140052a0 2023-11-28T17:47:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.613+0000 7f2f2e70f640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f281ef840 con 0x7f2f140047c0 2023-11-28T17:47:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.613+0000 7f2f2e70f640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f281f0660 con 0x7f2f14003970 2023-11-28T17:47:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.613+0000 7f2f2ccac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f140052a0 0x7f2f280b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.613+0000 7f2f2ccac640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f140052a0 0x7f2f280b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37336/0 (socket says 172.21.15.98:37336) 2023-11-28T17:47:38.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.614+0000 7f2f2ccac640 1 -- 172.21.15.98:0/1996963816 learned_addr learned my addr 172.21.15.98:0/1996963816 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.614+0000 7f2f27fff640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f14003970 0x7f2f281f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.614+0000 7f2f27fff640 1 -- 172.21.15.98:0/1996963816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f14003970 msgr2=0x7f2f281f36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-11-28T17:47:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.614+0000 7f2f27fff640 1 -- 172.21.15.98:0/1996963816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f14003970 msgr2=0x7f2f281f36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:47:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.614+0000 7f2f1ffff640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f140047c0 0x7f2f2815c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.614+0000 7f2f27fff640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f14003970 0x7f2f281f36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.614+0000 7f2f27fff640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f14003970 0x7f2f281f36b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.614+0000 7f2f2ccac640 1 -- 172.21.15.98:0/1996963816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f14003970 msgr2=0x7f2f281f36b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:47:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.614+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f14003970 0x7f2f281f36b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.614+0000 7f2f2ccac640 1 -- 172.21.15.98:0/1996963816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f140047c0 msgr2=0x7f2f2815c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:38.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.614+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f140047c0 0x7f2f2815c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.614+0000 7f2f2ccac640 1 -- 172.21.15.98:0/1996963816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f281efae0 con 0x7f2f140052a0 2023-11-28T17:47:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.614+0000 7f2f2ccac640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f140052a0 0x7f2f280b37d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009e40 tx=0x7f2f20015700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.615+0000 7f2f257fa640 1 -- 172.21.15.98:0/1996963816 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f20021410 con 0x7f2f140052a0 2023-11-28T17:47:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.615+0000 7f2f257fa640 1 -- 172.21.15.98:0/1996963816 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2000f040 con 0x7f2f140052a0 2023-11-28T17:47:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.615+0000 7f2f257fa640 1 -- 172.21.15.98:0/1996963816 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f200217f0 con 0x7f2f140052a0 2023-11-28T17:47:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.615+0000 7f2f2e70f640 1 -- 172.21.15.98:0/1996963816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f281efdc0 con 0x7f2f140052a0 2023-11-28T17:47:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.615+0000 7f2f2e70f640 1 -- 172.21.15.98:0/1996963816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f281f0380 con 0x7f2f140052a0 2023-11-28T17:47:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.617+0000 7f2f2e70f640 1 -- 172.21.15.98:0/1996963816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f280b34f0 con 0x7f2f140052a0 2023-11-28T17:47:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.617+0000 7f2f257fa640 1 -- 172.21.15.98:0/1996963816 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2f20021950 con 0x7f2f140052a0 2023-11-28T17:47:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.619+0000 7f2f257fa640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2f0806bb90 0x7f2f0806e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.619+0000 7f2f257fa640 1 -- 172.21.15.98:0/1996963816 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2f2008a980 con 0x7f2f140052a0 2023-11-28T17:47:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.619+0000 7f2f27fff640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2f0806bb90 0x7f2f0806e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.620+0000 7f2f27fff640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2f0806bb90 0x7f2f0806e050 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2f18036000 tx=0x7f2f18009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.623+0000 7f2f257fa640 1 -- 172.21.15.98:0/1996963816 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f280b34f0 con 0x7f2f140052a0 2023-11-28T17:47:38.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.879+0000 7f2f2e70f640 1 -- 172.21.15.98:0/1996963816 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2f280b34f0 con 0x7f2f140052a0 2023-11-28T17:47:38.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.880+0000 7f2f257fa640 1 -- 172.21.15.98:0/1996963816 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f2f280b34f0 con 0x7f2f140052a0 2023-11-28T17:47:38.882 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":68,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:47:38.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.883+0000 7f2f2e70f640 1 -- 172.21.15.98:0/1996963816 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2f0806bb90 msgr2=0x7f2f0806e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:38.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.883+0000 7f2f2e70f640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2f0806bb90 0x7f2f0806e050 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2f18036000 tx=0x7f2f18009030 comp rx=0 tx=0).stop 2023-11-28T17:47:38.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.883+0000 7f2f2e70f640 1 -- 172.21.15.98:0/1996963816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f140052a0 msgr2=0x7f2f280b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:38.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.883+0000 7f2f2e70f640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f140052a0 0x7f2f280b37d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009e40 tx=0x7f2f20015700 comp rx=0 tx=0).stop 2023-11-28T17:47:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.883+0000 7f2f2e70f640 1 -- 172.21.15.98:0/1996963816 shutdown_connections 2023-11-28T17:47:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.883+0000 7f2f2e70f640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f140052a0 0x7f2f280b37d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.883+0000 7f2f2e70f640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2f0806bb90 0x7f2f0806e050 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.883+0000 7f2f2e70f640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f140047c0 0x7f2f2815c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.883+0000 7f2f2e70f640 1 --2- 172.21.15.98:0/1996963816 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2f14003970 0x7f2f281f36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.883+0000 7f2f2e70f640 1 -- 172.21.15.98:0/1996963816 >> 172.21.15.98:0/1996963816 conn(0x7f2f28066410 msgr2=0x7f2f280cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.883+0000 7f2f2e70f640 1 -- 172.21.15.98:0/1996963816 shutdown_connections 2023-11-28T17:47:38.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:38.883+0000 7f2f2e70f640 1 -- 172.21.15.98:0/1996963816 wait complete. 2023-11-28T17:47:39.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:47:39.686 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:47:40.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~169s 2023-11-28T17:47:41.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:47:42.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.118+0000 7f13ffe07640 1 Processor -- start 2023-11-28T17:47:42.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.119+0000 7f13ffe07640 1 -- start start 2023-11-28T17:47:42.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.120+0000 7f13ffe07640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f13f8151d80 0x7f13f8157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.120+0000 7f13ffe07640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f815b150 con 0x7f13f80c4910 2023-11-28T17:47:42.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.120+0000 7f13ffe07640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f806e1f0 con 0x7f13f815bc10 2023-11-28T17:47:42.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.120+0000 7f13ffe07640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f806e4d0 con 0x7f13f8151d80 2023-11-28T17:47:42.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.121+0000 7f13fdba3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f13f8151d80 0x7f13f8157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.121+0000 7f13fdba3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f13f8151d80 0x7f13f8157b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57500/0 (socket says 172.21.15.98:57500) 2023-11-28T17:47:42.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.121+0000 7f13fdba3640 1 -- 172.21.15.98:0/3872621854 learned_addr learned my addr 172.21.15.98:0/3872621854 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:42.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.121+0000 7f13fdba3640 1 -- 172.21.15.98:0/3872621854 >> v1:172.21.15.113:6789/0 conn(0x7f13f815bc10 legacy=0x7f13f80c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.121+0000 7f13fdba3640 1 -- 172.21.15.98:0/3872621854 >> v1:172.21.15.98:6789/0 conn(0x7f13f80c4910 legacy=0x7f13f81624e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.121+0000 7f13fdba3640 1 -- 172.21.15.98:0/3872621854 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f806e7b0 con 0x7f13f8151d80 2023-11-28T17:47:42.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.121+0000 7f13fcba1640 1 -- 172.21.15.98:0/3872621854 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881517832 0 0) 0x7f13f815b150 con 0x7f13f80c4910 2023-11-28T17:47:42.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.122+0000 7f13fdba3640 1 --2- 172.21.15.98:0/3872621854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f8151d80 0x7f13f8157b60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f13f4009870 tx=0x7f13f402bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e6ee02408cced0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:42.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.123+0000 7f13fcba1640 1 -- 172.21.15.98:0/3872621854 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f402d870 con 0x7f13f8151d80 2023-11-28T17:47:42.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.123+0000 7f13fcba1640 1 -- 172.21.15.98:0/3872621854 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f402d9d0 con 0x7f13f8151d80 2023-11-28T17:47:42.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.123+0000 7f13fcba1640 1 -- 172.21.15.98:0/3872621854 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f402dd60 con 0x7f13f8151d80 2023-11-28T17:47:42.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.123+0000 7f13ffe07640 1 -- 172.21.15.98:0/3872621854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f8151d80 msgr2=0x7f13f8157b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.123+0000 7f13ffe07640 1 --2- 172.21.15.98:0/3872621854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f8151d80 0x7f13f8157b60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f13f4009870 tx=0x7f13f402bab0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.124+0000 7f13ffe07640 1 -- 172.21.15.98:0/3872621854 shutdown_connections 2023-11-28T17:47:42.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.124+0000 7f13ffe07640 1 --2- 172.21.15.98:0/3872621854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f8151d80 0x7f13f8157b60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.124+0000 7f13ffe07640 1 -- 172.21.15.98:0/3872621854 >> 172.21.15.98:0/3872621854 conn(0x7f13f806a4d0 msgr2=0x7f13f806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:42.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.124+0000 7f13ffe07640 1 -- 172.21.15.98:0/3872621854 shutdown_connections 2023-11-28T17:47:42.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.124+0000 7f13ffe07640 1 -- 172.21.15.98:0/3872621854 wait complete. 2023-11-28T17:47:42.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.125+0000 7f13ffe07640 1 Processor -- start 2023-11-28T17:47:42.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.126+0000 7f13ffe07640 1 -- start start 2023-11-28T17:47:42.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.126+0000 7f13ffe07640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13f80c4910 0x7f13f81575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.126+0000 7f13ffe07640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f8151d80 0x7f13f81efd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.126+0000 7f13ffe07640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13f815bc10 0x7f13f81f0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.127+0000 7f13ffe07640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f81f0780 con 0x7f13f8151d80 2023-11-28T17:47:42.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.127+0000 7f13ffe07640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f81f3e80 con 0x7f13f815bc10 2023-11-28T17:47:42.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.127+0000 7f13ffe07640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f80c41b0 con 0x7f13f80c4910 2023-11-28T17:47:42.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.127+0000 7f13fd3a2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f8151d80 0x7f13f81efd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.127+0000 7f13fd3a2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f8151d80 0x7f13f81efd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57506/0 (socket says 172.21.15.98:57506) 2023-11-28T17:47:42.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.127+0000 7f13fd3a2640 1 -- 172.21.15.98:0/1522084101 learned_addr learned my addr 172.21.15.98:0/1522084101 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:42.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.127+0000 7f13fdba3640 1 --2- 172.21.15.98:0/1522084101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13f80c4910 0x7f13f81575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.127+0000 7f13fe3a4640 1 --2- 172.21.15.98:0/1522084101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13f815bc10 0x7f13f81f0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.127+0000 7f13fd3a2640 1 -- 172.21.15.98:0/1522084101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13f80c4910 msgr2=0x7f13f81575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.127+0000 7f13fd3a2640 1 --2- 172.21.15.98:0/1522084101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13f80c4910 0x7f13f81575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.127+0000 7f13fd3a2640 1 -- 172.21.15.98:0/1522084101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13f815bc10 msgr2=0x7f13f81f0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.127+0000 7f13fd3a2640 1 --2- 172.21.15.98:0/1522084101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13f815bc10 0x7f13f81f0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.127+0000 7f13fd3a2640 1 -- 172.21.15.98:0/1522084101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f40094d0 con 0x7f13f8151d80 2023-11-28T17:47:42.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.128+0000 7f13fd3a2640 1 --2- 172.21.15.98:0/1522084101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f8151d80 0x7f13f81efd00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f13e800c450 tx=0x7f13e800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:42.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.128+0000 7f13ee7fc640 1 -- 172.21.15.98:0/1522084101 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e80052d0 con 0x7f13f8151d80 2023-11-28T17:47:42.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.128+0000 7f13ee7fc640 1 -- 172.21.15.98:0/1522084101 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e8005430 con 0x7f13f8151d80 2023-11-28T17:47:42.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.128+0000 7f13ee7fc640 1 -- 172.21.15.98:0/1522084101 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e8012870 con 0x7f13f8151d80 2023-11-28T17:47:42.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.128+0000 7f13ffe07640 1 -- 172.21.15.98:0/1522084101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f80c4450 con 0x7f13f8151d80 2023-11-28T17:47:42.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.128+0000 7f13ffe07640 1 -- 172.21.15.98:0/1522084101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f81f3550 con 0x7f13f8151d80 2023-11-28T17:47:42.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.129+0000 7f13ffe07640 1 -- 172.21.15.98:0/1522084101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f8067bf0 con 0x7f13f8151d80 2023-11-28T17:47:42.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.130+0000 7f13ee7fc640 1 -- 172.21.15.98:0/1522084101 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f13e80129d0 con 0x7f13f8151d80 2023-11-28T17:47:42.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.131+0000 7f13ee7fc640 1 --2- 172.21.15.98:0/1522084101 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f13dc06b2a0 0x7f13dc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.132+0000 7f13ee7fc640 1 -- 172.21.15.98:0/1522084101 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f13e8089830 con 0x7f13f8151d80 2023-11-28T17:47:42.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.132+0000 7f13fdba3640 1 --2- 172.21.15.98:0/1522084101 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f13dc06b2a0 0x7f13dc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.132+0000 7f13fdba3640 1 --2- 172.21.15.98:0/1522084101 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f13dc06b2a0 0x7f13dc06d760 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f13f4036000 tx=0x7f13f4008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:42.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.136+0000 7f13ee7fc640 1 -- 172.21.15.98:0/1522084101 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13e8054580 con 0x7f13f8151d80 2023-11-28T17:47:42.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.391+0000 7f13ffe07640 1 -- 172.21.15.98:0/1522084101 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f13f8074f60 con 0x7f13f8151d80 2023-11-28T17:47:42.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.392+0000 7f13ee7fc640 1 -- 172.21.15.98:0/1522084101 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7f13e8057ba0 con 0x7f13f8151d80 2023-11-28T17:47:42.396 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:47:42.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.394+0000 7f13ffe07640 1 -- 172.21.15.98:0/1522084101 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f13dc06b2a0 msgr2=0x7f13dc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.394+0000 7f13ffe07640 1 --2- 172.21.15.98:0/1522084101 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f13dc06b2a0 0x7f13dc06d760 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f13f4036000 tx=0x7f13f4008030 comp rx=0 tx=0).stop 2023-11-28T17:47:42.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.394+0000 7f13ffe07640 1 -- 172.21.15.98:0/1522084101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f8151d80 msgr2=0x7f13f81efd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.394+0000 7f13ffe07640 1 --2- 172.21.15.98:0/1522084101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f8151d80 0x7f13f81efd00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f13e800c450 tx=0x7f13e800c910 comp rx=0 tx=0).stop 2023-11-28T17:47:42.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.395+0000 7f13ffe07640 1 -- 172.21.15.98:0/1522084101 shutdown_connections 2023-11-28T17:47:42.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.395+0000 7f13ffe07640 1 --2- 172.21.15.98:0/1522084101 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f13dc06b2a0 0x7f13dc06d760 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.395+0000 7f13ffe07640 1 --2- 172.21.15.98:0/1522084101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13f815bc10 0x7f13f81f0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.395+0000 7f13ffe07640 1 --2- 172.21.15.98:0/1522084101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f8151d80 0x7f13f81efd00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.395+0000 7f13ffe07640 1 --2- 172.21.15.98:0/1522084101 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f13f80c4910 0x7f13f81575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.395+0000 7f13ffe07640 1 -- 172.21.15.98:0/1522084101 >> 172.21.15.98:0/1522084101 conn(0x7f13f806a4d0 msgr2=0x7f13f8150b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.395+0000 7f13ffe07640 1 -- 172.21.15.98:0/1522084101 shutdown_connections 2023-11-28T17:47:42.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.395+0000 7f13ffe07640 1 -- 172.21.15.98:0/1522084101 wait complete. 2023-11-28T17:47:42.415 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:47:42.416 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:47:42.416 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:47:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.610+0000 7f3258f78640 1 Processor -- start 2023-11-28T17:47:42.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.610+0000 7f3258f78640 1 -- start start 2023-11-28T17:47:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.612+0000 7f3258f78640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3254154030 0x7f3254070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.612+0000 7f3258f78640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3254154410 con 0x7f325406dcf0 2023-11-28T17:47:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.612+0000 7f3258f78640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32540715a0 con 0x7f32540c8600 2023-11-28T17:47:42.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.612+0000 7f3258f78640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32540c5da0 con 0x7f3254154030 2023-11-28T17:47:42.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.612+0000 7f3252d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f325406dcf0 0x7f3254164660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50716/0 (socket says 172.21.15.98:50716) 2023-11-28T17:47:42.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.612+0000 7f3252d9d640 1 -- 172.21.15.98:0/1599285052 learned_addr learned my addr 172.21.15.98:0/1599285052 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:47:42.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f325259c640 1 --2- 172.21.15.98:0/1599285052 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3254154030 0x7f3254070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f3251d9b640 1 -- 172.21.15.98:0/1599285052 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974544627 0 0) 0x7f3254154410 con 0x7f325406dcf0 2023-11-28T17:47:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f3251d9b640 1 -- 172.21.15.98:0/1599285052 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3240003520 con 0x7f325406dcf0 2023-11-28T17:47:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f325259c640 1 -- 172.21.15.98:0/1599285052 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3254154030 msgr2=0x7f3254070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:47:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f325259c640 1 -- 172.21.15.98:0/1599285052 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3254154030 msgr2=0x7f3254070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f325259c640 1 --2- 172.21.15.98:0/1599285052 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3254154030 0x7f3254070660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:42.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f325259c640 1 --2- 172.21.15.98:0/1599285052 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3254154030 0x7f3254070660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:42.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f3251d9b640 1 -- 172.21.15.98:0/1599285052 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3452800676 0 0) 0x7f3240003520 con 0x7f325406dcf0 2023-11-28T17:47:42.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f3251d9b640 1 -- 172.21.15.98:0/1599285052 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3254154030 msgr2=0x7f3254070660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:42.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f3251d9b640 1 --2- 172.21.15.98:0/1599285052 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3254154030 0x7f3254070660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f3251d9b640 1 -- 172.21.15.98:0/1599285052 >> v1:172.21.15.113:6789/0 conn(0x7f32540c8600 legacy=0x7f3254070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f3251d9b640 1 -- 172.21.15.98:0/1599285052 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325415d370 con 0x7f325406dcf0 2023-11-28T17:47:42.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f3251d9b640 1 -- 172.21.15.98:0/1599285052 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f324c001a10 con 0x7f325406dcf0 2023-11-28T17:47:42.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f3251d9b640 1 -- 172.21.15.98:0/1599285052 >> v1:172.21.15.98:6789/0 conn(0x7f325406dcf0 legacy=0x7f3254164660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f3251d9b640 1 --2- 172.21.15.98:0/1599285052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32400039a0 0x7f3240003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f3251d9b640 1 --2- 172.21.15.98:0/1599285052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32400047c0 0x7f3240004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.613+0000 7f3251d9b640 1 --2- 172.21.15.98:0/1599285052 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3254154030 0x7f32400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.614+0000 7f3251d9b640 1 -- 172.21.15.98:0/1599285052 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32540c5da0 con 0x7f32400039a0 2023-11-28T17:47:42.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.614+0000 7f3251d9b640 1 -- 172.21.15.98:0/1599285052 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324000bb40 con 0x7f32400047c0 2023-11-28T17:47:42.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.614+0000 7f3251d9b640 1 -- 172.21.15.98:0/1599285052 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f324000bd00 con 0x7f3254154030 2023-11-28T17:47:42.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.614+0000 7f325259c640 1 --2- 172.21.15.98:0/1599285052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32400039a0 0x7f3240003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.614+0000 7f324bfff640 1 --2- 172.21.15.98:0/1599285052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32400047c0 0x7f3240004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.614+0000 7f3252d9d640 1 --2- 172.21.15.98:0/1599285052 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3254154030 0x7f32400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.614+0000 7f325259c640 1 -- 172.21.15.98:0/1599285052 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3254154030 msgr2=0x7f32400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:42.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.614+0000 7f325259c640 1 --2- 172.21.15.98:0/1599285052 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3254154030 0x7f32400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.614+0000 7f325259c640 1 -- 172.21.15.98:0/1599285052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32400047c0 msgr2=0x7f3240004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.614+0000 7f325259c640 1 --2- 172.21.15.98:0/1599285052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32400047c0 0x7f3240004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.614+0000 7f325259c640 1 -- 172.21.15.98:0/1599285052 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3240003520 con 0x7f32400039a0 2023-11-28T17:47:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.615+0000 7f325259c640 1 --2- 172.21.15.98:0/1599285052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32400039a0 0x7f3240003dc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f324400b9e0 tx=0x7f32400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=633bd7aeceac51f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.615+0000 7f3251d9b640 1 -- 172.21.15.98:0/1599285052 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244032620 con 0x7f32400039a0 2023-11-28T17:47:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.615+0000 7f3251d9b640 1 -- 172.21.15.98:0/1599285052 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3244032780 con 0x7f32400039a0 2023-11-28T17:47:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.615+0000 7f3251d9b640 1 -- 172.21.15.98:0/1599285052 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244032b30 con 0x7f32400039a0 2023-11-28T17:47:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.616+0000 7f3258f78640 1 -- 172.21.15.98:0/1599285052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32400039a0 msgr2=0x7f3240003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.616+0000 7f3258f78640 1 --2- 172.21.15.98:0/1599285052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32400039a0 0x7f3240003dc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f324400b9e0 tx=0x7f32400057f0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.616+0000 7f3252d9d640 1 -- 172.21.15.98:0/1599285052 reap_dead start 2023-11-28T17:47:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.616+0000 7f3258f78640 1 -- 172.21.15.98:0/1599285052 shutdown_connections 2023-11-28T17:47:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.616+0000 7f3258f78640 1 -- 172.21.15.98:0/1599285052 >> 172.21.15.98:0/1599285052 conn(0x7f325406c0f0 msgr2=0x7f325406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.616+0000 7f3258f78640 1 -- 172.21.15.98:0/1599285052 shutdown_connections 2023-11-28T17:47:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.616+0000 7f3258f78640 1 -- 172.21.15.98:0/1599285052 wait complete. 2023-11-28T17:47:42.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.618+0000 7f3258f78640 1 Processor -- start 2023-11-28T17:47:42.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.618+0000 7f3258f78640 1 -- start start 2023-11-28T17:47:42.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.619+0000 7f3258f78640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32400039a0 0x7f3254148620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.619+0000 7f3258f78640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325414cf30 0x7f32540c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.619+0000 7f3258f78640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32541efba0 0x7f32541f0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.619+0000 7f3258f78640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32541ef780 con 0x7f32400039a0 2023-11-28T17:47:42.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.619+0000 7f3258f78640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3254149310 con 0x7f32541efba0 2023-11-28T17:47:42.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.619+0000 7f3258f78640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32541ef470 con 0x7f325414cf30 2023-11-28T17:47:42.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.619+0000 7f325259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32400039a0 0x7f3254148620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.619+0000 7f325259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32400039a0 0x7f3254148620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57518/0 (socket says 172.21.15.98:57518) 2023-11-28T17:47:42.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.619+0000 7f325259c640 1 -- 172.21.15.98:0/3621735316 learned_addr learned my addr 172.21.15.98:0/3621735316 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:42.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.619+0000 7f325259c640 1 -- 172.21.15.98:0/3621735316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325414cf30 msgr2=0x7f32540c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.619+0000 7f324bfff640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325414cf30 0x7f32540c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.620+0000 7f325259c640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f325414cf30 0x7f32540c3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.620+0000 7f325259c640 1 -- 172.21.15.98:0/3621735316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32541efba0 msgr2=0x7f32541f0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.620+0000 7f325259c640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32541efba0 0x7f32541f0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.620+0000 7f325259c640 1 -- 172.21.15.98:0/3621735316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32540c9600 con 0x7f32400039a0 2023-11-28T17:47:42.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.620+0000 7f325259c640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32400039a0 0x7f3254148620 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3244035040 tx=0x7f3244004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:42.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.620+0000 7f324affd640 1 -- 172.21.15.98:0/3621735316 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324403b420 con 0x7f32400039a0 2023-11-28T17:47:42.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.620+0000 7f324affd640 1 -- 172.21.15.98:0/3621735316 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324403b580 con 0x7f32400039a0 2023-11-28T17:47:42.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.620+0000 7f324affd640 1 -- 172.21.15.98:0/3621735316 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324403b8f0 con 0x7f32400039a0 2023-11-28T17:47:42.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.620+0000 7f3258f78640 1 -- 172.21.15.98:0/3621735316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32540c9080 con 0x7f32400039a0 2023-11-28T17:47:42.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.620+0000 7f3258f78640 1 -- 172.21.15.98:0/3621735316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32540c9320 con 0x7f32400039a0 2023-11-28T17:47:42.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.621+0000 7f3258f78640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3254146cd0 0x7f32541470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.621+0000 7f3258f78640 1 -- 172.21.15.98:0/3621735316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32540744d0 con 0x7f3254146cd0 2023-11-28T17:47:42.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.622+0000 7f324bfff640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3254146cd0 0x7f32541470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.622+0000 7f324bfff640 1 -- 172.21.15.98:0/3621735316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f325415dc10 con 0x7f3254146cd0 2023-11-28T17:47:42.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.622+0000 7f324bfff640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3254146cd0 0x7f32541470f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f323c00f450 tx=0x7f323c00f910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:42.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.622+0000 7f324affd640 1 -- 172.21.15.98:0/3621735316 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c0053a0 con 0x7f3254146cd0 2023-11-28T17:47:42.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.622+0000 7f324affd640 1 -- 172.21.15.98:0/3621735316 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f324403ba50 con 0x7f32400039a0 2023-11-28T17:47:42.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.623+0000 7f324affd640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f323406b370 0x7f323406d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.623+0000 7f324affd640 1 -- 172.21.15.98:0/3621735316 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f32440ab9f0 con 0x7f32400039a0 2023-11-28T17:47:42.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.623+0000 7f3252d9d640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f323406b370 0x7f323406d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.624+0000 7f324affd640 1 -- 172.21.15.98:0/3621735316 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f323c007020 con 0x7f3254146cd0 2023-11-28T17:47:42.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.624+0000 7f324affd640 1 -- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3254146cd0 msgr2=0x7f32541470f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.624+0000 7f324affd640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3254146cd0 0x7f32541470f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f323c00f450 tx=0x7f323c00f910 comp rx=0 tx=0).stop 2023-11-28T17:47:42.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.624+0000 7f324affd640 1 -- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3254146cd0 msgr2=0x7f32541470f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:42.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.624+0000 7f324affd640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3254146cd0 0x7f32541470f0 secure :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f323c00f450 tx=0x7f323c00f910 comp rx=0 tx=0).stop 2023-11-28T17:47:42.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.624+0000 7f3252d9d640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f323406b370 0x7f323406d830 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f325415e3f0 tx=0x7f324c0020e0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:42.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.631+0000 7f3258f78640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f325414b1d0 0x7f325414b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.631+0000 7f3258f78640 1 -- 172.21.15.98:0/3621735316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325415d930 con 0x7f325414b1d0 2023-11-28T17:47:42.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.631+0000 7f324bfff640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f325414b1d0 0x7f325414b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.631+0000 7f324bfff640 1 -- 172.21.15.98:0/3621735316 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f325415e130 con 0x7f325414b1d0 2023-11-28T17:47:42.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.631+0000 7f324bfff640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f325414b1d0 0x7f325414b5e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f323c017000 tx=0x7f323c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:42.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.632+0000 7f324affd640 1 -- 172.21.15.98:0/3621735316 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c022070 con 0x7f325414b1d0 2023-11-28T17:47:42.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.632+0000 7f324affd640 1 -- 172.21.15.98:0/3621735316 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4977 (secure 0 0 0) 0x7f323c0067d0 con 0x7f325414b1d0 2023-11-28T17:47:42.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.632+0000 7f324affd640 1 -- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f325414b1d0 msgr2=0x7f325414b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.632+0000 7f324affd640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f325414b1d0 0x7f325414b5e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f323c017000 tx=0x7f323c001120 comp rx=0 tx=0).stop 2023-11-28T17:47:42.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.632+0000 7f324affd640 1 -- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f325414b1d0 msgr2=0x7f325414b5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:42.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.632+0000 7f324affd640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f325414b1d0 0x7f325414b5e0 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f323c017000 tx=0x7f323c001120 comp rx=0 tx=0).stop 2023-11-28T17:47:42.635 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:47:42.636 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:47:42.636 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:47:42.636 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:47:42.636 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 70, 2023-11-28T17:47:42.636 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:47:42.636 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:47:42.637 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:47:42.637 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:47:42.637 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:42.637 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2023-11-28T17:47:42.637 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:42.637 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:47:42.637 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:47:42.637 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:42.638 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:42.638 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:42.638 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:42.638 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:42.638 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:42.638 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:42.638 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:42.639 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:42.639 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:42.639 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:42.639 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:47:42.639 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:47:42.639 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:42.639 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:42.640 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:42.640 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:42.640 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:42.640 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:42.640 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:42.640 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:42.640 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:42.641 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:42.641 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:42.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.641 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:47:42.641 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:47:42.641 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:47:42.641 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:47:42.642 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:47:42.642 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:47:42.642 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:42.642 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:42.642 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:47:42.642 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:47:42.642 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:47:42.643 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:47:42.643 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:47:42.643 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:47:42.643 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:47:42.643 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:42.643 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:47:42.643 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:42.644 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:42.644 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:42.644 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:42.644 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:42.644 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:42.644 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:42.644 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:42.645 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:42.645 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:42.645 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:42.645 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:47:42.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.645 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:47:42.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.645 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:47:42.646 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:47:42.646 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:42.646 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:42.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.646 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:42.647 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:47:42.647 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:42.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.647 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:42.647 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:47:42.648 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:42.648 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.648 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:42.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.648 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:47:42.648 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:47:42.649 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:42.649 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:42.649 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:42.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.649 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:47:42.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:47:42.649 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:42.650 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:42.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:42.650 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:47:42.650 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:42.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:42.651 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:47:42.651 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:42.651 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.651 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:42.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.652 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:47:42.652 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:47:42.652 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:42.652 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:42.652 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:42.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.653 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:47:42.653 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:47:42.653 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:42.653 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:42.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.654 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:42.654 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:47:42.654 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:42.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.654 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:42.654 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:47:42.655 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:42.655 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.655 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:42.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.655 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:47:42.655 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:47:42.655 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:42.656 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:42.656 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:42.656 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.656 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:42.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.656 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:47:42.656 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:47:42.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.657 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:42.657 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:42.657 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:47:42.657 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.657 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:42.657 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:47:42.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.658 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:42.658 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:42.658 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:47:42.658 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.658 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:42.658 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:47:42.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.659 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:42.659 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:42.659 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 6 2023-11-28T17:47:42.659 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.659 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:42.659 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:47:42.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.660 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:42.660 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:42.660 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:47:42.660 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.660 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:42.660 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:47:42.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.661 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:42.661 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:42.661 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:47:42.661 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.661 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:42.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.661 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:47:42.662 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:47:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.632+0000 7f3258f78640 1 -- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f323406b370 msgr2=0x7f323406d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.632+0000 7f3258f78640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f323406b370 0x7f323406d830 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f325415e3f0 tx=0x7f324c0020e0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.632+0000 7f3258f78640 1 -- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32400039a0 msgr2=0x7f3254148620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.632+0000 7f3258f78640 1 --2- 172.21.15.98:0/3621735316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32400039a0 0x7f3254148620 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3244035040 tx=0x7f3244004730 comp rx=0 tx=0).stop 2023-11-28T17:47:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.632+0000 7f3252d9d640 1 -- 172.21.15.98:0/3621735316 reap_dead start 2023-11-28T17:47:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.632+0000 7f3252d9d640 1 -- 172.21.15.98:0/3621735316 reap_dead start 2023-11-28T17:47:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.632+0000 7f3258f78640 1 -- 172.21.15.98:0/3621735316 shutdown_connections 2023-11-28T17:47:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.632+0000 7f3258f78640 1 -- 172.21.15.98:0/3621735316 >> 172.21.15.98:0/3621735316 conn(0x7f325406c0f0 msgr2=0x7f3254152660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.633+0000 7f3258f78640 1 -- 172.21.15.98:0/3621735316 shutdown_connections 2023-11-28T17:47:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.633+0000 7f3258f78640 1 -- 172.21.15.98:0/3621735316 wait complete. 2023-11-28T17:47:42.664 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:47:42.665 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:47:42.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:47:42.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.876+0000 7f3b4d590640 1 Processor -- start 2023-11-28T17:47:42.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.876+0000 7f3b4d590640 1 -- start start 2023-11-28T17:47:42.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.878+0000 7f3b4d590640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b480c4380 0x7f3b480c2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.878+0000 7f3b4d590640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b48156980 con 0x7f3b480c4760 2023-11-28T17:47:42.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.878+0000 7f3b4d590640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b48156c40 con 0x7f3b4815a0f0 2023-11-28T17:47:42.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.878+0000 7f3b4d590640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b480c2a60 con 0x7f3b480c4380 2023-11-28T17:47:42.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.879+0000 7f3b3e7fc640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3b480c4760 0x7f3b480c6020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50726/0 (socket says 172.21.15.98:50726) 2023-11-28T17:47:42.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.879+0000 7f3b3e7fc640 1 -- 172.21.15.98:0/485437242 learned_addr learned my addr 172.21.15.98:0/485437242 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:47:42.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.879+0000 7f3b46ffd640 1 --2- 172.21.15.98:0/485437242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b480c4380 0x7f3b480c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.879+0000 7f3b467fc640 1 -- 172.21.15.98:0/485437242 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226680190 0 0) 0x7f3b48156980 con 0x7f3b480c4760 2023-11-28T17:47:42.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.879+0000 7f3b467fc640 1 -- 172.21.15.98:0/485437242 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b34005730 con 0x7f3b480c4760 2023-11-28T17:47:42.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.879+0000 7f3b46ffd640 1 -- 172.21.15.98:0/485437242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b480c4380 msgr2=0x7f3b480c2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:47:42.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.879+0000 7f3b46ffd640 1 -- 172.21.15.98:0/485437242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b480c4380 msgr2=0x7f3b480c2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:47:42.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.879+0000 7f3b46ffd640 1 --2- 172.21.15.98:0/485437242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b480c4380 0x7f3b480c2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:42.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.879+0000 7f3b46ffd640 1 --2- 172.21.15.98:0/485437242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b480c4380 0x7f3b480c2520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:42.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.879+0000 7f3b467fc640 1 -- 172.21.15.98:0/485437242 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3145739743 0 0) 0x7f3b34005730 con 0x7f3b480c4760 2023-11-28T17:47:42.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.879+0000 7f3b467fc640 1 -- 172.21.15.98:0/485437242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b480c4380 msgr2=0x7f3b480c2520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:42.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.879+0000 7f3b467fc640 1 --2- 172.21.15.98:0/485437242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b480c4380 0x7f3b480c2520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.879+0000 7f3b467fc640 1 -- 172.21.15.98:0/485437242 >> v1:172.21.15.113:6789/0 conn(0x7f3b4815a0f0 legacy=0x7f3b4815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.879+0000 7f3b467fc640 1 -- 172.21.15.98:0/485437242 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b48156f20 con 0x7f3b480c4760 2023-11-28T17:47:42.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.879+0000 7f3b467fc640 1 -- 172.21.15.98:0/485437242 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f3b300019f0 con 0x7f3b480c4760 2023-11-28T17:47:42.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.880+0000 7f3b467fc640 1 -- 172.21.15.98:0/485437242 >> v1:172.21.15.98:6789/0 conn(0x7f3b480c4760 legacy=0x7f3b480c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.880+0000 7f3b467fc640 1 --2- 172.21.15.98:0/485437242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b34005bb0 0x7f3b34005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.880+0000 7f3b467fc640 1 --2- 172.21.15.98:0/485437242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b480c4380 0x7f3b340069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.880+0000 7f3b467fc640 1 --2- 172.21.15.98:0/485437242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b34006f10 0x7f3b34007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.880+0000 7f3b467fc640 1 -- 172.21.15.98:0/485437242 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b480c2a60 con 0x7f3b34006f10 2023-11-28T17:47:42.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.880+0000 7f3b467fc640 1 -- 172.21.15.98:0/485437242 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3400ddb0 con 0x7f3b34005bb0 2023-11-28T17:47:42.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.880+0000 7f3b467fc640 1 -- 172.21.15.98:0/485437242 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3400df40 con 0x7f3b480c4380 2023-11-28T17:47:42.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.880+0000 7f3b477fe640 1 --2- 172.21.15.98:0/485437242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b34006f10 0x7f3b34007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.880+0000 7f3b46ffd640 1 --2- 172.21.15.98:0/485437242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b34005bb0 0x7f3b34005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.880+0000 7f3b3e7fc640 1 --2- 172.21.15.98:0/485437242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b480c4380 0x7f3b340069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.880+0000 7f3b477fe640 1 -- 172.21.15.98:0/485437242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b480c4380 msgr2=0x7f3b340069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.880+0000 7f3b477fe640 1 --2- 172.21.15.98:0/485437242 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b480c4380 0x7f3b340069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.880+0000 7f3b477fe640 1 -- 172.21.15.98:0/485437242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b34005bb0 msgr2=0x7f3b34005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.880+0000 7f3b477fe640 1 --2- 172.21.15.98:0/485437242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b34005bb0 0x7f3b34005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.880+0000 7f3b477fe640 1 -- 172.21.15.98:0/485437242 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b34005730 con 0x7f3b34006f10 2023-11-28T17:47:42.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.881+0000 7f3b477fe640 1 --2- 172.21.15.98:0/485437242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b34006f10 0x7f3b34007390 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3b40001780 tx=0x7f3b34007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a36d26b81bfd4a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.882+0000 7f3b467fc640 1 -- 172.21.15.98:0/485437242 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b400052d0 con 0x7f3b34006f10 2023-11-28T17:47:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.882+0000 7f3b467fc640 1 -- 172.21.15.98:0/485437242 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4002e050 con 0x7f3b34006f10 2023-11-28T17:47:42.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.882+0000 7f3b467fc640 1 -- 172.21.15.98:0/485437242 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b400035f0 con 0x7f3b34006f10 2023-11-28T17:47:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.882+0000 7f3b4d590640 1 -- 172.21.15.98:0/485437242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b34006f10 msgr2=0x7f3b34007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.882+0000 7f3b4d590640 1 --2- 172.21.15.98:0/485437242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b34006f10 0x7f3b34007390 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3b40001780 tx=0x7f3b34007a00 comp rx=0 tx=0).stop 2023-11-28T17:47:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.882+0000 7f3b477fe640 1 -- 172.21.15.98:0/485437242 reap_dead start 2023-11-28T17:47:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.882+0000 7f3b4d590640 1 -- 172.21.15.98:0/485437242 shutdown_connections 2023-11-28T17:47:42.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.882+0000 7f3b4d590640 1 -- 172.21.15.98:0/485437242 >> 172.21.15.98:0/485437242 conn(0x7f3b4806c0f0 msgr2=0x7f3b4806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:42.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.882+0000 7f3b4d590640 1 -- 172.21.15.98:0/485437242 shutdown_connections 2023-11-28T17:47:42.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.882+0000 7f3b4d590640 1 -- 172.21.15.98:0/485437242 wait complete. 2023-11-28T17:47:42.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.884+0000 7f3b4d590640 1 Processor -- start 2023-11-28T17:47:42.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.884+0000 7f3b4d590640 1 -- start start 2023-11-28T17:47:42.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.885+0000 7f3b4d590640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b34006f10 0x7f3b4814ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.885+0000 7f3b4d590640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b481ef2c0 0x7f3b4814a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.885+0000 7f3b4d590640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b481f2fe0 0x7f3b481f3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.885+0000 7f3b4d590640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b481f2870 con 0x7f3b481ef2c0 2023-11-28T17:47:42.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.885+0000 7f3b4d590640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4814e520 con 0x7f3b481f2fe0 2023-11-28T17:47:42.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.885+0000 7f3b4d590640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b481ef820 con 0x7f3b34006f10 2023-11-28T17:47:42.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.885+0000 7f3b3e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b481ef2c0 0x7f3b4814a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.886+0000 7f3b3e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b481ef2c0 0x7f3b4814a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57556/0 (socket says 172.21.15.98:57556) 2023-11-28T17:47:42.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.886+0000 7f3b3e7fc640 1 -- 172.21.15.98:0/3917029781 learned_addr learned my addr 172.21.15.98:0/3917029781 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:42.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.886+0000 7f3b46ffd640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b34006f10 0x7f3b4814ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.886+0000 7f3b477fe640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b481f2fe0 0x7f3b481f3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.886+0000 7f3b3e7fc640 1 -- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b34006f10 msgr2=0x7f3b4814ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.886+0000 7f3b3e7fc640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f3b34006f10 0x7f3b4814ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.886+0000 7f3b3e7fc640 1 -- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b481f2fe0 msgr2=0x7f3b481f3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.886+0000 7f3b3e7fc640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b481f2fe0 0x7f3b481f3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.886+0000 7f3b3e7fc640 1 -- 172.21.15.98:0/3917029781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b481efb00 con 0x7f3b481ef2c0 2023-11-28T17:47:42.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.886+0000 7f3b477fe640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b481f2fe0 0x7f3b481f3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:47:42.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.886+0000 7f3b3e7fc640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b481ef2c0 0x7f3b4814a290 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3b30005fd0 tx=0x7f3b3000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:42.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.886+0000 7f3b3ffff640 1 -- 172.21.15.98:0/3917029781 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3000ee50 con 0x7f3b481ef2c0 2023-11-28T17:47:42.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.887+0000 7f3b3ffff640 1 -- 172.21.15.98:0/3917029781 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b30013420 con 0x7f3b481ef2c0 2023-11-28T17:47:42.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.887+0000 7f3b4d590640 1 -- 172.21.15.98:0/3917029781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b481efde0 con 0x7f3b481ef2c0 2023-11-28T17:47:42.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.887+0000 7f3b3ffff640 1 -- 172.21.15.98:0/3917029781 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b300137b0 con 0x7f3b481ef2c0 2023-11-28T17:47:42.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.887+0000 7f3b4d590640 1 -- 172.21.15.98:0/3917029781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b481f2ac0 con 0x7f3b481ef2c0 2023-11-28T17:47:42.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.887+0000 7f3b4d590640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b48158d40 0x7f3b48159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.888+0000 7f3b4d590640 1 -- 172.21.15.98:0/3917029781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b480744d0 con 0x7f3b48158d40 2023-11-28T17:47:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.888+0000 7f3b46ffd640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b48158d40 0x7f3b48159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.889+0000 7f3b3ffff640 1 -- 172.21.15.98:0/3917029781 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3b30013910 con 0x7f3b481ef2c0 2023-11-28T17:47:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.889+0000 7f3b46ffd640 1 -- 172.21.15.98:0/3917029781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b481577c0 con 0x7f3b48158d40 2023-11-28T17:47:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.889+0000 7f3b46ffd640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b48158d40 0x7f3b48159120 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3b38006fd0 tx=0x7f3b38011920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.889+0000 7f3b3ffff640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3b2806b2a0 0x7f3b2806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.889+0000 7f3b3ffff640 1 -- 172.21.15.98:0/3917029781 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3b3008a600 con 0x7f3b481ef2c0 2023-11-28T17:47:42.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.890+0000 7f3b477fe640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3b2806b2a0 0x7f3b2806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.890+0000 7f3b3ffff640 1 -- 172.21.15.98:0/3917029781 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38015850 con 0x7f3b48158d40 2023-11-28T17:47:42.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.890+0000 7f3b3ffff640 1 -- 172.21.15.98:0/3917029781 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f3b380159b0 con 0x7f3b48158d40 2023-11-28T17:47:42.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.890+0000 7f3b3ffff640 1 -- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b48158d40 msgr2=0x7f3b48159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.890+0000 7f3b3ffff640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b48158d40 0x7f3b48159120 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3b38006fd0 tx=0x7f3b38011920 comp rx=0 tx=0).stop 2023-11-28T17:47:42.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.890+0000 7f3b3ffff640 1 -- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b48158d40 msgr2=0x7f3b48159120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:42.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.890+0000 7f3b3ffff640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b48158d40 0x7f3b48159120 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.890+0000 7f3b477fe640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3b2806b2a0 0x7f3b2806d760 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3b40033000 tx=0x7f3b40009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:42.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.897+0000 7f3b4d590640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b48159660 0x7f3b4802b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:42.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.897+0000 7f3b4d590640 1 -- 172.21.15.98:0/3917029781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b481574e0 con 0x7f3b48159660 2023-11-28T17:47:42.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.898+0000 7f3b46ffd640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b48159660 0x7f3b4802b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:42.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.899+0000 7f3b46ffd640 1 -- 172.21.15.98:0/3917029781 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3b480c5c60 con 0x7f3b48159660 2023-11-28T17:47:42.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.899+0000 7f3b46ffd640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b48159660 0x7f3b4802b810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3b3801d000 tx=0x7f3b380046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:42.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.899+0000 7f3b3ffff640 1 -- 172.21.15.98:0/3917029781 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3802e070 con 0x7f3b48159660 2023-11-28T17:47:42.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.900+0000 7f3b3ffff640 1 -- 172.21.15.98:0/3917029781 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4635 (secure 0 0 0) 0x7f3b380136d0 con 0x7f3b48159660 2023-11-28T17:47:42.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.900+0000 7f3b3ffff640 1 -- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b48159660 msgr2=0x7f3b4802b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.900+0000 7f3b3ffff640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b48159660 0x7f3b4802b810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3b3801d000 tx=0x7f3b380046c0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.900+0000 7f3b3ffff640 1 -- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b48159660 msgr2=0x7f3b4802b810 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:42.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.900+0000 7f3b3ffff640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b48159660 0x7f3b4802b810 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3b3801d000 tx=0x7f3b380046c0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.904 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:47:42.904 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:47:42.904 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:47:42.905 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:47:42.905 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 70, 2023-11-28T17:47:42.905 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:47:42.905 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:47:42.905 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:47:42.905 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:47:42.905 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:42.906 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2023-11-28T17:47:42.906 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:42.906 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:47:42.906 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:47:42.906 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:42.906 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:42.906 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:42.907 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:42.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:42.907 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:42.907 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:42.907 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:42.907 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:42.907 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:42.908 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:42.908 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:47:42.908 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:47:42.908 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:42.908 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:42.908 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:42.908 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:42.909 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:42.909 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:42.909 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:42.909 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:42.909 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:42.909 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:42.909 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:42.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.910 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:47:42.910 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:47:42.910 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:47:42.910 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:47:42.910 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:47:42.910 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:47:42.911 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:42.911 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:42.911 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:47:42.911 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:47:42.911 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:47:42.911 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:47:42.911 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:47:42.912 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:47:42.912 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:47:42.912 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:42.912 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:47:42.912 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:42.912 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:42.912 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:42.912 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:42.913 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:42.913 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:42.913 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:42.913 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:42.913 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:42.913 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:42.913 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:42.914 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:47:42.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.914 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:47:42.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.914 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:47:42.914 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:47:42.914 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:42.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:42.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:42.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:47:42.915 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:42.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:42.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:47:42.916 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:42.916 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.916 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:42.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.917 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:47:42.917 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:47:42.917 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:42.917 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:42.917 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:42.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.918 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:47:42.918 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:47:42.918 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:42.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:42.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:42.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:47:42.918 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:42.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:42.919 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:47:42.919 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:42.919 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.919 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:42.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.920 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:47:42.920 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:47:42.920 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:42.920 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:42.920 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:42.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.921 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:47:42.921 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:47:42.921 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:42.921 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:42.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:42.922 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:47:42.922 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:42.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.922 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:42.922 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:47:42.922 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:42.923 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.923 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:42.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.923 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:47:42.923 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:47:42.923 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:42.923 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:42.924 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:42.924 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.924 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:42.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.924 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:47:42.924 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:47:42.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.925 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:42.925 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:42.925 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:47:42.925 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.925 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:42.925 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:47:42.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.926 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:42.926 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:42.926 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:47:42.926 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.926 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:42.926 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:47:42.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:42.927 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:42.927 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:42.927 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:47:42.927 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:42.927 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:42.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:42.927 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:47:42.928 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:47:42.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.901+0000 7f3b4d590640 1 -- 172.21.15.98:0/3917029781 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3b2806b2a0 msgr2=0x7f3b2806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.901+0000 7f3b4d590640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f3b2806b2a0 0x7f3b2806d760 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3b40033000 tx=0x7f3b40009030 comp rx=0 tx=0).stop 2023-11-28T17:47:42.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.901+0000 7f3b4d590640 1 -- 172.21.15.98:0/3917029781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b481ef2c0 msgr2=0x7f3b4814a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.901+0000 7f3b4d590640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b481ef2c0 0x7f3b4814a290 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3b30005fd0 tx=0x7f3b3000e530 comp rx=0 tx=0).stop 2023-11-28T17:47:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.901+0000 7f3b477fe640 1 -- 172.21.15.98:0/3917029781 reap_dead start 2023-11-28T17:47:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.901+0000 7f3b4d590640 1 -- 172.21.15.98:0/3917029781 shutdown_connections 2023-11-28T17:47:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.901+0000 7f3b4d590640 1 --2- 172.21.15.98:0/3917029781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b481ef2c0 0x7f3b4814a290 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.901+0000 7f3b4d590640 1 -- 172.21.15.98:0/3917029781 >> 172.21.15.98:0/3917029781 conn(0x7f3b4806c0f0 msgr2=0x7f3b4806dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.901+0000 7f3b4d590640 1 -- 172.21.15.98:0/3917029781 shutdown_connections 2023-11-28T17:47:42.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:42.901+0000 7f3b4d590640 1 -- 172.21.15.98:0/3917029781 wait complete. 2023-11-28T17:47:42.931 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:47:42.931 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:47:42.931 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:47:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.144+0000 7fe750e90640 1 Processor -- start 2023-11-28T17:47:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.144+0000 7fe750e90640 1 -- start start 2023-11-28T17:47:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.145+0000 7fe750e90640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe74c0c4760 0x7fe74c0c2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.145+0000 7fe750e90640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c156980 con 0x7fe74c15a0f0 2023-11-28T17:47:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.145+0000 7fe750e90640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c156c40 con 0x7fe74c0c4380 2023-11-28T17:47:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.145+0000 7fe750e90640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe74c0c2a60 con 0x7fe74c0c4760 2023-11-28T17:47:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.146+0000 7fe743fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe74c0c4760 0x7fe74c0c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.146+0000 7fe743fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe74c0c4760 0x7fe74c0c2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57566/0 (socket says 172.21.15.98:57566) 2023-11-28T17:47:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.146+0000 7fe743fff640 1 -- 172.21.15.98:0/3286496948 learned_addr learned my addr 172.21.15.98:0/3286496948 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.146+0000 7fe743fff640 1 -- 172.21.15.98:0/3286496948 >> v1:172.21.15.113:6789/0 conn(0x7fe74c0c4380 legacy=0x7fe74c0c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.146+0000 7fe743fff640 1 -- 172.21.15.98:0/3286496948 >> v1:172.21.15.98:6789/0 conn(0x7fe74c15a0f0 legacy=0x7fe74c15a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.146+0000 7fe743fff640 1 -- 172.21.15.98:0/3286496948 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c156f20 con 0x7fe74c0c4760 2023-11-28T17:47:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.146+0000 7fe749d9b640 1 -- 172.21.15.98:0/3286496948 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446042482 0 0) 0x7fe74c156980 con 0x7fe74c15a0f0 2023-11-28T17:47:43.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.147+0000 7fe743fff640 1 --2- 172.21.15.98:0/3286496948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe74c0c4760 0x7fe74c0c2520 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe73400bb10 tx=0x7fe73402e290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3046d1c231eee7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:43.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.148+0000 7fe749d9b640 1 -- 172.21.15.98:0/3286496948 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734036640 con 0x7fe74c0c4760 2023-11-28T17:47:43.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.148+0000 7fe749d9b640 1 -- 172.21.15.98:0/3286496948 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7340367a0 con 0x7fe74c0c4760 2023-11-28T17:47:43.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.148+0000 7fe749d9b640 1 -- 172.21.15.98:0/3286496948 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734036b30 con 0x7fe74c0c4760 2023-11-28T17:47:43.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.148+0000 7fe750e90640 1 -- 172.21.15.98:0/3286496948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe74c0c4760 msgr2=0x7fe74c0c2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:43.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.148+0000 7fe750e90640 1 --2- 172.21.15.98:0/3286496948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe74c0c4760 0x7fe74c0c2520 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe73400bb10 tx=0x7fe73402e290 comp rx=0 tx=0).stop 2023-11-28T17:47:43.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.148+0000 7fe750e90640 1 -- 172.21.15.98:0/3286496948 shutdown_connections 2023-11-28T17:47:43.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.148+0000 7fe750e90640 1 --2- 172.21.15.98:0/3286496948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe74c0c4760 0x7fe74c0c2520 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:43.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.148+0000 7fe750e90640 1 -- 172.21.15.98:0/3286496948 >> 172.21.15.98:0/3286496948 conn(0x7fe74c06c0f0 msgr2=0x7fe74c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:43.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.149+0000 7fe750e90640 1 -- 172.21.15.98:0/3286496948 shutdown_connections 2023-11-28T17:47:43.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.149+0000 7fe750e90640 1 -- 172.21.15.98:0/3286496948 wait complete. 2023-11-28T17:47:43.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.150+0000 7fe750e90640 1 Processor -- start 2023-11-28T17:47:43.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.150+0000 7fe750e90640 1 -- start start 2023-11-28T17:47:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.151+0000 7fe750e90640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c0c4380 0x7fe74c159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.151+0000 7fe750e90640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe74c0c4760 0x7fe74c0ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.151+0000 7fe750e90640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe74c15a0f0 0x7fe74c141950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.151+0000 7fe750e90640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c159c90 con 0x7fe74c0c4760 2023-11-28T17:47:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.151+0000 7fe750e90640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c14d140 con 0x7fe74c15a0f0 2023-11-28T17:47:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.151+0000 7fe750e90640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c1599b0 con 0x7fe74c0c4380 2023-11-28T17:47:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.151+0000 7fe743fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe74c0c4760 0x7fe74c0ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.151+0000 7fe743fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe74c0c4760 0x7fe74c0ca280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57578/0 (socket says 172.21.15.98:57578) 2023-11-28T17:47:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.151+0000 7fe743fff640 1 -- 172.21.15.98:0/1939329615 learned_addr learned my addr 172.21.15.98:0/1939329615 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.152+0000 7fe743fff640 1 -- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c0c4380 msgr2=0x7fe74c159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.152+0000 7fe74a59c640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c0c4380 0x7fe74c159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.152+0000 7fe743fff640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c0c4380 0x7fe74c159170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:43.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.152+0000 7fe743fff640 1 -- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe74c15a0f0 msgr2=0x7fe74c141950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:43.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.152+0000 7fe74ad9d640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe74c15a0f0 0x7fe74c141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:43.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.152+0000 7fe743fff640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe74c15a0f0 0x7fe74c141950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:43.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.152+0000 7fe743fff640 1 -- 172.21.15.98:0/1939329615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73400b770 con 0x7fe74c0c4760 2023-11-28T17:47:43.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.152+0000 7fe74a59c640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c0c4380 0x7fe74c159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:47:43.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.152+0000 7fe743fff640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe74c0c4760 0x7fe74c0ca280 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe734000c00 tx=0x7fe7340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:43.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.153+0000 7fe742ffd640 1 -- 172.21.15.98:0/1939329615 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734034b80 con 0x7fe74c0c4760 2023-11-28T17:47:43.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.153+0000 7fe742ffd640 1 -- 172.21.15.98:0/1939329615 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe734034d00 con 0x7fe74c0c4760 2023-11-28T17:47:43.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.153+0000 7fe750e90640 1 -- 172.21.15.98:0/1939329615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe74c14c810 con 0x7fe74c0c4760 2023-11-28T17:47:43.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.153+0000 7fe742ffd640 1 -- 172.21.15.98:0/1939329615 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734041680 con 0x7fe74c0c4760 2023-11-28T17:47:43.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.153+0000 7fe750e90640 1 -- 172.21.15.98:0/1939329615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74c14cdd0 con 0x7fe74c0c4760 2023-11-28T17:47:43.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.154+0000 7fe750e90640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 0x7fe74c141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:43.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.154+0000 7fe750e90640 1 -- 172.21.15.98:0/1939329615 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c0744d0 con 0x7fe74c14aac0 2023-11-28T17:47:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.154+0000 7fe74a59c640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 0x7fe74c141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.155+0000 7fe74a59c640 1 -- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 msgr2=0x7fe74c141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2023-11-28T17:47:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.155+0000 7fe74a59c640 1 -- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 msgr2=0x7fe74c141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:47:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.155+0000 7fe74a59c640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 0x7fe74c141e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.155+0000 7fe74a59c640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 0x7fe74c141e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.155+0000 7fe742ffd640 1 -- 172.21.15.98:0/1939329615 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe7340403e0 con 0x7fe74c0c4760 2023-11-28T17:47:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.156+0000 7fe742ffd640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe72c06b2a0 0x7fe72c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.156+0000 7fe742ffd640 1 -- 172.21.15.98:0/1939329615 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe73402ee40 con 0x7fe74c0c4760 2023-11-28T17:47:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.156+0000 7fe74ad9d640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe72c06b2a0 0x7fe72c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:43.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.157+0000 7fe74ad9d640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe72c06b2a0 0x7fe72c06d760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe7440012e0 tx=0x7fe744009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:43.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.356+0000 7fe74a59c640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 0x7fe74c141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:43.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.357+0000 7fe74a59c640 1 -- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 msgr2=0x7fe74c141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2023-11-28T17:47:43.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.357+0000 7fe74a59c640 1 -- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 msgr2=0x7fe74c141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:47:43.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.357+0000 7fe74a59c640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 0x7fe74c141e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:47:43.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.357+0000 7fe74a59c640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 0x7fe74c141e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:47:43.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.758+0000 7fe74a59c640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 0x7fe74c141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:43.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.759+0000 7fe74a59c640 1 -- 172.21.15.98:0/1939329615 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe74c0713f0 con 0x7fe74c14aac0 2023-11-28T17:47:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.760+0000 7fe74a59c640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 0x7fe74c141e90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe73c0013d0 tx=0x7fe73c001890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:43.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.761+0000 7fe742ffd640 1 -- 172.21.15.98:0/1939329615 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c007020 con 0x7fe74c14aac0 2023-11-28T17:47:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.762+0000 7fe742ffd640 1 -- 172.21.15.98:0/1939329615 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fe73c00d070 con 0x7fe74c14aac0 2023-11-28T17:47:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.762+0000 7fe742ffd640 1 -- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 msgr2=0x7fe74c141e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.762+0000 7fe742ffd640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 0x7fe74c141e90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe73c0013d0 tx=0x7fe73c001890 comp rx=0 tx=0).stop 2023-11-28T17:47:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.762+0000 7fe742ffd640 1 -- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 msgr2=0x7fe74c141e90 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:43.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.762+0000 7fe742ffd640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c14aac0 0x7fe74c141e90 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe73c0013d0 tx=0x7fe73c001890 comp rx=0 tx=0).stop 2023-11-28T17:47:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.770+0000 7fe750e90640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c1551e0 0x7fe74c1555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.770+0000 7fe750e90640 1 -- 172.21.15.98:0/1939329615 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c14c070 con 0x7fe74c1551e0 2023-11-28T17:47:43.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.771+0000 7fe74a59c640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c1551e0 0x7fe74c1555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.771+0000 7fe74a59c640 1 -- 172.21.15.98:0/1939329615 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe74c1577c0 con 0x7fe74c1551e0 2023-11-28T17:47:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.772+0000 7fe74a59c640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c1551e0 0x7fe74c1555f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe73c005040 tx=0x7fe73c001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:43.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.772+0000 7fe742ffd640 1 -- 172.21.15.98:0/1939329615 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c023070 con 0x7fe74c1551e0 2023-11-28T17:47:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.773+0000 7fe742ffd640 1 -- 172.21.15.98:0/1939329615 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4635 (secure 0 0 0) 0x7fe73c006e60 con 0x7fe74c1551e0 2023-11-28T17:47:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.773+0000 7fe742ffd640 1 -- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c1551e0 msgr2=0x7fe74c1555f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.773+0000 7fe742ffd640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c1551e0 0x7fe74c1555f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe73c005040 tx=0x7fe73c001120 comp rx=0 tx=0).stop 2023-11-28T17:47:43.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.773+0000 7fe742ffd640 1 -- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c1551e0 msgr2=0x7fe74c1555f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:43.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.773+0000 7fe742ffd640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fe74c1551e0 0x7fe74c1555f0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe73c005040 tx=0x7fe73c001120 comp rx=0 tx=0).stop 2023-11-28T17:47:43.776 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:47:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:47:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:47:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:47:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 70, 2023-11-28T17:47:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:47:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:47:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:47:43.776 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:47:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2023-11-28T17:47:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:47:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:47:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:43.777 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:47:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:47:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:43.778 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:43.779 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:43.779 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:43.779 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:43.779 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:43.779 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:43.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:43.779 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:43.779 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:43.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:43.779 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:47:43.779 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:47:43.780 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:47:43.780 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:47:43.780 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:47:43.780 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:47:43.780 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:43.780 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:43.780 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:47:43.780 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:47:43.780 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:47:43.780 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:47:43.780 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:47:43.780 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:47:43.781 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:47:43.781 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:43.781 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:47:43.781 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:43.781 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:43.781 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:43.781 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:43.781 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:43.781 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:43.781 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:43.781 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:43.781 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:43.781 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:43.781 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:47:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:47:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:47:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:47:43.782 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:47:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:47:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:43.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:47:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:47:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:47:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:47:43.784 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:47:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:47:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:43.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:47:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:47:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:47:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:47:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:43.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:47:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:47:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:43.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:47:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:47:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:47:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:47:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:43.788 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:47:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:47:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:47:43.789 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:47:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:47:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:43.790 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:47:43.790 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:47:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.774+0000 7fe750e90640 1 -- 172.21.15.98:0/1939329615 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe72c06b2a0 msgr2=0x7fe72c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.774+0000 7fe750e90640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fe72c06b2a0 0x7fe72c06d760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe7440012e0 tx=0x7fe744009040 comp rx=0 tx=0).stop 2023-11-28T17:47:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.774+0000 7fe750e90640 1 -- 172.21.15.98:0/1939329615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe74c0c4760 msgr2=0x7fe74c0ca280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.774+0000 7fe750e90640 1 --2- 172.21.15.98:0/1939329615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe74c0c4760 0x7fe74c0ca280 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe734000c00 tx=0x7fe7340046c0 comp rx=0 tx=0).stop 2023-11-28T17:47:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.774+0000 7fe74ad9d640 1 -- 172.21.15.98:0/1939329615 reap_dead start 2023-11-28T17:47:43.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.774+0000 7fe74ad9d640 1 -- 172.21.15.98:0/1939329615 reap_dead start 2023-11-28T17:47:43.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.774+0000 7fe750e90640 1 -- 172.21.15.98:0/1939329615 shutdown_connections 2023-11-28T17:47:43.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.774+0000 7fe750e90640 1 -- 172.21.15.98:0/1939329615 >> 172.21.15.98:0/1939329615 conn(0x7fe74c06c0f0 msgr2=0x7fe74c1504b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:43.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.775+0000 7fe750e90640 1 -- 172.21.15.98:0/1939329615 shutdown_connections 2023-11-28T17:47:43.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:43.775+0000 7fe750e90640 1 -- 172.21.15.98:0/1939329615 wait complete. 2023-11-28T17:47:43.800 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:47:43.800 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-11-28T17:47:43.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-11-28T17:47:44.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.017+0000 7f705d578640 1 Processor -- start 2023-11-28T17:47:44.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.017+0000 7f705d578640 1 -- start start 2023-11-28T17:47:44.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.019+0000 7f705d578640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7058151d80 0x7f7058157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:44.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.019+0000 7f705d578640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f705815bc10 0x7f70580c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:44.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.019+0000 7f705d578640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f705815b150 con 0x7f70580c6140 2023-11-28T17:47:44.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.019+0000 7f705d578640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f705806e1f0 con 0x7f705815bc10 2023-11-28T17:47:44.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.019+0000 7f705d578640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705806e4d0 con 0x7f7058151d80 2023-11-28T17:47:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.019+0000 7f7056ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7058151d80 0x7f7058157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.019+0000 7f70567fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f705815bc10 0x7f70580c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.020+0000 7f7056ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7058151d80 0x7f7058157b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:53950/0 (socket says 172.21.15.98:53950) 2023-11-28T17:47:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.020+0000 7f7056ffd640 1 -- 172.21.15.98:0/1838668210 learned_addr learned my addr 172.21.15.98:0/1838668210 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.020+0000 7f70567fc640 1 -- 172.21.15.98:0/1838668210 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7058151d80 msgr2=0x7f7058157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.020+0000 7f70567fc640 1 --2- 172.21.15.98:0/1838668210 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7058151d80 0x7f7058157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.020+0000 7f70567fc640 1 -- 172.21.15.98:0/1838668210 >> v1:172.21.15.113:6789/0 conn(0x7f70580c6140 legacy=0x7f70580c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:44.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.020+0000 7f70567fc640 1 -- 172.21.15.98:0/1838668210 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705806e7b0 con 0x7f705815bc10 2023-11-28T17:47:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.021+0000 7f70567fc640 1 --2- 172.21.15.98:0/1838668210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705815bc10 0x7f70580c5c00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f704c0097a0 tx=0x7f704c02b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da2706f387da0d25 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.022+0000 7f7055ffb640 1 -- 172.21.15.98:0/1838668210 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c030800 con 0x7f705815bc10 2023-11-28T17:47:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.022+0000 7f7055ffb640 1 -- 172.21.15.98:0/1838668210 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704c030960 con 0x7f705815bc10 2023-11-28T17:47:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.022+0000 7f7055ffb640 1 -- 172.21.15.98:0/1838668210 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c030cf0 con 0x7f705815bc10 2023-11-28T17:47:44.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.023+0000 7f705d578640 1 -- 172.21.15.98:0/1838668210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705815bc10 msgr2=0x7f70580c5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:44.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.023+0000 7f705d578640 1 --2- 172.21.15.98:0/1838668210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705815bc10 0x7f70580c5c00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f704c0097a0 tx=0x7f704c02b940 comp rx=0 tx=0).stop 2023-11-28T17:47:44.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.023+0000 7f705d578640 1 -- 172.21.15.98:0/1838668210 shutdown_connections 2023-11-28T17:47:44.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.023+0000 7f705d578640 1 --2- 172.21.15.98:0/1838668210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705815bc10 0x7f70580c5c00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:44.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.023+0000 7f705d578640 1 --2- 172.21.15.98:0/1838668210 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7058151d80 0x7f7058157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:44.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.023+0000 7f705d578640 1 -- 172.21.15.98:0/1838668210 >> 172.21.15.98:0/1838668210 conn(0x7f705806a4d0 msgr2=0x7f705806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:44.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.023+0000 7f705d578640 1 -- 172.21.15.98:0/1838668210 shutdown_connections 2023-11-28T17:47:44.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.023+0000 7f705d578640 1 -- 172.21.15.98:0/1838668210 wait complete. 2023-11-28T17:47:44.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.025+0000 7f705d578640 1 Processor -- start 2023-11-28T17:47:44.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.025+0000 7f705d578640 1 -- start start 2023-11-28T17:47:44.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.026+0000 7f705d578640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70580c6140 0x7f7058144f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:44.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.026+0000 7f705d578640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7058151d80 0x7f7058145460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:44.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.026+0000 7f705d578640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705815bc10 0x7f7058159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:44.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.026+0000 7f705d578640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705814fec0 con 0x7f70580c6140 2023-11-28T17:47:44.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.027+0000 7f705d578640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7058156720 con 0x7f705815bc10 2023-11-28T17:47:44.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.027+0000 7f705d578640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70581569c0 con 0x7f7058151d80 2023-11-28T17:47:44.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.027+0000 7f7056ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70580c6140 0x7f7058144f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.027+0000 7f7056ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70580c6140 0x7f7058144f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57586/0 (socket says 172.21.15.98:57586) 2023-11-28T17:47:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.027+0000 7f7056ffd640 1 -- 172.21.15.98:0/1536748693 learned_addr learned my addr 172.21.15.98:0/1536748693 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.027+0000 7f70567fc640 1 --2- 172.21.15.98:0/1536748693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7058151d80 0x7f7058145460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.027+0000 7f70577fe640 1 --2- 172.21.15.98:0/1536748693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705815bc10 0x7f7058159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:44.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.027+0000 7f7056ffd640 1 -- 172.21.15.98:0/1536748693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7058151d80 msgr2=0x7f7058145460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.027+0000 7f7056ffd640 1 --2- 172.21.15.98:0/1536748693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7058151d80 0x7f7058145460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.027+0000 7f7056ffd640 1 -- 172.21.15.98:0/1536748693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705815bc10 msgr2=0x7f7058159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.027+0000 7f7056ffd640 1 --2- 172.21.15.98:0/1536748693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705815bc10 0x7f7058159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.027+0000 7f7056ffd640 1 -- 172.21.15.98:0/1536748693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704c009400 con 0x7f70580c6140 2023-11-28T17:47:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.028+0000 7f7056ffd640 1 --2- 172.21.15.98:0/1536748693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70580c6140 0x7f7058144f20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7048009fd0 tx=0x7f704800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:44.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.028+0000 7f703f7fe640 1 -- 172.21.15.98:0/1536748693 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704800c940 con 0x7f70580c6140 2023-11-28T17:47:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.028+0000 7f703f7fe640 1 -- 172.21.15.98:0/1536748693 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704800caa0 con 0x7f70580c6140 2023-11-28T17:47:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.028+0000 7f705d578640 1 -- 172.21.15.98:0/1536748693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7058156ca0 con 0x7f70580c6140 2023-11-28T17:47:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.028+0000 7f703f7fe640 1 -- 172.21.15.98:0/1536748693 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704800ce30 con 0x7f70580c6140 2023-11-28T17:47:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.028+0000 7f705d578640 1 -- 172.21.15.98:0/1536748693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7058159000 con 0x7f70580c6140 2023-11-28T17:47:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.030+0000 7f705d578640 1 -- 172.21.15.98:0/1536748693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7058067bf0 con 0x7f70580c6140 2023-11-28T17:47:44.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.030+0000 7f703f7fe640 1 -- 172.21.15.98:0/1536748693 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f7048013470 con 0x7f70580c6140 2023-11-28T17:47:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.031+0000 7f703f7fe640 1 --2- 172.21.15.98:0/1536748693 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f704406b2a0 0x7f704406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.031+0000 7f703f7fe640 1 -- 172.21.15.98:0/1536748693 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f704808a180 con 0x7f70580c6140 2023-11-28T17:47:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.031+0000 7f70567fc640 1 --2- 172.21.15.98:0/1536748693 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f704406b2a0 0x7f704406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.032+0000 7f70567fc640 1 --2- 172.21.15.98:0/1536748693 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f704406b2a0 0x7f704406d760 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f704c036000 tx=0x7f704c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.035+0000 7f703f7fe640 1 -- 172.21.15.98:0/1536748693 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7048054f70 con 0x7f70580c6140 2023-11-28T17:47:44.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.296+0000 7f705d578640 1 -- 172.21.15.98:0/1536748693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f705806b2d0 con 0x7f70580c6140 2023-11-28T17:47:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.297+0000 7f703f7fe640 1 -- 172.21.15.98:0/1536748693 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f7048058590 con 0x7f70580c6140 2023-11-28T17:47:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.299+0000 7f705d578640 1 -- 172.21.15.98:0/1536748693 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f704406b2a0 msgr2=0x7f704406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.299+0000 7f705d578640 1 --2- 172.21.15.98:0/1536748693 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f704406b2a0 0x7f704406d760 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f704c036000 tx=0x7f704c008030 comp rx=0 tx=0).stop 2023-11-28T17:47:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.299+0000 7f705d578640 1 -- 172.21.15.98:0/1536748693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70580c6140 msgr2=0x7f7058144f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.299+0000 7f705d578640 1 --2- 172.21.15.98:0/1536748693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70580c6140 0x7f7058144f20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7048009fd0 tx=0x7f704800ed40 comp rx=0 tx=0).stop 2023-11-28T17:47:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.300+0000 7f705d578640 1 -- 172.21.15.98:0/1536748693 shutdown_connections 2023-11-28T17:47:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.300+0000 7f705d578640 1 --2- 172.21.15.98:0/1536748693 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f704406b2a0 0x7f704406d760 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.300+0000 7f705d578640 1 --2- 172.21.15.98:0/1536748693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705815bc10 0x7f7058159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.300+0000 7f705d578640 1 --2- 172.21.15.98:0/1536748693 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7058151d80 0x7f7058145460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.300+0000 7f705d578640 1 --2- 172.21.15.98:0/1536748693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70580c6140 0x7f7058144f20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.300+0000 7f705d578640 1 -- 172.21.15.98:0/1536748693 >> 172.21.15.98:0/1536748693 conn(0x7f705806a4d0 msgr2=0x7f705814f510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.300+0000 7f705d578640 1 -- 172.21.15.98:0/1536748693 shutdown_connections 2023-11-28T17:47:44.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:44.300+0000 7f705d578640 1 -- 172.21.15.98:0/1536748693 wait complete. 2023-11-28T17:47:44.322 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2023-11-28T17:47:47.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~175s 2023-11-28T17:47:53.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~182s 2023-11-28T17:47:54.323 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:47:54.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:47:54.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.545+0000 7f134a15a640 1 Processor -- start 2023-11-28T17:47:54.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.545+0000 7f134a15a640 1 -- start start 2023-11-28T17:47:54.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.547+0000 7f134a15a640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1344157b60 0x7f134415bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:54.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.547+0000 7f134a15a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134415b150 con 0x7f1344151d80 2023-11-28T17:47:54.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.547+0000 7f134a15a640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134406e1f0 con 0x7f13440c4910 2023-11-28T17:47:54.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.547+0000 7f134a15a640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1344157f90 con 0x7f1344157b60 2023-11-28T17:47:54.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.547+0000 7f13437fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1344151d80 0x7f13440c5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43562/0 (socket says 172.21.15.98:43562) 2023-11-28T17:47:54.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.547+0000 7f13437fe640 1 -- 172.21.15.98:0/3741402956 learned_addr learned my addr 172.21.15.98:0/3741402956 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:47:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f1342ffd640 1 --2- 172.21.15.98:0/3741402956 >> v2:172.21.15.113:3300/0 conn(0x7f1344157b60 0x7f134415bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f13427fc640 1 -- 172.21.15.98:0/3741402956 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999232482 0 0) 0x7f134415b150 con 0x7f1344151d80 2023-11-28T17:47:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f13427fc640 1 -- 172.21.15.98:0/3741402956 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1330003520 con 0x7f1344151d80 2023-11-28T17:47:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f13427fc640 1 -- 172.21.15.98:0/3741402956 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3810113004 0 0) 0x7f1330003520 con 0x7f1344151d80 2023-11-28T17:47:54.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f13427fc640 1 -- 172.21.15.98:0/3741402956 >> v2:172.21.15.113:3300/0 conn(0x7f1344157b60 msgr2=0x7f134415bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:54.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f13427fc640 1 --2- 172.21.15.98:0/3741402956 >> v2:172.21.15.113:3300/0 conn(0x7f1344157b60 0x7f134415bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:54.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f13427fc640 1 -- 172.21.15.98:0/3741402956 >> v1:172.21.15.113:6789/0 conn(0x7f13440c4910 legacy=0x7f13441624e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:54.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f13427fc640 1 -- 172.21.15.98:0/3741402956 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134406e4d0 con 0x7f1344151d80 2023-11-28T17:47:54.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f13427fc640 1 -- 172.21.15.98:0/3741402956 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f13340019f0 con 0x7f1344151d80 2023-11-28T17:47:54.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f13427fc640 1 -- 172.21.15.98:0/3741402956 >> v1:172.21.15.98:6789/0 conn(0x7f1344151d80 legacy=0x7f13440c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:54.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f13427fc640 1 --2- 172.21.15.98:0/3741402956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1330003970 0x7f1330003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:54.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f13427fc640 1 --2- 172.21.15.98:0/3741402956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13300047c0 0x7f1330004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:54.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f13427fc640 1 --2- 172.21.15.98:0/3741402956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1330005180 0x7f133000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:54.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f13427fc640 1 -- 172.21.15.98:0/3741402956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133000c000 con 0x7f13300047c0 2023-11-28T17:47:54.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f13427fc640 1 -- 172.21.15.98:0/3741402956 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133000c170 con 0x7f1330003970 2023-11-28T17:47:54.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.548+0000 7f13427fc640 1 -- 172.21.15.98:0/3741402956 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133000c330 con 0x7f1330005180 2023-11-28T17:47:54.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.549+0000 7f1342ffd640 1 --2- 172.21.15.98:0/3741402956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13300047c0 0x7f1330004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:54.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.549+0000 7f13437fe640 1 --2- 172.21.15.98:0/3741402956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1330003970 0x7f1330003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:54.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.549+0000 7f1342ffd640 1 -- 172.21.15.98:0/3741402956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1330005180 msgr2=0x7f133000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:54.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.549+0000 7f1343fff640 1 --2- 172.21.15.98:0/3741402956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1330005180 0x7f133000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:54.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.549+0000 7f1342ffd640 1 --2- 172.21.15.98:0/3741402956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1330005180 0x7f133000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:54.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.549+0000 7f1342ffd640 1 -- 172.21.15.98:0/3741402956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1330003970 msgr2=0x7f1330003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:54.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.549+0000 7f1342ffd640 1 --2- 172.21.15.98:0/3741402956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1330003970 0x7f1330003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:54.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.549+0000 7f1342ffd640 1 -- 172.21.15.98:0/3741402956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1330003520 con 0x7f13300047c0 2023-11-28T17:47:54.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.549+0000 7f1343fff640 1 --2- 172.21.15.98:0/3741402956 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1330005180 0x7f133000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:47:54.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.549+0000 7f1343fff640 1 -- 172.21.15.98:0/3741402956 reap_dead start 2023-11-28T17:47:54.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.550+0000 7f1342ffd640 1 --2- 172.21.15.98:0/3741402956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13300047c0 0x7f1330004c40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f132c009ac0 tx=0x7f13300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ecc798ee9604c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:54.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.551+0000 7f13427fc640 1 -- 172.21.15.98:0/3741402956 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c02c6b0 con 0x7f13300047c0 2023-11-28T17:47:54.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.551+0000 7f13427fc640 1 -- 172.21.15.98:0/3741402956 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132c02c810 con 0x7f13300047c0 2023-11-28T17:47:54.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.551+0000 7f13427fc640 1 -- 172.21.15.98:0/3741402956 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c02cba0 con 0x7f13300047c0 2023-11-28T17:47:54.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.551+0000 7f134a15a640 1 -- 172.21.15.98:0/3741402956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13300047c0 msgr2=0x7f1330004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:54.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.551+0000 7f134a15a640 1 --2- 172.21.15.98:0/3741402956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13300047c0 0x7f1330004c40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f132c009ac0 tx=0x7f13300057f0 comp rx=0 tx=0).stop 2023-11-28T17:47:54.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.551+0000 7f134a15a640 1 -- 172.21.15.98:0/3741402956 shutdown_connections 2023-11-28T17:47:54.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.551+0000 7f134a15a640 1 --2- 172.21.15.98:0/3741402956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13300047c0 0x7f1330004c40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:54.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.551+0000 7f134a15a640 1 -- 172.21.15.98:0/3741402956 >> 172.21.15.98:0/3741402956 conn(0x7f134406a4d0 msgr2=0x7f134406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:54.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.551+0000 7f134a15a640 1 -- 172.21.15.98:0/3741402956 shutdown_connections 2023-11-28T17:47:54.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.551+0000 7f134a15a640 1 -- 172.21.15.98:0/3741402956 wait complete. 2023-11-28T17:47:54.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.553+0000 7f134a15a640 1 Processor -- start 2023-11-28T17:47:54.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.554+0000 7f134a15a640 1 -- start start 2023-11-28T17:47:54.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.554+0000 7f134a15a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13300047c0 0x7f134414daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:54.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.554+0000 7f134a15a640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1344071570 0x7f13441f1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:54.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.554+0000 7f134a15a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13441ee100 0x7f13441e45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:54.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.554+0000 7f134a15a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13441ed6f0 con 0x7f13441ee100 2023-11-28T17:47:54.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.554+0000 7f134a15a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134414fdf0 con 0x7f13300047c0 2023-11-28T17:47:54.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.554+0000 7f134a15a640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13441f1bb0 con 0x7f1344071570 2023-11-28T17:47:54.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.555+0000 7f1343fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13441ee100 0x7f13441e45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:54.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.555+0000 7f1343fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13441ee100 0x7f13441e45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60514/0 (socket says 172.21.15.98:60514) 2023-11-28T17:47:54.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.555+0000 7f1343fff640 1 -- 172.21.15.98:0/4262727449 learned_addr learned my addr 172.21.15.98:0/4262727449 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:54.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.555+0000 7f1342ffd640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1344071570 0x7f13441f1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:54.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.555+0000 7f13437fe640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13300047c0 0x7f134414daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:54.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.555+0000 7f1343fff640 1 -- 172.21.15.98:0/4262727449 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1344071570 msgr2=0x7f13441f1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:54.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.555+0000 7f1343fff640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1344071570 0x7f13441f1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:54.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.555+0000 7f1343fff640 1 -- 172.21.15.98:0/4262727449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13300047c0 msgr2=0x7f134414daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:54.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.555+0000 7f1343fff640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13300047c0 0x7f134414daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:54.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.555+0000 7f1343fff640 1 -- 172.21.15.98:0/4262727449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134414f830 con 0x7f13441ee100 2023-11-28T17:47:54.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.555+0000 7f1342ffd640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1344071570 0x7f13441f1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:47:54.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.555+0000 7f13437fe640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13300047c0 0x7f134414daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:47:54.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.556+0000 7f1343fff640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13441ee100 0x7f13441e45a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f13440c56e0 tx=0x7f133c00d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:54.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.556+0000 7f133bfff640 1 -- 172.21.15.98:0/4262727449 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c00b070 con 0x7f13441ee100 2023-11-28T17:47:54.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.556+0000 7f133bfff640 1 -- 172.21.15.98:0/4262727449 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133c0035f0 con 0x7f13441ee100 2023-11-28T17:47:54.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.556+0000 7f133bfff640 1 -- 172.21.15.98:0/4262727449 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c007ea0 con 0x7f13441ee100 2023-11-28T17:47:54.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.556+0000 7f134a15a640 1 -- 172.21.15.98:0/4262727449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134414fb10 con 0x7f13441ee100 2023-11-28T17:47:54.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.556+0000 7f134a15a640 1 -- 172.21.15.98:0/4262727449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13441edbe0 con 0x7f13441ee100 2023-11-28T17:47:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.557+0000 7f134a15a640 1 -- 172.21.15.98:0/4262727449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1344067bf0 con 0x7f13441ee100 2023-11-28T17:47:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.558+0000 7f133bfff640 1 -- 172.21.15.98:0/4262727449 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f133c0039a0 con 0x7f13441ee100 2023-11-28T17:47:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.559+0000 7f133bfff640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f132406b2a0 0x7f132406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.559+0000 7f133bfff640 1 -- 172.21.15.98:0/4262727449 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f133c089260 con 0x7f13441ee100 2023-11-28T17:47:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.559+0000 7f13437fe640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f132406b2a0 0x7f132406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.560+0000 7f13437fe640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f132406b2a0 0x7f132406d760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1334001580 tx=0x7f1334001490 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:54.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.563+0000 7f133bfff640 1 -- 172.21.15.98:0/4262727449 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f133c053fc0 con 0x7f13441ee100 2023-11-28T17:47:54.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.823+0000 7f134a15a640 1 -- 172.21.15.98:0/4262727449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1344074f60 con 0x7f13441ee100 2023-11-28T17:47:54.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.824+0000 7f133bfff640 1 -- 172.21.15.98:0/4262727449 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f133c018080 con 0x7f13441ee100 2023-11-28T17:47:54.826 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:47:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.826+0000 7f134a15a640 1 -- 172.21.15.98:0/4262727449 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f132406b2a0 msgr2=0x7f132406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.826+0000 7f134a15a640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f132406b2a0 0x7f132406d760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1334001580 tx=0x7f1334001490 comp rx=0 tx=0).stop 2023-11-28T17:47:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.826+0000 7f134a15a640 1 -- 172.21.15.98:0/4262727449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13441ee100 msgr2=0x7f13441e45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.826+0000 7f134a15a640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13441ee100 0x7f13441e45a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f13440c56e0 tx=0x7f133c00d890 comp rx=0 tx=0).stop 2023-11-28T17:47:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.827+0000 7f134a15a640 1 -- 172.21.15.98:0/4262727449 shutdown_connections 2023-11-28T17:47:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.827+0000 7f134a15a640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13441ee100 0x7f13441e45a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.827+0000 7f134a15a640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1344071570 0x7f13441f1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.827+0000 7f134a15a640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f132406b2a0 0x7f132406d760 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.827+0000 7f134a15a640 1 --2- 172.21.15.98:0/4262727449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f13300047c0 0x7f134414daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.827+0000 7f134a15a640 1 -- 172.21.15.98:0/4262727449 >> 172.21.15.98:0/4262727449 conn(0x7f134406a4d0 msgr2=0x7f13440ca010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.827+0000 7f134a15a640 1 -- 172.21.15.98:0/4262727449 shutdown_connections 2023-11-28T17:47:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:54.827+0000 7f134a15a640 1 -- 172.21.15.98:0/4262727449 wait complete. 2023-11-28T17:47:54.845 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:47:54.845 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-11-28T17:47:54.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:47:54.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:47:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.051+0000 7f418f694640 1 Processor -- start 2023-11-28T17:47:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.051+0000 7f418f694640 1 -- start start 2023-11-28T17:47:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.052+0000 7f418f694640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f418806a740 0x7f41880c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.053+0000 7f418f694640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41880c81a0 con 0x7f4188154120 2023-11-28T17:47:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.053+0000 7f418f694640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418806e8f0 con 0x7f418806ab20 2023-11-28T17:47:55.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.053+0000 7f418f694640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418806dce0 con 0x7f418806a740 2023-11-28T17:47:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.053+0000 7f418dc31640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4188154120 0x7f4188159ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43566/0 (socket says 172.21.15.98:43566) 2023-11-28T17:47:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.053+0000 7f418dc31640 1 -- 172.21.15.98:0/2859397731 learned_addr learned my addr 172.21.15.98:0/2859397731 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:47:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.053+0000 7f418d430640 1 --2- 172.21.15.98:0/2859397731 >> v2:172.21.15.113:3300/0 conn(0x7f418806a740 0x7f41880c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.053+0000 7f417bfff640 1 -- 172.21.15.98:0/2859397731 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950716516 0 0) 0x7f41880c81a0 con 0x7f4188154120 2023-11-28T17:47:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.053+0000 7f417bfff640 1 -- 172.21.15.98:0/2859397731 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4174003520 con 0x7f4188154120 2023-11-28T17:47:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f417bfff640 1 -- 172.21.15.98:0/2859397731 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1207630693 0 0) 0x7f4174003520 con 0x7f4188154120 2023-11-28T17:47:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f417bfff640 1 -- 172.21.15.98:0/2859397731 >> v2:172.21.15.113:3300/0 conn(0x7f418806a740 msgr2=0x7f41880c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f417bfff640 1 --2- 172.21.15.98:0/2859397731 >> v2:172.21.15.113:3300/0 conn(0x7f418806a740 0x7f41880c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f417bfff640 1 -- 172.21.15.98:0/2859397731 >> v1:172.21.15.113:6789/0 conn(0x7f418806ab20 legacy=0x7f4188153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f417bfff640 1 -- 172.21.15.98:0/2859397731 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418806dfc0 con 0x7f4188154120 2023-11-28T17:47:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f417bfff640 1 -- 172.21.15.98:0/2859397731 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f41800019f0 con 0x7f4188154120 2023-11-28T17:47:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f417bfff640 1 -- 172.21.15.98:0/2859397731 >> v1:172.21.15.98:6789/0 conn(0x7f4188154120 legacy=0x7f4188159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f417bfff640 1 --2- 172.21.15.98:0/2859397731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4174003970 0x7f4174003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f417bfff640 1 --2- 172.21.15.98:0/2859397731 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41740047c0 0x7f4174004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f417bfff640 1 --2- 172.21.15.98:0/2859397731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4174005180 0x7f417400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f417bfff640 1 -- 172.21.15.98:0/2859397731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418806dce0 con 0x7f4174005180 2023-11-28T17:47:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f417bfff640 1 -- 172.21.15.98:0/2859397731 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417400c0b0 con 0x7f4174003970 2023-11-28T17:47:55.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f417bfff640 1 -- 172.21.15.98:0/2859397731 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417400c270 con 0x7f41740047c0 2023-11-28T17:47:55.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f418dc31640 1 --2- 172.21.15.98:0/2859397731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4174005180 0x7f417400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f418d430640 1 --2- 172.21.15.98:0/2859397731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4174003970 0x7f4174003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f418dc31640 1 -- 172.21.15.98:0/2859397731 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41740047c0 msgr2=0x7f4174004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:55.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f418dc31640 1 --2- 172.21.15.98:0/2859397731 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f41740047c0 0x7f4174004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f418dc31640 1 -- 172.21.15.98:0/2859397731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4174003970 msgr2=0x7f4174003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:55.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f418dc31640 1 --2- 172.21.15.98:0/2859397731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4174003970 0x7f4174003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.054+0000 7f418dc31640 1 -- 172.21.15.98:0/2859397731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4174003520 con 0x7f4174005180 2023-11-28T17:47:55.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.055+0000 7f418dc31640 1 -- 172.21.15.98:0/2859397731 reap_dead start 2023-11-28T17:47:55.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.055+0000 7f418dc31640 1 --2- 172.21.15.98:0/2859397731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4174005180 0x7f417400b980 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4188157870 tx=0x7f41740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d1a82805aa69d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.056+0000 7f417bfff640 1 -- 172.21.15.98:0/2859397731 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41800072d0 con 0x7f4174005180 2023-11-28T17:47:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.056+0000 7f417bfff640 1 -- 172.21.15.98:0/2859397731 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418002f070 con 0x7f4174005180 2023-11-28T17:47:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.056+0000 7f417bfff640 1 -- 172.21.15.98:0/2859397731 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4180007670 con 0x7f4174005180 2023-11-28T17:47:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.056+0000 7f418f694640 1 -- 172.21.15.98:0/2859397731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4174005180 msgr2=0x7f417400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.056+0000 7f418f694640 1 --2- 172.21.15.98:0/2859397731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4174005180 0x7f417400b980 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4188157870 tx=0x7f41740057f0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.056+0000 7f418f694640 1 -- 172.21.15.98:0/2859397731 shutdown_connections 2023-11-28T17:47:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.056+0000 7f418f694640 1 --2- 172.21.15.98:0/2859397731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4174005180 0x7f417400b980 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.056+0000 7f418f694640 1 -- 172.21.15.98:0/2859397731 >> 172.21.15.98:0/2859397731 conn(0x7f418806ef10 msgr2=0x7f418806f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.056+0000 7f418f694640 1 -- 172.21.15.98:0/2859397731 shutdown_connections 2023-11-28T17:47:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.056+0000 7f418f694640 1 -- 172.21.15.98:0/2859397731 wait complete. 2023-11-28T17:47:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.058+0000 7f418f694640 1 Processor -- start 2023-11-28T17:47:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.058+0000 7f418f694640 1 -- start start 2023-11-28T17:47:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.059+0000 7f418f694640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4174005180 0x7f4188156720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.059+0000 7f418f694640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41880ca630 0x7f4188158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.059+0000 7f418f694640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4188157180 0x7f4188154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.059+0000 7f418f694640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4188155ed0 con 0x7f41880ca630 2023-11-28T17:47:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.059+0000 7f418f694640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41880cac40 con 0x7f4174005180 2023-11-28T17:47:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.059+0000 7f418f694640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41880cb0d0 con 0x7f4188157180 2023-11-28T17:47:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.059+0000 7f418cc2f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41880ca630 0x7f4188158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.059+0000 7f418cc2f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41880ca630 0x7f4188158d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60520/0 (socket says 172.21.15.98:60520) 2023-11-28T17:47:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.059+0000 7f418cc2f640 1 -- 172.21.15.98:0/654534349 learned_addr learned my addr 172.21.15.98:0/654534349 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.060+0000 7f418d430640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4174005180 0x7f4188156720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.060+0000 7f418dc31640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4188157180 0x7f4188154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.060+0000 7f418cc2f640 1 -- 172.21.15.98:0/654534349 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4188157180 msgr2=0x7f4188154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.060+0000 7f418cc2f640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4188157180 0x7f4188154570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.060+0000 7f418cc2f640 1 -- 172.21.15.98:0/654534349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4174005180 msgr2=0x7f4188156720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.060+0000 7f418cc2f640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4174005180 0x7f4188156720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.060+0000 7f418cc2f640 1 -- 172.21.15.98:0/654534349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41881efdb0 con 0x7f41880ca630 2023-11-28T17:47:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.060+0000 7f418cc2f640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41880ca630 0x7f4188158d40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f417c00c530 tx=0x7f417c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.060+0000 7f4179ffb640 1 -- 172.21.15.98:0/654534349 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c0052d0 con 0x7f41880ca630 2023-11-28T17:47:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.060+0000 7f4179ffb640 1 -- 172.21.15.98:0/654534349 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c005430 con 0x7f41880ca630 2023-11-28T17:47:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.060+0000 7f4179ffb640 1 -- 172.21.15.98:0/654534349 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c012870 con 0x7f41880ca630 2023-11-28T17:47:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.060+0000 7f418f694640 1 -- 172.21.15.98:0/654534349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41881563c0 con 0x7f41880ca630 2023-11-28T17:47:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.060+0000 7f418f694640 1 -- 172.21.15.98:0/654534349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41881560e0 con 0x7f41880ca630 2023-11-28T17:47:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.061+0000 7f418f694640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4188154ab0 0x7f4188148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.061+0000 7f418f694640 1 -- 172.21.15.98:0/654534349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4188074460 con 0x7f4188154ab0 2023-11-28T17:47:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.062+0000 7f418d430640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4188154ab0 0x7f4188148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.062+0000 7f418d430640 1 -- 172.21.15.98:0/654534349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f418806e580 con 0x7f4188154ab0 2023-11-28T17:47:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.062+0000 7f418d430640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4188154ab0 0x7f4188148db0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4184011500 tx=0x7f41840119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.062+0000 7f4179ffb640 1 -- 172.21.15.98:0/654534349 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f417c0129d0 con 0x7f41880ca630 2023-11-28T17:47:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.063+0000 7f4179ffb640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f416c06b2a0 0x7f416c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.063+0000 7f4179ffb640 1 -- 172.21.15.98:0/654534349 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f417c089800 con 0x7f41880ca630 2023-11-28T17:47:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.064+0000 7f418dc31640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f416c06b2a0 0x7f416c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.064+0000 7f4179ffb640 1 -- 172.21.15.98:0/654534349 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4184015850 con 0x7f4188154ab0 2023-11-28T17:47:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.064+0000 7f4179ffb640 1 -- 172.21.15.98:0/654534349 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f41840159b0 con 0x7f4188154ab0 2023-11-28T17:47:55.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.064+0000 7f4179ffb640 1 -- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4188154ab0 msgr2=0x7f4188148db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.064+0000 7f4179ffb640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4188154ab0 0x7f4188148db0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4184011500 tx=0x7f41840119c0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.064+0000 7f4179ffb640 1 -- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4188154ab0 msgr2=0x7f4188148db0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.064+0000 7f4179ffb640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4188154ab0 0x7f4188148db0 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4184011500 tx=0x7f41840119c0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.064+0000 7f418dc31640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f416c06b2a0 0x7f416c06d760 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f417400c080 tx=0x7f417c001120 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.071+0000 7f418f694640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41881492f0 0x7f41881f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.071+0000 7f418f694640 1 -- 172.21.15.98:0/654534349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4188156270 con 0x7f41881492f0 2023-11-28T17:47:55.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.071+0000 7f418d430640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41881492f0 0x7f41881f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.071+0000 7f418d430640 1 -- 172.21.15.98:0/654534349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4188074f60 con 0x7f41881492f0 2023-11-28T17:47:55.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.071+0000 7f418d430640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41881492f0 0x7f41881f2810 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4184011500 tx=0x7f41840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.072+0000 7f4179ffb640 1 -- 172.21.15.98:0/654534349 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4184004220 con 0x7f41881492f0 2023-11-28T17:47:55.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.072+0000 7f4179ffb640 1 -- 172.21.15.98:0/654534349 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4978 (secure 0 0 0) 0x7f4184004380 con 0x7f41881492f0 2023-11-28T17:47:55.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.072+0000 7f4179ffb640 1 -- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41881492f0 msgr2=0x7f41881f2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.072+0000 7f4179ffb640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41881492f0 0x7f41881f2810 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4184011500 tx=0x7f41840046c0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.072+0000 7f4179ffb640 1 -- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41881492f0 msgr2=0x7f41881f2810 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:55.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.072+0000 7f4179ffb640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41881492f0 0x7f41881f2810 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4184011500 tx=0x7f41840046c0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.074 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:47:55.075 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:47:55.075 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:47:55.075 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:47:55.075 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 70, 2023-11-28T17:47:55.075 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:47:55.075 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:47:55.075 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:47:55.075 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:47:55.075 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.075 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 15, 2023-11-28T17:47:55.075 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:55.075 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:47:55.076 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:47:55.076 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:55.076 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:55.076 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:55.076 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:55.076 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:55.076 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:55.076 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:55.076 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:55.076 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:55.076 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:55.076 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.076 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:47:55.077 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:47:55.077 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:55.077 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:55.077 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:55.077 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:55.077 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:55.077 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:55.077 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:55.077 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:55.077 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:55.077 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:55.077 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.078 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:47:55.078 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:47:55.078 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:47:55.078 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:47:55.078 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:47:55.078 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:47:55.078 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:55.078 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:55.078 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:47:55.078 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:47:55.078 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:47:55.078 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:47:55.079 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:47:55.079 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:47:55.079 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:47:55.079 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:55.079 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:47:55.079 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:55.079 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:55.079 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:55.079 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:55.079 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:55.079 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:55.079 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:55.079 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:55.080 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:55.080 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:55.080 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.080 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:47:55.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.080 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:47:55.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.080 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:47:55.080 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:47:55.080 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:55.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:55.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:55.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:47:55.081 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.081 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:55.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:47:55.081 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.081 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.081 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:47:55.081 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:47:55.082 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:55.082 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:55.082 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:55.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.082 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:47:55.082 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:47:55.082 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:55.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:55.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:55.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:47:55.082 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:55.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:47:55.083 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.083 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.083 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:47:55.084 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:47:55.084 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:55.084 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:55.084 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:55.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.084 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:47:55.084 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:47:55.084 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:55.084 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:55.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.084 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:55.084 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:47:55.085 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:55.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:47:55.085 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.085 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.085 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:47:55.085 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:47:55.085 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:55.086 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:55.086 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:55.086 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.086 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.086 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:47:55.086 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:47:55.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.086 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:55.086 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:55.086 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:47:55.086 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.086 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.087 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:47:55.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.087 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:55.087 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:55.087 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:47:55.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.087 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.087 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:47:55.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.087 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:55.087 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:55.087 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 6 2023-11-28T17:47:55.088 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.088 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.088 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:47:55.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.088 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:55.088 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:55.088 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:47:55.088 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.088 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.088 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:47:55.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.089 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:55.089 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:55.089 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:47:55.089 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.089 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.089 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:47:55.089 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:47:55.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.072+0000 7f418f694640 1 -- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f416c06b2a0 msgr2=0x7f416c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.072+0000 7f418f694640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f416c06b2a0 0x7f416c06d760 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f417400c080 tx=0x7f417c001120 comp rx=0 tx=0).stop 2023-11-28T17:47:55.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.072+0000 7f418f694640 1 -- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41880ca630 msgr2=0x7f4188158d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.072+0000 7f418f694640 1 --2- 172.21.15.98:0/654534349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41880ca630 0x7f4188158d40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f417c00c530 tx=0x7f417c00c9f0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.072+0000 7f418dc31640 1 -- 172.21.15.98:0/654534349 reap_dead start 2023-11-28T17:47:55.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.072+0000 7f418dc31640 1 -- 172.21.15.98:0/654534349 reap_dead start 2023-11-28T17:47:55.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.072+0000 7f418f694640 1 -- 172.21.15.98:0/654534349 shutdown_connections 2023-11-28T17:47:55.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.072+0000 7f418f694640 1 -- 172.21.15.98:0/654534349 >> 172.21.15.98:0/654534349 conn(0x7f418806ef10 msgr2=0x7f4188152560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:55.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.073+0000 7f418f694640 1 -- 172.21.15.98:0/654534349 shutdown_connections 2023-11-28T17:47:55.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.073+0000 7f418f694640 1 -- 172.21.15.98:0/654534349 wait complete. 2023-11-28T17:47:55.091 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:47:55.092 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:47:55.092 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:47:55.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.293+0000 7fb2a8eec640 1 Processor -- start 2023-11-28T17:47:55.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.294+0000 7fb2a8eec640 1 -- start start 2023-11-28T17:47:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.295+0000 7fb2a8eec640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb2a406e490 0x7fb2a40c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.295+0000 7fb2a8eec640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a40c8150 con 0x7fb2a40c5a60 2023-11-28T17:47:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.295+0000 7fb2a8eec640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a40c8850 con 0x7fb2a4151be0 2023-11-28T17:47:55.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.295+0000 7fb2a8eec640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a40c5f50 con 0x7fb2a406e490 2023-11-28T17:47:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.295+0000 7fb2a2d9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb2a406e490 0x7fb2a40c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.296+0000 7fb29bfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb2a40c5a60 0x7fb2a406dd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43582/0 (socket says 172.21.15.98:43582) 2023-11-28T17:47:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.296+0000 7fb29bfff640 1 -- 172.21.15.98:0/4074381308 learned_addr learned my addr 172.21.15.98:0/4074381308 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:47:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.296+0000 7fb2a2d9d640 1 -- 172.21.15.98:0/4074381308 >> v1:172.21.15.113:6789/0 conn(0x7fb2a4151be0 legacy=0x7fb2a40c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.296+0000 7fb2a2d9d640 1 -- 172.21.15.98:0/4074381308 >> v1:172.21.15.98:6789/0 conn(0x7fb2a40c5a60 legacy=0x7fb2a406dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.296+0000 7fb2a2d9d640 1 -- 172.21.15.98:0/4074381308 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a40c7b90 con 0x7fb2a406e490 2023-11-28T17:47:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.297+0000 7fb2a2d9d640 1 --2- 172.21.15.98:0/4074381308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a406e490 0x7fb2a40c75a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb29c009d90 tx=0x7fb29c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7974f6e5762893ab server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.297+0000 7fb2a1d9b640 1 -- 172.21.15.98:0/4074381308 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29c02ccf0 con 0x7fb2a406e490 2023-11-28T17:47:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.297+0000 7fb2a1d9b640 1 -- 172.21.15.98:0/4074381308 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29c02ce50 con 0x7fb2a406e490 2023-11-28T17:47:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.297+0000 7fb2a1d9b640 1 -- 172.21.15.98:0/4074381308 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29c034660 con 0x7fb2a406e490 2023-11-28T17:47:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.298+0000 7fb2a8eec640 1 -- 172.21.15.98:0/4074381308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a406e490 msgr2=0x7fb2a40c75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.298+0000 7fb2a8eec640 1 --2- 172.21.15.98:0/4074381308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a406e490 0x7fb2a40c75a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb29c009d90 tx=0x7fb29c02bf40 comp rx=0 tx=0).stop 2023-11-28T17:47:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.298+0000 7fb2a8eec640 1 -- 172.21.15.98:0/4074381308 shutdown_connections 2023-11-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.298+0000 7fb2a8eec640 1 --2- 172.21.15.98:0/4074381308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a406e490 0x7fb2a40c75a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.298+0000 7fb2a8eec640 1 -- 172.21.15.98:0/4074381308 >> 172.21.15.98:0/4074381308 conn(0x7fb2a406c0f0 msgr2=0x7fb2a406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.298+0000 7fb2a8eec640 1 -- 172.21.15.98:0/4074381308 shutdown_connections 2023-11-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.298+0000 7fb2a8eec640 1 -- 172.21.15.98:0/4074381308 wait complete. 2023-11-28T17:47:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.300+0000 7fb2a8eec640 1 Processor -- start 2023-11-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.300+0000 7fb2a8eec640 1 -- start start 2023-11-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a8eec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a406e490 0x7fb2a414ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a8eec640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2a40c5a60 0x7fb2a41547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a8eec640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a4151be0 0x7fb2a414d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a8eec640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a41ef860 con 0x7fb2a406e490 2023-11-28T17:47:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a8eec640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a41efb40 con 0x7fb2a4151be0 2023-11-28T17:47:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a8eec640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a41efe20 con 0x7fb2a40c5a60 2023-11-28T17:47:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a406e490 0x7fb2a414ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a406e490 0x7fb2a414ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60548/0 (socket says 172.21.15.98:60548) 2023-11-28T17:47:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a259c640 1 -- 172.21.15.98:0/4148763037 learned_addr learned my addr 172.21.15.98:0/4148763037 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb29bfff640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2a40c5a60 0x7fb2a41547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a2d9d640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a4151be0 0x7fb2a414d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a259c640 1 -- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2a40c5a60 msgr2=0x7fb2a41547d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a259c640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb2a40c5a60 0x7fb2a41547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a259c640 1 -- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a4151be0 msgr2=0x7fb2a414d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a259c640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a4151be0 0x7fb2a414d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a259c640 1 -- 172.21.15.98:0/4148763037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c009a70 con 0x7fb2a406e490 2023-11-28T17:47:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.301+0000 7fb2a2d9d640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a4151be0 0x7fb2a414d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:47:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.302+0000 7fb2a259c640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a406e490 0x7fb2a414ffa0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb29400c450 tx=0x7fb29400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.302+0000 7fb29affd640 1 -- 172.21.15.98:0/4148763037 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2940052d0 con 0x7fb2a406e490 2023-11-28T17:47:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.302+0000 7fb29affd640 1 -- 172.21.15.98:0/4148763037 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb294005430 con 0x7fb2a406e490 2023-11-28T17:47:55.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.302+0000 7fb2a8eec640 1 -- 172.21.15.98:0/4148763037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a41f0100 con 0x7fb2a406e490 2023-11-28T17:47:55.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.302+0000 7fb29affd640 1 -- 172.21.15.98:0/4148763037 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb294012870 con 0x7fb2a406e490 2023-11-28T17:47:55.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.302+0000 7fb2a8eec640 1 -- 172.21.15.98:0/4148763037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a4158170 con 0x7fb2a406e490 2023-11-28T17:47:55.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.303+0000 7fb2a8eec640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a414de50 0x7fb2a414c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.303+0000 7fb2a8eec640 1 -- 172.21.15.98:0/4148763037 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a40744d0 con 0x7fb2a414de50 2023-11-28T17:47:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.304+0000 7fb29bfff640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a414de50 0x7fb2a414c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.304+0000 7fb29affd640 1 -- 172.21.15.98:0/4148763037 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb2940129d0 con 0x7fb2a406e490 2023-11-28T17:47:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.304+0000 7fb29bfff640 1 -- 172.21.15.98:0/4148763037 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2a41efcb0 con 0x7fb2a414de50 2023-11-28T17:47:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.305+0000 7fb29bfff640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a414de50 0x7fb2a414c890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4155390 tx=0x7fb28c00d8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.305+0000 7fb29affd640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb28406b2a0 0x7fb28406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.305+0000 7fb29affd640 1 -- 172.21.15.98:0/4148763037 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb2940898b0 con 0x7fb2a406e490 2023-11-28T17:47:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.305+0000 7fb2a2d9d640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb28406b2a0 0x7fb28406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.305+0000 7fb29affd640 1 -- 172.21.15.98:0/4148763037 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c00b070 con 0x7fb2a414de50 2023-11-28T17:47:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.306+0000 7fb29affd640 1 -- 172.21.15.98:0/4148763037 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fb28c005cc0 con 0x7fb2a414de50 2023-11-28T17:47:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.306+0000 7fb29affd640 1 -- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a414de50 msgr2=0x7fb2a414c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.306+0000 7fb29affd640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a414de50 0x7fb2a414c890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4155390 tx=0x7fb28c00d8a0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.306+0000 7fb29affd640 1 -- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a414de50 msgr2=0x7fb2a414c890 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:55.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.306+0000 7fb29affd640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a414de50 0x7fb2a414c890 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4155390 tx=0x7fb28c00d8a0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.306+0000 7fb2a2d9d640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb28406b2a0 0x7fb28406d760 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb29c036000 tx=0x7fb29c030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.313+0000 7fb2a8eec640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a414cdd0 0x7fb2a414d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.313+0000 7fb2a8eec640 1 -- 172.21.15.98:0/4148763037 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a41eff90 con 0x7fb2a414cdd0 2023-11-28T17:47:55.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.313+0000 7fb29bfff640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a414cdd0 0x7fb2a414d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.314+0000 7fb29bfff640 1 -- 172.21.15.98:0/4148763037 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb2a40c6240 con 0x7fb2a414cdd0 2023-11-28T17:47:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.314+0000 7fb29bfff640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a414cdd0 0x7fb2a414d1e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40c1a20 tx=0x7fb28c0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.315+0000 7fb29affd640 1 -- 172.21.15.98:0/4148763037 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c020070 con 0x7fb2a414cdd0 2023-11-28T17:47:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.315+0000 7fb29affd640 1 -- 172.21.15.98:0/4148763037 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4636 (secure 0 0 0) 0x7fb28c01be50 con 0x7fb2a414cdd0 2023-11-28T17:47:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.315+0000 7fb29affd640 1 -- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a414cdd0 msgr2=0x7fb2a414d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.315+0000 7fb29affd640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a414cdd0 0x7fb2a414d1e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40c1a20 tx=0x7fb28c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.315+0000 7fb29affd640 1 -- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a414cdd0 msgr2=0x7fb2a414d1e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.315+0000 7fb29affd640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb2a414cdd0 0x7fb2a414d1e0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40c1a20 tx=0x7fb28c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.320 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:47:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:47:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:47:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:47:55.320 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 70, 2023-11-28T17:47:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:47:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:47:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:47:55.321 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:47:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 15, 2023-11-28T17:47:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:47:55.322 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:47:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:47:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:47:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:55.325 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.326 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:47:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:47:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:47:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:47:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:47:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:47:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:55.327 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:47:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:47:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:47:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:47:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:47:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:47:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:47:55.328 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:47:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:55.329 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.330 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:47:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:47:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:47:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:47:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:55.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:47:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:47:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:47:55.333 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:47:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:47:55.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:47:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:47:55.335 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:47:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.336 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:47:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:47:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:55.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:47:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:47:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:55.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:47:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:47:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:47:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:47:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:47:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:47:55.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:47:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.341 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:47:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:47:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:47:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:47:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.344 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:47:55.344 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:47:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.316+0000 7fb2a8eec640 1 -- 172.21.15.98:0/4148763037 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb28406b2a0 msgr2=0x7fb28406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.316+0000 7fb2a8eec640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb28406b2a0 0x7fb28406d760 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb29c036000 tx=0x7fb29c030000 comp rx=0 tx=0).stop 2023-11-28T17:47:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.316+0000 7fb2a8eec640 1 -- 172.21.15.98:0/4148763037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a406e490 msgr2=0x7fb2a414ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.316+0000 7fb2a8eec640 1 --2- 172.21.15.98:0/4148763037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a406e490 0x7fb2a414ffa0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb29400c450 tx=0x7fb29400c910 comp rx=0 tx=0).stop 2023-11-28T17:47:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.316+0000 7fb2a2d9d640 1 -- 172.21.15.98:0/4148763037 reap_dead start 2023-11-28T17:47:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.316+0000 7fb2a2d9d640 1 -- 172.21.15.98:0/4148763037 reap_dead start 2023-11-28T17:47:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.316+0000 7fb2a8eec640 1 -- 172.21.15.98:0/4148763037 shutdown_connections 2023-11-28T17:47:55.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.316+0000 7fb2a8eec640 1 -- 172.21.15.98:0/4148763037 >> 172.21.15.98:0/4148763037 conn(0x7fb2a406c0f0 msgr2=0x7fb2a40c4530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.316+0000 7fb2a8eec640 1 -- 172.21.15.98:0/4148763037 shutdown_connections 2023-11-28T17:47:55.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.317+0000 7fb2a8eec640 1 -- 172.21.15.98:0/4148763037 wait complete. 2023-11-28T17:47:55.347 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:47:55.347 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:47:55.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:47:55.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.558+0000 7f3682990640 1 Processor -- start 2023-11-28T17:47:55.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.558+0000 7f3682990640 1 -- start start 2023-11-28T17:47:55.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.559+0000 7f3682990640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f367c06e490 0x7f367c0c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.560+0000 7f3682990640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367c0c8150 con 0x7f367c151be0 2023-11-28T17:47:55.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.560+0000 7f3682990640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367c0c8850 con 0x7f367c0c5a60 2023-11-28T17:47:55.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.560+0000 7f3682990640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367c0c5f50 con 0x7f367c06e490 2023-11-28T17:47:55.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.560+0000 7f3680f2d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f367c06e490 0x7f367c0c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.560+0000 7f367bfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f367c151be0 0x7f367c0c4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43598/0 (socket says 172.21.15.98:43598) 2023-11-28T17:47:55.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.560+0000 7f367bfff640 1 -- 172.21.15.98:0/3919296751 learned_addr learned my addr 172.21.15.98:0/3919296751 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:47:55.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.560+0000 7f3680f2d640 1 -- 172.21.15.98:0/3919296751 >> v1:172.21.15.113:6789/0 conn(0x7f367c0c5a60 legacy=0x7f367c06dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:55.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.560+0000 7f3680f2d640 1 -- 172.21.15.98:0/3919296751 >> v1:172.21.15.98:6789/0 conn(0x7f367c151be0 legacy=0x7f367c0c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.560+0000 7f3680f2d640 1 -- 172.21.15.98:0/3919296751 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367c0c7b90 con 0x7f367c06e490 2023-11-28T17:47:55.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.562+0000 7f3680f2d640 1 --2- 172.21.15.98:0/3919296751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f367c06e490 0x7f367c0c75a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3674009d90 tx=0x7f367402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=481bdb37993c151b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.562+0000 7f367b7fe640 1 -- 172.21.15.98:0/3919296751 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367402ccf0 con 0x7f367c06e490 2023-11-28T17:47:55.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.562+0000 7f367b7fe640 1 -- 172.21.15.98:0/3919296751 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367402ce50 con 0x7f367c06e490 2023-11-28T17:47:55.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.562+0000 7f367b7fe640 1 -- 172.21.15.98:0/3919296751 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3674034660 con 0x7f367c06e490 2023-11-28T17:47:55.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.562+0000 7f3682990640 1 -- 172.21.15.98:0/3919296751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f367c06e490 msgr2=0x7f367c0c75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.562+0000 7f3682990640 1 --2- 172.21.15.98:0/3919296751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f367c06e490 0x7f367c0c75a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3674009d90 tx=0x7f367402bf40 comp rx=0 tx=0).stop 2023-11-28T17:47:55.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.563+0000 7f3682990640 1 -- 172.21.15.98:0/3919296751 shutdown_connections 2023-11-28T17:47:55.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.563+0000 7f3682990640 1 --2- 172.21.15.98:0/3919296751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f367c06e490 0x7f367c0c75a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.563+0000 7f3682990640 1 -- 172.21.15.98:0/3919296751 >> 172.21.15.98:0/3919296751 conn(0x7f367c06c0f0 msgr2=0x7f367c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:55.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.563+0000 7f3682990640 1 -- 172.21.15.98:0/3919296751 shutdown_connections 2023-11-28T17:47:55.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.563+0000 7f3682990640 1 -- 172.21.15.98:0/3919296751 wait complete. 2023-11-28T17:47:55.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.564+0000 7f3682990640 1 Processor -- start 2023-11-28T17:47:55.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.565+0000 7f3682990640 1 -- start start 2023-11-28T17:47:55.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.565+0000 7f3682990640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c06e490 0x7f367c0c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.565+0000 7f3682990640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f367c0c5a60 0x7f367c148af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.565+0000 7f3682990640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f367c151be0 0x7f367c14bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.565+0000 7f3682990640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c149030 con 0x7f367c0c5a60 2023-11-28T17:47:55.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.565+0000 7f3682990640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c0ca9c0 con 0x7f367c151be0 2023-11-28T17:47:55.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.565+0000 7f3682990640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367c0c9880 con 0x7f367c06e490 2023-11-28T17:47:55.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.566+0000 7f3673fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f367c0c5a60 0x7f367c148af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.566+0000 7f3673fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f367c0c5a60 0x7f367c148af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60566/0 (socket says 172.21.15.98:60566) 2023-11-28T17:47:55.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.566+0000 7f3673fff640 1 -- 172.21.15.98:0/2636286929 learned_addr learned my addr 172.21.15.98:0/2636286929 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:55.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.566+0000 7f367bfff640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c06e490 0x7f367c0c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.566+0000 7f3680f2d640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f367c151be0 0x7f367c14bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.566+0000 7f3673fff640 1 -- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c06e490 msgr2=0x7f367c0c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.566+0000 7f3673fff640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c06e490 0x7f367c0c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.566+0000 7f3673fff640 1 -- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f367c151be0 msgr2=0x7f367c14bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.566+0000 7f3673fff640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f367c151be0 0x7f367c14bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.566+0000 7f3673fff640 1 -- 172.21.15.98:0/2636286929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3674009a70 con 0x7f367c0c5a60 2023-11-28T17:47:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.566+0000 7f3680f2d640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f367c151be0 0x7f367c14bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:47:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.567+0000 7f3673fff640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f367c0c5a60 0x7f367c148af0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f366400c480 tx=0x7f366400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.567+0000 7f36797fa640 1 -- 172.21.15.98:0/2636286929 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36640052d0 con 0x7f367c0c5a60 2023-11-28T17:47:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.567+0000 7f36797fa640 1 -- 172.21.15.98:0/2636286929 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3664005430 con 0x7f367c0c5a60 2023-11-28T17:47:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.567+0000 7f3682990640 1 -- 172.21.15.98:0/2636286929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367c1ee070 con 0x7f367c0c5a60 2023-11-28T17:47:55.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.567+0000 7f36797fa640 1 -- 172.21.15.98:0/2636286929 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664012870 con 0x7f367c0c5a60 2023-11-28T17:47:55.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.567+0000 7f3682990640 1 -- 172.21.15.98:0/2636286929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367c0c8f50 con 0x7f367c0c5a60 2023-11-28T17:47:55.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.568+0000 7f3682990640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c0ca520 0x7f367c0c9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.568+0000 7f3682990640 1 -- 172.21.15.98:0/2636286929 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367c0744d0 con 0x7f367c0ca520 2023-11-28T17:47:55.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.569+0000 7f367bfff640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c0ca520 0x7f367c0c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.569+0000 7f36797fa640 1 -- 172.21.15.98:0/2636286929 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f36640129d0 con 0x7f367c0c5a60 2023-11-28T17:47:55.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.569+0000 7f367bfff640 1 -- 172.21.15.98:0/2636286929 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f367c0c84e0 con 0x7f367c0ca520 2023-11-28T17:47:55.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.570+0000 7f36797fa640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f365c06b2a0 0x7f365c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.570+0000 7f36797fa640 1 -- 172.21.15.98:0/2636286929 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3664089760 con 0x7f367c0c5a60 2023-11-28T17:47:55.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.570+0000 7f367bfff640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c0ca520 0x7f367c0c9110 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f366c000f60 tx=0x7f366c00e940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.570+0000 7f3680f2d640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f365c06b2a0 0x7f365c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.570+0000 7f36797fa640 1 -- 172.21.15.98:0/2636286929 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c005ac0 con 0x7f367c0ca520 2023-11-28T17:47:55.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.571+0000 7f3680f2d640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f365c06b2a0 0x7f365c06d760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3674036000 tx=0x7f3674030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.571+0000 7f36797fa640 1 -- 172.21.15.98:0/2636286929 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f366c005c20 con 0x7f367c0ca520 2023-11-28T17:47:55.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.571+0000 7f36797fa640 1 -- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c0ca520 msgr2=0x7f367c0c9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.571+0000 7f36797fa640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c0ca520 0x7f367c0c9110 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f366c000f60 tx=0x7f366c00e940 comp rx=0 tx=0).stop 2023-11-28T17:47:55.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.571+0000 7f36797fa640 1 -- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c0ca520 msgr2=0x7f367c0c9110 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:55.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.571+0000 7f36797fa640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c0ca520 0x7f367c0c9110 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f366c000f60 tx=0x7f366c00e940 comp rx=0 tx=0).stop 2023-11-28T17:47:55.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.578+0000 7f3682990640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c0cac50 0x7f367c1f0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.578+0000 7f3682990640 1 -- 172.21.15.98:0/2636286929 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367c0c99f0 con 0x7f367c0cac50 2023-11-28T17:47:55.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.579+0000 7f367bfff640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c0cac50 0x7f367c1f0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.580+0000 7f367bfff640 1 -- 172.21.15.98:0/2636286929 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f367c0c9650 con 0x7f367c0cac50 2023-11-28T17:47:55.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.580+0000 7f367bfff640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c0cac50 0x7f367c1f0a50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f366c000f60 tx=0x7f366c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.581+0000 7f36797fa640 1 -- 172.21.15.98:0/2636286929 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c007730 con 0x7f367c0cac50 2023-11-28T17:47:55.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.581+0000 7f36797fa640 1 -- 172.21.15.98:0/2636286929 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4636 (secure 0 0 0) 0x7f366c013420 con 0x7f367c0cac50 2023-11-28T17:47:55.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.581+0000 7f36797fa640 1 -- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c0cac50 msgr2=0x7f367c1f0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.581+0000 7f36797fa640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c0cac50 0x7f367c1f0a50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f366c000f60 tx=0x7f366c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.581+0000 7f36797fa640 1 -- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c0cac50 msgr2=0x7f367c1f0a50 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:47:55.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.581+0000 7f36797fa640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f367c0cac50 0x7f367c1f0a50 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f366c000f60 tx=0x7f366c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.585 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:47:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:47:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:47:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:47:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 70, 2023-11-28T17:47:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:47:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:47:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:47:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:47:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 15, 2023-11-28T17:47:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:47:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:47:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:47:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:47:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:47:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:47:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:47:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:47:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:47:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:47:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:47:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:47:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:47:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:47:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:47:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:47:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:47:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:47:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:47:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:47:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:47:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:47:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:47:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:47:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:47:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:47:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:47:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:47:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:47:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:47:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:47:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:47:55.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:47:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:47:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:47:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:47:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:47:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:47:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:55.593 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:47:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:47:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:47:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:47:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:47:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:47:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:55.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:47:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:47:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:47:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:47:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:47:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:47:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:47:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:47:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:47:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:47:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:47:55.597 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:47:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:47:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:47:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:47:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:47:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:55.598 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:47:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:47:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:47:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:47:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:47:55.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:47:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:47:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:47:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:47:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:47:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:47:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:47:55.600 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:47:55.600 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:47:55.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.582+0000 7f3682990640 1 -- 172.21.15.98:0/2636286929 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f365c06b2a0 msgr2=0x7f365c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.582+0000 7f3682990640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f365c06b2a0 0x7f365c06d760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3674036000 tx=0x7f3674030000 comp rx=0 tx=0).stop 2023-11-28T17:47:55.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.582+0000 7f3682990640 1 -- 172.21.15.98:0/2636286929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f367c0c5a60 msgr2=0x7f367c148af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.582+0000 7f3682990640 1 --2- 172.21.15.98:0/2636286929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f367c0c5a60 0x7f367c148af0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f366400c480 tx=0x7f366400c940 comp rx=0 tx=0).stop 2023-11-28T17:47:55.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.582+0000 7f3680f2d640 1 -- 172.21.15.98:0/2636286929 reap_dead start 2023-11-28T17:47:55.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.582+0000 7f3680f2d640 1 -- 172.21.15.98:0/2636286929 reap_dead start 2023-11-28T17:47:55.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.582+0000 7f3682990640 1 -- 172.21.15.98:0/2636286929 shutdown_connections 2023-11-28T17:47:55.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.582+0000 7f3682990640 1 -- 172.21.15.98:0/2636286929 >> 172.21.15.98:0/2636286929 conn(0x7f367c06c0f0 msgr2=0x7f367c156ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:55.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.582+0000 7f3682990640 1 -- 172.21.15.98:0/2636286929 shutdown_connections 2023-11-28T17:47:55.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.582+0000 7f3682990640 1 -- 172.21.15.98:0/2636286929 wait complete. 2023-11-28T17:47:55.603 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:47:55.603 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2023-11-28T17:47:55.603 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-11-28T17:47:55.603 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2023-11-28T17:47:55.603 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-11-28T17:47:55.603 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-28T17:47:55.603 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:47:55.629 INFO:tasks.ceph.mon.c:Stopped 2023-11-28T17:47:55.629 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-11-28T17:47:55.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:47:55.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.841+0000 7f2ef2116640 1 Processor -- start 2023-11-28T17:47:55.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.841+0000 7f2ef2116640 1 -- start start 2023-11-28T17:47:55.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.843+0000 7f2ef2116640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2eec06afe0 0x7f2eec0c0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.843+0000 7f2ef2116640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2eec0c0d70 0x7f2eec06dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.843+0000 7f2ef2116640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eec154440 con 0x7f2eec154060 2023-11-28T17:47:55.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.843+0000 7f2ef2116640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eec0c11e0 con 0x7f2eec0c0d70 2023-11-28T17:47:55.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.843+0000 7f2ef2116640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec159c60 con 0x7f2eec06afe0 2023-11-28T17:47:55.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.843+0000 7f2ef0913640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2eec06afe0 msgr2=0x7f2eec0c0830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:55.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.843+0000 7f2ef0913640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2eec06afe0 0x7f2eec0c0830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:55.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.843+0000 7f2ef1915640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2eec0c0d70 0x7f2eec06dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.843+0000 7f2ef1915640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2eec0c0d70 0x7f2eec06dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60582/0 (socket says 172.21.15.98:60582) 2023-11-28T17:47:55.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.843+0000 7f2ef1915640 1 -- 172.21.15.98:0/164509594 learned_addr learned my addr 172.21.15.98:0/164509594 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:55.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.843+0000 7f2ef1915640 1 -- 172.21.15.98:0/164509594 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2eec06afe0 msgr2=0x7f2eec0c0830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:55.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.843+0000 7f2ef1915640 1 --2- 172.21.15.98:0/164509594 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2eec06afe0 0x7f2eec0c0830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.843+0000 7f2ef1915640 1 -- 172.21.15.98:0/164509594 >> v1:172.21.15.98:6789/0 conn(0x7f2eec154060 legacy=0x7f2eec0bff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.843+0000 7f2ef1915640 1 -- 172.21.15.98:0/164509594 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eec159e80 con 0x7f2eec0c0d70 2023-11-28T17:47:55.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.845+0000 7f2ef1915640 1 --2- 172.21.15.98:0/164509594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eec0c0d70 0x7f2eec06dd30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4009e80 tx=0x7f2ee402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3efcebb8a1bc9263 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.845+0000 7f2ee3fff640 1 -- 172.21.15.98:0/164509594 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee402cb00 con 0x7f2eec0c0d70 2023-11-28T17:47:55.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.845+0000 7f2ee3fff640 1 -- 172.21.15.98:0/164509594 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee402cc60 con 0x7f2eec0c0d70 2023-11-28T17:47:55.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.845+0000 7f2ee3fff640 1 -- 172.21.15.98:0/164509594 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee40343f0 con 0x7f2eec0c0d70 2023-11-28T17:47:55.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.845+0000 7f2ef2116640 1 -- 172.21.15.98:0/164509594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eec0c0d70 msgr2=0x7f2eec06dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.845+0000 7f2ef2116640 1 --2- 172.21.15.98:0/164509594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eec0c0d70 0x7f2eec06dd30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4009e80 tx=0x7f2ee402bf60 comp rx=0 tx=0).stop 2023-11-28T17:47:55.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.846+0000 7f2ef2116640 1 -- 172.21.15.98:0/164509594 shutdown_connections 2023-11-28T17:47:55.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.846+0000 7f2ef2116640 1 --2- 172.21.15.98:0/164509594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eec0c0d70 0x7f2eec06dd30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.846+0000 7f2ef2116640 1 --2- 172.21.15.98:0/164509594 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2eec06afe0 0x7f2eec0c0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.846+0000 7f2ef2116640 1 -- 172.21.15.98:0/164509594 >> 172.21.15.98:0/164509594 conn(0x7f2eec066230 msgr2=0x7f2eec06a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:55.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.846+0000 7f2ef2116640 1 -- 172.21.15.98:0/164509594 shutdown_connections 2023-11-28T17:47:55.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.846+0000 7f2ef2116640 1 -- 172.21.15.98:0/164509594 wait complete. 2023-11-28T17:47:55.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.847+0000 7f2ef2116640 1 Processor -- start 2023-11-28T17:47:55.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.847+0000 7f2ef2116640 1 -- start start 2023-11-28T17:47:55.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.848+0000 7f2ef2116640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2eec06afe0 0x7f2eec072220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.848+0000 7f2ef2116640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eec0c0d70 0x7f2eec0ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.848+0000 7f2ef2116640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec154060 0x7f2eec0c3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.848+0000 7f2ef2116640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec154e90 con 0x7f2eec0c0d70 2023-11-28T17:47:55.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.848+0000 7f2ef2116640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec155490 con 0x7f2eec154060 2023-11-28T17:47:55.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.848+0000 7f2ef2116640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec0cac30 con 0x7f2eec06afe0 2023-11-28T17:47:55.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.849+0000 7f2ef1114640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2eec06afe0 msgr2=0x7f2eec072220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:55.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.849+0000 7f2ef1114640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2eec06afe0 0x7f2eec072220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:55.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.849+0000 7f2ef0913640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eec0c0d70 0x7f2eec0ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.849+0000 7f2ef0913640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eec0c0d70 0x7f2eec0ca190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60592/0 (socket says 172.21.15.98:60592) 2023-11-28T17:47:55.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.849+0000 7f2ef0913640 1 -- 172.21.15.98:0/49260662 learned_addr learned my addr 172.21.15.98:0/49260662 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:55.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.849+0000 7f2ef1915640 1 --2- 172.21.15.98:0/49260662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec154060 0x7f2eec0c3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.849+0000 7f2ef0913640 1 -- 172.21.15.98:0/49260662 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2eec06afe0 msgr2=0x7f2eec072220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:47:55.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.849+0000 7f2ef0913640 1 --2- 172.21.15.98:0/49260662 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2eec06afe0 0x7f2eec072220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.849+0000 7f2ef0913640 1 -- 172.21.15.98:0/49260662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec154060 msgr2=0x7f2eec0c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:55.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.849+0000 7f2ef0913640 1 --2- 172.21.15.98:0/49260662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec154060 0x7f2eec0c3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:55.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.849+0000 7f2ef0913640 1 -- 172.21.15.98:0/49260662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee4009b30 con 0x7f2eec0c0d70 2023-11-28T17:47:55.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.849+0000 7f2ef1915640 1 --2- 172.21.15.98:0/49260662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec154060 0x7f2eec0c3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:47:55.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.849+0000 7f2ef0913640 1 --2- 172.21.15.98:0/49260662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eec0c0d70 0x7f2eec0ca190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2edc00b550 tx=0x7f2edc00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.850+0000 7f2ee1ffb640 1 -- 172.21.15.98:0/49260662 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc003dc0 con 0x7f2eec0c0d70 2023-11-28T17:47:55.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.850+0000 7f2ee1ffb640 1 -- 172.21.15.98:0/49260662 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2edc005070 con 0x7f2eec0c0d70 2023-11-28T17:47:55.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.850+0000 7f2ee1ffb640 1 -- 172.21.15.98:0/49260662 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc005420 con 0x7f2eec0c0d70 2023-11-28T17:47:55.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.850+0000 7f2ef2116640 1 -- 172.21.15.98:0/49260662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eec0c33d0 con 0x7f2eec0c0d70 2023-11-28T17:47:55.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.850+0000 7f2ef2116640 1 -- 172.21.15.98:0/49260662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eec0c2010 con 0x7f2eec0c0d70 2023-11-28T17:47:55.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.851+0000 7f2ef2116640 1 -- 172.21.15.98:0/49260662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eec0c2350 con 0x7f2eec0c0d70 2023-11-28T17:47:55.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.852+0000 7f2ee1ffb640 1 -- 172.21.15.98:0/49260662 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2edc005580 con 0x7f2eec0c0d70 2023-11-28T17:47:55.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.853+0000 7f2ee1ffb640 1 --2- 172.21.15.98:0/49260662 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2ed006b280 0x7f2ed006d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:55.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.853+0000 7f2ee1ffb640 1 -- 172.21.15.98:0/49260662 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2edc089bf0 con 0x7f2eec0c0d70 2023-11-28T17:47:55.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.853+0000 7f2ef1114640 1 --2- 172.21.15.98:0/49260662 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2ed006b280 0x7f2ed006d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:55.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.854+0000 7f2ef1114640 1 --2- 172.21.15.98:0/49260662 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2ed006b280 0x7f2ed006d740 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8001030 tx=0x7f2ee8007a30 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:55.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:55.857+0000 7f2ee1ffb640 1 -- 172.21.15.98:0/49260662 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2edc054940 con 0x7f2eec0c0d70 2023-11-28T17:47:56.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:56.121+0000 7f2ef2116640 1 -- 172.21.15.98:0/49260662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2eec074db0 con 0x7f2eec0c0d70 2023-11-28T17:47:56.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:56.122+0000 7f2ee1ffb640 1 -- 172.21.15.98:0/49260662 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f2edc013080 con 0x7f2eec0c0d70 2023-11-28T17:47:56.123 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:47:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:56.124+0000 7f2ef2116640 1 -- 172.21.15.98:0/49260662 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2ed006b280 msgr2=0x7f2ed006d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:56.124+0000 7f2ef2116640 1 --2- 172.21.15.98:0/49260662 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2ed006b280 0x7f2ed006d740 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8001030 tx=0x7f2ee8007a30 comp rx=0 tx=0).stop 2023-11-28T17:47:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:56.125+0000 7f2ef2116640 1 -- 172.21.15.98:0/49260662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eec0c0d70 msgr2=0x7f2eec0ca190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:56.125+0000 7f2ef2116640 1 --2- 172.21.15.98:0/49260662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eec0c0d70 0x7f2eec0ca190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2edc00b550 tx=0x7f2edc00ba10 comp rx=0 tx=0).stop 2023-11-28T17:47:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:56.125+0000 7f2ef2116640 1 -- 172.21.15.98:0/49260662 shutdown_connections 2023-11-28T17:47:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:56.125+0000 7f2ef2116640 1 --2- 172.21.15.98:0/49260662 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f2ed006b280 0x7f2ed006d740 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:56.125+0000 7f2ef2116640 1 --2- 172.21.15.98:0/49260662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eec154060 0x7f2eec0c3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:56.125+0000 7f2ef2116640 1 --2- 172.21.15.98:0/49260662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eec0c0d70 0x7f2eec0ca190 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:56.125+0000 7f2ef2116640 1 --2- 172.21.15.98:0/49260662 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2eec06afe0 0x7f2eec072220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:56.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:56.125+0000 7f2ef2116640 1 -- 172.21.15.98:0/49260662 >> 172.21.15.98:0/49260662 conn(0x7f2eec066230 msgr2=0x7f2eec0ad8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:56.125+0000 7f2ef2116640 1 -- 172.21.15.98:0/49260662 shutdown_connections 2023-11-28T17:47:56.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:56.125+0000 7f2ef2116640 1 -- 172.21.15.98:0/49260662 wait complete. 2023-11-28T17:47:59.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:47:59.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.362+0000 7fbeba12e640 1 Processor -- start 2023-11-28T17:47:59.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.362+0000 7fbeba12e640 1 -- start start 2023-11-28T17:47:59.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.364+0000 7fbeba12e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb4153bb0 0x7fbeb415a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:59.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.364+0000 7fbeba12e640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbeb40ca390 0x7fbeb40ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:59.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.364+0000 7fbeba12e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb415d330 con 0x7fbeb40cf280 2023-11-28T17:47:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.364+0000 7fbeba12e640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeb415d120 con 0x7fbeb40ca390 2023-11-28T17:47:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.364+0000 7fbeba12e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb415d610 con 0x7fbeb4153bb0 2023-11-28T17:47:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.364+0000 7fbeb912c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb4153bb0 msgr2=0x7fbeb415a6d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.364+0000 7fbeb912c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb4153bb0 0x7fbeb415a6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.364+0000 7fbeb992d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbeb40cf280 0x7fbeb40b3760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43628/0 (socket says 172.21.15.98:43628) 2023-11-28T17:47:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.364+0000 7fbeb992d640 1 -- 172.21.15.98:0/1858089749 learned_addr learned my addr 172.21.15.98:0/1858089749 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:47:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeb892b640 1 --2- 172.21.15.98:0/1858089749 >> v2:172.21.15.113:3300/0 conn(0x7fbeb40ca390 0x7fbeb40ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 -- 172.21.15.98:0/1858089749 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265192905 0 0) 0x7fbeb415d330 con 0x7fbeb40cf280 2023-11-28T17:47:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 -- 172.21.15.98:0/1858089749 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea0003520 con 0x7fbeb40cf280 2023-11-28T17:47:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 -- 172.21.15.98:0/1858089749 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3970444105 0 0) 0x7fbea0003520 con 0x7fbeb40cf280 2023-11-28T17:47:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 -- 172.21.15.98:0/1858089749 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb4153bb0 msgr2=0x7fbeb415a6d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 --2- 172.21.15.98:0/1858089749 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb4153bb0 0x7fbeb415a6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 -- 172.21.15.98:0/1858089749 >> v2:172.21.15.113:3300/0 conn(0x7fbeb40ca390 msgr2=0x7fbeb40ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 --2- 172.21.15.98:0/1858089749 >> v2:172.21.15.113:3300/0 conn(0x7fbeb40ca390 0x7fbeb40ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 -- 172.21.15.98:0/1858089749 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb415d8f0 con 0x7fbeb40cf280 2023-11-28T17:47:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 -- 172.21.15.98:0/1858089749 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7fbeac001670 con 0x7fbeb40cf280 2023-11-28T17:47:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 -- 172.21.15.98:0/1858089749 >> v1:172.21.15.98:6789/0 conn(0x7fbeb40cf280 legacy=0x7fbeb40b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 --2- 172.21.15.98:0/1858089749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbea0003970 0x7fbea0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 --2- 172.21.15.98:0/1858089749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbea0004360 0x7fbea0004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 --2- 172.21.15.98:0/1858089749 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb4153bb0 0x7fbea00050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 -- 172.21.15.98:0/1858089749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb415d120 con 0x7fbea0003970 2023-11-28T17:47:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 -- 172.21.15.98:0/1858089749 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb415d610 con 0x7fbea0004360 2023-11-28T17:47:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeabfff640 1 -- 172.21.15.98:0/1858089749 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea000bac0 con 0x7fbeb4153bb0 2023-11-28T17:47:59.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.365+0000 7fbeb992d640 1 -- 172.21.15.98:0/1858089749 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb4153bb0 msgr2=0x7fbea00050c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:59.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.366+0000 7fbeb992d640 1 --2- 172.21.15.98:0/1858089749 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb4153bb0 0x7fbea00050c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:59.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.366+0000 7fbeb912c640 1 --2- 172.21.15.98:0/1858089749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbea0003970 0x7fbea0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:59.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.366+0000 7fbeb892b640 1 --2- 172.21.15.98:0/1858089749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbea0004360 0x7fbea0004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:59.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.366+0000 7fbeb912c640 1 -- 172.21.15.98:0/1858089749 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb4153bb0 msgr2=0x7fbea00050c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:47:59.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.366+0000 7fbeb912c640 1 --2- 172.21.15.98:0/1858089749 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb4153bb0 0x7fbea00050c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:59.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.366+0000 7fbeb912c640 1 -- 172.21.15.98:0/1858089749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbea0004360 msgr2=0x7fbea0004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:47:59.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.366+0000 7fbeb912c640 1 --2- 172.21.15.98:0/1858089749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbea0004360 0x7fbea0004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:59.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.366+0000 7fbeb912c640 1 -- 172.21.15.98:0/1858089749 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea0003520 con 0x7fbea0003970 2023-11-28T17:47:59.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.367+0000 7fbeb912c640 1 --2- 172.21.15.98:0/1858089749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbea0003970 0x7fbea0003d90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009e30 tx=0x7fbea00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae4c4d266d8bc09b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:59.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.367+0000 7fbeabfff640 1 -- 172.21.15.98:0/1858089749 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb002cd30 con 0x7fbea0003970 2023-11-28T17:47:59.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.367+0000 7fbeabfff640 1 -- 172.21.15.98:0/1858089749 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb002ce90 con 0x7fbea0003970 2023-11-28T17:47:59.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.367+0000 7fbeabfff640 1 -- 172.21.15.98:0/1858089749 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb00356b0 con 0x7fbea0003970 2023-11-28T17:47:59.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.368+0000 7fbeba12e640 1 -- 172.21.15.98:0/1858089749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbea0003970 msgr2=0x7fbea0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:59.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.368+0000 7fbeba12e640 1 --2- 172.21.15.98:0/1858089749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbea0003970 0x7fbea0003d90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009e30 tx=0x7fbea00057f0 comp rx=0 tx=0).stop 2023-11-28T17:47:59.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.368+0000 7fbeb992d640 1 -- 172.21.15.98:0/1858089749 reap_dead start 2023-11-28T17:47:59.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.368+0000 7fbeba12e640 1 -- 172.21.15.98:0/1858089749 shutdown_connections 2023-11-28T17:47:59.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.368+0000 7fbeba12e640 1 -- 172.21.15.98:0/1858089749 >> 172.21.15.98:0/1858089749 conn(0x7fbeb4071c00 msgr2=0x7fbeb4072010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:59.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.368+0000 7fbeba12e640 1 -- 172.21.15.98:0/1858089749 shutdown_connections 2023-11-28T17:47:59.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.368+0000 7fbeba12e640 1 -- 172.21.15.98:0/1858089749 wait complete. 2023-11-28T17:47:59.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.370+0000 7fbeba12e640 1 Processor -- start 2023-11-28T17:47:59.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.370+0000 7fbeba12e640 1 -- start start 2023-11-28T17:47:59.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.371+0000 7fbeba12e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbea0003970 0x7fbeb414d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:59.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.371+0000 7fbeba12e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeb41f3990 0x7fbeb41e4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:59.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.371+0000 7fbeba12e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb40c5640 0x7fbeb41e2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:59.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.371+0000 7fbeba12e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb41f3570 con 0x7fbea0003970 2023-11-28T17:47:59.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.371+0000 7fbeba12e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb41ef6b0 con 0x7fbeb41f3990 2023-11-28T17:47:59.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.371+0000 7fbeba12e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb41f2c60 con 0x7fbeb40c5640 2023-11-28T17:47:59.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.371+0000 7fbeb992d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb40c5640 msgr2=0x7fbeb41e2f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:47:59.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.371+0000 7fbeb912c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbea0003970 0x7fbeb414d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:59.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.371+0000 7fbeb992d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb40c5640 0x7fbeb41e2f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:47:59.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.371+0000 7fbeb912c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbea0003970 0x7fbeb414d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60622/0 (socket says 172.21.15.98:60622) 2023-11-28T17:47:59.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.371+0000 7fbeb912c640 1 -- 172.21.15.98:0/1942141688 learned_addr learned my addr 172.21.15.98:0/1942141688 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:47:59.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.371+0000 7fbeb892b640 1 --2- 172.21.15.98:0/1942141688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeb41f3990 0x7fbeb41e4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:59.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.372+0000 7fbeb912c640 1 -- 172.21.15.98:0/1942141688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb40c5640 msgr2=0x7fbeb41e2f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:47:59.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.372+0000 7fbeb912c640 1 --2- 172.21.15.98:0/1942141688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb40c5640 0x7fbeb41e2f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:59.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.372+0000 7fbeb912c640 1 -- 172.21.15.98:0/1942141688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeb41f3990 msgr2=0x7fbeb41e4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:59.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.372+0000 7fbeb912c640 1 --2- 172.21.15.98:0/1942141688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeb41f3990 0x7fbeb41e4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:59.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.372+0000 7fbeb912c640 1 -- 172.21.15.98:0/1942141688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb41ef8c0 con 0x7fbea0003970 2023-11-28T17:47:59.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.372+0000 7fbeb912c640 1 --2- 172.21.15.98:0/1942141688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbea0003970 0x7fbeb414d0f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0000f80 tx=0x7fbeb0005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:59.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.372+0000 7fbea9ffb640 1 -- 172.21.15.98:0/1942141688 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb003c050 con 0x7fbea0003970 2023-11-28T17:47:59.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.372+0000 7fbea9ffb640 1 -- 172.21.15.98:0/1942141688 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb003d040 con 0x7fbea0003970 2023-11-28T17:47:59.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.372+0000 7fbea9ffb640 1 -- 172.21.15.98:0/1942141688 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb00428e0 con 0x7fbea0003970 2023-11-28T17:47:59.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.372+0000 7fbeba12e640 1 -- 172.21.15.98:0/1942141688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb41f2eb0 con 0x7fbea0003970 2023-11-28T17:47:59.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.372+0000 7fbeba12e640 1 -- 172.21.15.98:0/1942141688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb41e4780 con 0x7fbea0003970 2023-11-28T17:47:59.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.373+0000 7fbeba12e640 1 -- 172.21.15.98:0/1942141688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb4074db0 con 0x7fbea0003970 2023-11-28T17:47:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.374+0000 7fbea9ffb640 1 -- 172.21.15.98:0/1942141688 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fbeb0032070 con 0x7fbea0003970 2023-11-28T17:47:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.375+0000 7fbea9ffb640 1 --2- 172.21.15.98:0/1942141688 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbe9806b250 0x7fbe9806d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:47:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.375+0000 7fbea9ffb640 1 -- 172.21.15.98:0/1942141688 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbeb0008050 con 0x7fbea0003970 2023-11-28T17:47:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.376+0000 7fbeb892b640 1 --2- 172.21.15.98:0/1942141688 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbe9806b250 0x7fbe9806d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:47:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.376+0000 7fbeb892b640 1 --2- 172.21.15.98:0/1942141688 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbe9806b250 0x7fbe9806d710 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbeb40c9860 tx=0x7fbea4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:47:59.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.380+0000 7fbea9ffb640 1 -- 172.21.15.98:0/1942141688 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbeb00769b0 con 0x7fbea0003970 2023-11-28T17:47:59.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.638+0000 7fbeba12e640 1 -- 172.21.15.98:0/1942141688 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbeb40cb620 con 0x7fbea0003970 2023-11-28T17:47:59.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.639+0000 7fbea9ffb640 1 -- 172.21.15.98:0/1942141688 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7fbeb0047620 con 0x7fbea0003970 2023-11-28T17:47:59.641 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:47:59.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.642+0000 7fbeba12e640 1 -- 172.21.15.98:0/1942141688 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbe9806b250 msgr2=0x7fbe9806d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:59.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.642+0000 7fbeba12e640 1 --2- 172.21.15.98:0/1942141688 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbe9806b250 0x7fbe9806d710 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbeb40c9860 tx=0x7fbea4008040 comp rx=0 tx=0).stop 2023-11-28T17:47:59.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.642+0000 7fbeba12e640 1 -- 172.21.15.98:0/1942141688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbea0003970 msgr2=0x7fbeb414d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:47:59.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.642+0000 7fbeba12e640 1 --2- 172.21.15.98:0/1942141688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbea0003970 0x7fbeb414d0f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0000f80 tx=0x7fbeb0005d90 comp rx=0 tx=0).stop 2023-11-28T17:47:59.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.642+0000 7fbeba12e640 1 -- 172.21.15.98:0/1942141688 shutdown_connections 2023-11-28T17:47:59.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.642+0000 7fbeba12e640 1 --2- 172.21.15.98:0/1942141688 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbeb40c5640 0x7fbeb41e2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:59.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.642+0000 7fbeba12e640 1 --2- 172.21.15.98:0/1942141688 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbe9806b250 0x7fbe9806d710 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:59.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.642+0000 7fbeba12e640 1 --2- 172.21.15.98:0/1942141688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbeb41f3990 0x7fbeb41e4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:59.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.642+0000 7fbeba12e640 1 --2- 172.21.15.98:0/1942141688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbea0003970 0x7fbeb414d0f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:47:59.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.642+0000 7fbeba12e640 1 -- 172.21.15.98:0/1942141688 >> 172.21.15.98:0/1942141688 conn(0x7fbeb4071c00 msgr2=0x7fbeb40b4230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:47:59.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.642+0000 7fbeba12e640 1 -- 172.21.15.98:0/1942141688 shutdown_connections 2023-11-28T17:47:59.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:47:59.642+0000 7fbeba12e640 1 -- 172.21.15.98:0/1942141688 wait complete. 2023-11-28T17:48:00.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~189s 2023-11-28T17:48:02.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:48:02.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.884+0000 7f0a0759e640 1 Processor -- start 2023-11-28T17:48:02.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.884+0000 7f0a0759e640 1 -- start start 2023-11-28T17:48:02.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.886+0000 7f0a0759e640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0a08153bb0 0x7f0a0815a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:02.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.886+0000 7f0a0759e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a080ca390 0x7f0a080ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:02.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.886+0000 7f0a0759e640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0815d330 con 0x7f0a080cf280 2023-11-28T17:48:02.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.886+0000 7f0a0759e640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a0815d120 con 0x7f0a08153bb0 2023-11-28T17:48:02.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.886+0000 7f0a0759e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0815d610 con 0x7f0a080ca390 2023-11-28T17:48:02.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.886+0000 7f09fdd9b640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a080ca390 msgr2=0x7f0a080ca7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:48:02.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.886+0000 7f09fdd9b640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a080ca390 0x7f0a080ca7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:02.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.887+0000 7f0a0659c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0a08153bb0 0x7f0a0815a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:02.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.887+0000 7f0a0659c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0a08153bb0 0x7f0a0815a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:51598/0 (socket says 172.21.15.98:51598) 2023-11-28T17:48:02.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.887+0000 7f0a0659c640 1 -- 172.21.15.98:0/2430522362 learned_addr learned my addr 172.21.15.98:0/2430522362 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:02.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.887+0000 7f0a0659c640 1 -- 172.21.15.98:0/2430522362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a080ca390 msgr2=0x7f0a080ca7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:48:02.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.887+0000 7f0a0659c640 1 --2- 172.21.15.98:0/2430522362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a080ca390 0x7f0a080ca7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:02.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.887+0000 7f0a0659c640 1 -- 172.21.15.98:0/2430522362 >> v1:172.21.15.113:6789/0 conn(0x7f0a080cf280 legacy=0x7f0a080b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:02.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.887+0000 7f0a0659c640 1 -- 172.21.15.98:0/2430522362 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0815d8f0 con 0x7f0a08153bb0 2023-11-28T17:48:02.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.888+0000 7f0a0659c640 1 --2- 172.21.15.98:0/2430522362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a08153bb0 0x7f0a0815a6d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009e50 tx=0x7f09f802c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65738299ef3a3ed3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:02.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.889+0000 7f0a05d9b640 1 -- 172.21.15.98:0/2430522362 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f802cbb0 con 0x7f0a08153bb0 2023-11-28T17:48:02.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.889+0000 7f0a05d9b640 1 -- 172.21.15.98:0/2430522362 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f802cd10 con 0x7f0a08153bb0 2023-11-28T17:48:02.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.889+0000 7f0a05d9b640 1 -- 172.21.15.98:0/2430522362 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f8034610 con 0x7f0a08153bb0 2023-11-28T17:48:02.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.889+0000 7f0a0759e640 1 -- 172.21.15.98:0/2430522362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a08153bb0 msgr2=0x7f0a0815a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:02.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.889+0000 7f0a0759e640 1 --2- 172.21.15.98:0/2430522362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a08153bb0 0x7f0a0815a6d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009e50 tx=0x7f09f802c010 comp rx=0 tx=0).stop 2023-11-28T17:48:02.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.890+0000 7f0a0759e640 1 -- 172.21.15.98:0/2430522362 shutdown_connections 2023-11-28T17:48:02.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.890+0000 7f0a0759e640 1 --2- 172.21.15.98:0/2430522362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a080ca390 0x7f0a080ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:02.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.890+0000 7f0a0759e640 1 --2- 172.21.15.98:0/2430522362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a08153bb0 0x7f0a0815a6d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:02.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.890+0000 7f0a0759e640 1 -- 172.21.15.98:0/2430522362 >> 172.21.15.98:0/2430522362 conn(0x7f0a08071c00 msgr2=0x7f0a08072010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:02.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.890+0000 7f0a0759e640 1 -- 172.21.15.98:0/2430522362 shutdown_connections 2023-11-28T17:48:02.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.890+0000 7f0a0759e640 1 -- 172.21.15.98:0/2430522362 wait complete. 2023-11-28T17:48:02.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.891+0000 7f0a0759e640 1 Processor -- start 2023-11-28T17:48:02.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.892+0000 7f0a0759e640 1 -- start start 2023-11-28T17:48:02.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.892+0000 7f0a0759e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a080ca390 0x7f0a081545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:02.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f0a0759e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a080cf280 0x7f0a08151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:02.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f0a0759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a08153bb0 0x7f0a08151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:02.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f0a0759e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a081f3b80 con 0x7f0a08153bb0 2023-11-28T17:48:02.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f0a0759e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a081f35e0 con 0x7f0a080ca390 2023-11-28T17:48:02.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f0a0759e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a081f38a0 con 0x7f0a080cf280 2023-11-28T17:48:02.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f09fdd9b640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a080cf280 msgr2=0x7f0a08151640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:48:02.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f09fdd9b640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a080cf280 0x7f0a08151640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:02.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f0a06d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a08153bb0 0x7f0a08151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f0a06d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a08153bb0 0x7f0a08151b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43830/0 (socket says 172.21.15.98:43830) 2023-11-28T17:48:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f0a06d9d640 1 -- 172.21.15.98:0/2473913899 learned_addr learned my addr 172.21.15.98:0/2473913899 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f0a0659c640 1 --2- 172.21.15.98:0/2473913899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a080ca390 0x7f0a081545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f0a06d9d640 1 -- 172.21.15.98:0/2473913899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a080cf280 msgr2=0x7f0a08151640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:48:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f0a06d9d640 1 --2- 172.21.15.98:0/2473913899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a080cf280 0x7f0a08151640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f0a06d9d640 1 -- 172.21.15.98:0/2473913899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a080ca390 msgr2=0x7f0a081545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f0a06d9d640 1 --2- 172.21.15.98:0/2473913899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a080ca390 0x7f0a081545d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f0a06d9d640 1 -- 172.21.15.98:0/2473913899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f8009b30 con 0x7f0a08153bb0 2023-11-28T17:48:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.893+0000 7f0a0659c640 1 --2- 172.21.15.98:0/2473913899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a080ca390 0x7f0a081545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:48:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.894+0000 7f0a06d9d640 1 --2- 172.21.15.98:0/2473913899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a08153bb0 0x7f0a08151b80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0a00001eb0 tx=0x7f0a0000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:02.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.894+0000 7f09ff7fe640 1 -- 172.21.15.98:0/2473913899 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a000052d0 con 0x7f0a08153bb0 2023-11-28T17:48:02.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.894+0000 7f09ff7fe640 1 -- 172.21.15.98:0/2473913899 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a00005430 con 0x7f0a08153bb0 2023-11-28T17:48:02.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.894+0000 7f09ff7fe640 1 -- 172.21.15.98:0/2473913899 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a00003810 con 0x7f0a08153bb0 2023-11-28T17:48:02.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.894+0000 7f0a0759e640 1 -- 172.21.15.98:0/2473913899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a080c4ad0 con 0x7f0a08153bb0 2023-11-28T17:48:02.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.894+0000 7f0a0759e640 1 -- 172.21.15.98:0/2473913899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a081577b0 con 0x7f0a08153bb0 2023-11-28T17:48:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.895+0000 7f0a0759e640 1 -- 172.21.15.98:0/2473913899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a08074db0 con 0x7f0a08153bb0 2023-11-28T17:48:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.896+0000 7f09ff7fe640 1 -- 172.21.15.98:0/2473913899 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0a00003970 con 0x7f0a08153bb0 2023-11-28T17:48:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.898+0000 7f09ff7fe640 1 --2- 172.21.15.98:0/2473913899 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f09e806b250 0x7f09e806d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.898+0000 7f09ff7fe640 1 -- 172.21.15.98:0/2473913899 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0a00089140 con 0x7f0a08153bb0 2023-11-28T17:48:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.898+0000 7f0a0659c640 1 --2- 172.21.15.98:0/2473913899 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f09e806b250 0x7f09e806d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.898+0000 7f0a0659c640 1 --2- 172.21.15.98:0/2473913899 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f09e806b250 0x7f09e806d710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f09f8035000 tx=0x7f09f8030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:02.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:02.902+0000 7f09ff7fe640 1 -- 172.21.15.98:0/2473913899 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a00053fc0 con 0x7f0a08153bb0 2023-11-28T17:48:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:03.198+0000 7f0a0759e640 1 -- 172.21.15.98:0/2473913899 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0a080caf40 con 0x7f0a08153bb0 2023-11-28T17:48:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:03.199+0000 7f09ff7fe640 1 -- 172.21.15.98:0/2473913899 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f0a000575e0 con 0x7f0a08153bb0 2023-11-28T17:48:03.201 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:48:03.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:03.202+0000 7f0a0759e640 1 -- 172.21.15.98:0/2473913899 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f09e806b250 msgr2=0x7f09e806d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:03.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:03.202+0000 7f0a0759e640 1 --2- 172.21.15.98:0/2473913899 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f09e806b250 0x7f09e806d710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f09f8035000 tx=0x7f09f8030000 comp rx=0 tx=0).stop 2023-11-28T17:48:03.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:03.202+0000 7f0a0759e640 1 -- 172.21.15.98:0/2473913899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a08153bb0 msgr2=0x7f0a08151b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:03.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:03.202+0000 7f0a0759e640 1 --2- 172.21.15.98:0/2473913899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a08153bb0 0x7f0a08151b80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0a00001eb0 tx=0x7f0a0000c890 comp rx=0 tx=0).stop 2023-11-28T17:48:03.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:03.202+0000 7f0a0759e640 1 -- 172.21.15.98:0/2473913899 shutdown_connections 2023-11-28T17:48:03.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:03.202+0000 7f0a0759e640 1 --2- 172.21.15.98:0/2473913899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a08153bb0 0x7f0a08151b80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:03.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:03.202+0000 7f0a0759e640 1 --2- 172.21.15.98:0/2473913899 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0a080cf280 0x7f0a08151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:03.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:03.202+0000 7f0a0759e640 1 --2- 172.21.15.98:0/2473913899 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f09e806b250 0x7f09e806d710 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:03.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:03.202+0000 7f0a0759e640 1 --2- 172.21.15.98:0/2473913899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a080ca390 0x7f0a081545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:03.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:03.202+0000 7f0a0759e640 1 -- 172.21.15.98:0/2473913899 >> 172.21.15.98:0/2473913899 conn(0x7f0a08071c00 msgr2=0x7f0a08071fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:03.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:03.202+0000 7f0a0759e640 1 -- 172.21.15.98:0/2473913899 shutdown_connections 2023-11-28T17:48:03.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:03.202+0000 7f0a0759e640 1 -- 172.21.15.98:0/2473913899 wait complete. 2023-11-28T17:48:06.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:48:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.442+0000 7facef778640 1 Processor -- start 2023-11-28T17:48:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.442+0000 7facef778640 1 -- start start 2023-11-28T17:48:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.443+0000 7facef778640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 0x7face8157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.443+0000 7facef778640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face815bc10 0x7face80c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.443+0000 7facef778640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face815b150 con 0x7face80c6140 2023-11-28T17:48:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.443+0000 7facef778640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face806e1f0 con 0x7face8151d80 2023-11-28T17:48:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.443+0000 7facef778640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face806e4d0 con 0x7face815bc10 2023-11-28T17:48:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.444+0000 7facecd13640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face815bc10 msgr2=0x7face80c5c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:48:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.444+0000 7facecd13640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face815bc10 0x7face80c5c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.444+0000 7faced514640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 0x7face8157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.444+0000 7faced514640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 0x7face8157b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43834/0 (socket says 172.21.15.98:43834) 2023-11-28T17:48:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.444+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 learned_addr learned my addr 172.21.15.98:0/2089762455 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.444+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 msgr2=0x7face8157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:48:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.444+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 msgr2=0x7face8157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:48:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.444+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 0x7face8157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:48:06.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.444+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 0x7face8157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:06.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.645+0000 7facecd13640 1 -- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face815bc10 msgr2=0x7face80c5c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:48:06.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.645+0000 7facecd13640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face815bc10 0x7face80c5c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:48:06.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.645+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 0x7face8157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:06.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.645+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 msgr2=0x7face8157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:48:06.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.645+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 msgr2=0x7face8157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:48:06.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.645+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 0x7face8157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:48:06.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:06.645+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 0x7face8157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:48:07.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:07.046+0000 7facecd13640 1 -- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face815bc10 msgr2=0x7face80c5c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:48:07.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:07.046+0000 7facecd13640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face815bc10 0x7face80c5c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:48:07.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:07.046+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 0x7face8157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:07.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:07.047+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 msgr2=0x7face8157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:48:07.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:07.047+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 msgr2=0x7face8157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:48:07.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:07.047+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 0x7face8157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:48:07.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:07.047+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 0x7face8157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:48:07.395 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~196s 2023-11-28T17:48:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:07.847+0000 7facecd13640 1 -- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face815bc10 msgr2=0x7face80c5c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:48:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:07.847+0000 7facecd13640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face815bc10 0x7face80c5c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:48:07.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:07.848+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 0x7face8157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:07.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:07.849+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 msgr2=0x7face8157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:48:07.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:07.849+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 msgr2=0x7face8157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:48:07.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:07.849+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 0x7face8157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:48:07.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:07.849+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 0x7face8157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:48:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.444+0000 7facdeffd640 1 -- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face815bc10 msgr2=0x7face80c5c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:48:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.444+0000 7facdeffd640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7face815bc10 0x7face80c5c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.444+0000 7facdeffd640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 msgr2=0x7face8157b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:48:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.444+0000 7facdeffd640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.98:3300/0 conn(0x7face8151d80 0x7face8157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.444+0000 7facdeffd640 1 -- 172.21.15.98:0/2089762455 >> v1:172.21.15.113:6789/0 conn(0x7face80c6140 legacy=0x7face80c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.444+0000 7facdeffd640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 0x7faccc005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.444+0000 7facdeffd640 1 -- 172.21.15.98:0/2089762455 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face806e7b0 con 0x7faccc006220 2023-11-28T17:48:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.444+0000 7facdeffd640 1 -- 172.21.15.98:0/2089762455 --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face806e1f0 con 0x7faccc0036c0 2023-11-28T17:48:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.445+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 0x7faccc005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.445+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 msgr2=0x7faccc005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:48:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.445+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 msgr2=0x7faccc005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:48:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.445+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 0x7faccc005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:48:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.445+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 0x7faccc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:09.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.647+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 0x7faccc005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:09.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.647+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 msgr2=0x7faccc005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:48:09.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.647+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 msgr2=0x7faccc005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:48:09.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.647+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 0x7faccc005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:48:09.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:09.647+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 0x7faccc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:48:09.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:48:09.832 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:48:10.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.048+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 0x7faccc005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:10.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.049+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 msgr2=0x7faccc005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:48:10.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.049+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 msgr2=0x7faccc005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:48:10.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.049+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 0x7faccc005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:48:10.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.049+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 0x7faccc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:48:10.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.544+0000 7facdffff640 1 -- 172.21.15.98:0/2089762455 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645374234 0 0) 0x7face806e7b0 con 0x7faccc006220 2023-11-28T17:48:10.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.544+0000 7facdffff640 1 -- 172.21.15.98:0/2089762455 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facd4003520 con 0x7faccc006220 2023-11-28T17:48:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.546+0000 7facdffff640 1 -- 172.21.15.98:0/2089762455 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1348811670 0 0) 0x7facd4003520 con 0x7faccc006220 2023-11-28T17:48:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.546+0000 7facdffff640 1 -- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 msgr2=0x7faccc005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:48:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.546+0000 7facdffff640 1 --2- 172.21.15.98:0/2089762455 >> v2:172.21.15.113:3300/0 conn(0x7faccc0036c0 0x7faccc005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.546+0000 7facdffff640 1 -- 172.21.15.98:0/2089762455 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc00afc0 con 0x7faccc006220 2023-11-28T17:48:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.546+0000 7facdffff640 1 -- 172.21.15.98:0/2089762455 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7facd8004340 con 0x7faccc006220 2023-11-28T17:48:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.547+0000 7facdffff640 1 -- 172.21.15.98:0/2089762455 >> v1:172.21.15.98:6789/0 conn(0x7faccc006220 legacy=0x7faccc0086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.547+0000 7facedd15640 1 -- 172.21.15.98:0/2089762455 reap_dead start 2023-11-28T17:48:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.547+0000 7facdffff640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd4003970 0x7facd4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.547+0000 7facdffff640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facd40047c0 0x7facd4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.547+0000 7facdffff640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd40051d0 0x7facd400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:10.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.547+0000 7facdffff640 1 -- 172.21.15.98:0/2089762455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face806e1f0 con 0x7facd40051d0 2023-11-28T17:48:10.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.547+0000 7facdffff640 1 -- 172.21.15.98:0/2089762455 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd400c100 con 0x7facd40047c0 2023-11-28T17:48:10.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.547+0000 7facdffff640 1 -- 172.21.15.98:0/2089762455 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd400c2c0 con 0x7facd4003970 2023-11-28T17:48:10.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.547+0000 7faced514640 1 -- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd4003970 msgr2=0x7facd4003d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:48:10.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.547+0000 7faced514640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd4003970 0x7facd4003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:10.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.548+0000 7facedd15640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd40051d0 0x7facd400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:10.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.548+0000 7facecd13640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facd40047c0 0x7facd4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:10.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.549+0000 7facedd15640 1 -- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd4003970 msgr2=0x7facd4003d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:48:10.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.549+0000 7facedd15640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd4003970 0x7facd4003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:10.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.549+0000 7facedd15640 1 -- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facd40047c0 msgr2=0x7facd4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:10.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.549+0000 7facedd15640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facd40047c0 0x7facd4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:10.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.549+0000 7facedd15640 1 -- 172.21.15.98:0/2089762455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd4003520 con 0x7facd40051d0 2023-11-28T17:48:10.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.552+0000 7facedd15640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd40051d0 0x7facd400b980 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7face80c56e0 tx=0x7facd40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3fc88a2e0edfffe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:10.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.552+0000 7facdffff640 1 -- 172.21.15.98:0/2089762455 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face0005d40 con 0x7facd40051d0 2023-11-28T17:48:10.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.552+0000 7facdffff640 1 -- 172.21.15.98:0/2089762455 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face002e070 con 0x7facd40051d0 2023-11-28T17:48:10.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.552+0000 7facdffff640 1 -- 172.21.15.98:0/2089762455 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face00072b0 con 0x7facd40051d0 2023-11-28T17:48:10.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.553+0000 7facef778640 1 -- 172.21.15.98:0/2089762455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd40051d0 msgr2=0x7facd400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:10.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.553+0000 7facef778640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd40051d0 0x7facd400b980 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7face80c56e0 tx=0x7facd40057f0 comp rx=0 tx=0).stop 2023-11-28T17:48:10.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.553+0000 7facef778640 1 -- 172.21.15.98:0/2089762455 shutdown_connections 2023-11-28T17:48:10.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.553+0000 7facef778640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd40051d0 0x7facd400b980 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:10.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.553+0000 7facef778640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facd40047c0 0x7facd4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:10.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.553+0000 7facef778640 1 --2- 172.21.15.98:0/2089762455 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd4003970 0x7facd4003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:10.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.553+0000 7facef778640 1 -- 172.21.15.98:0/2089762455 >> 172.21.15.98:0/2089762455 conn(0x7face806a4d0 msgr2=0x7face806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:10.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.554+0000 7facef778640 1 -- 172.21.15.98:0/2089762455 shutdown_connections 2023-11-28T17:48:10.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.554+0000 7facef778640 1 -- 172.21.15.98:0/2089762455 wait complete. 2023-11-28T17:48:10.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.558+0000 7facef778640 1 Processor -- start 2023-11-28T17:48:10.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.559+0000 7facef778640 1 -- start start 2023-11-28T17:48:10.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facef778640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facd4003970 0x7face81f0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:10.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facef778640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd40047c0 0x7face81f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:10.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facef778640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd40051d0 0x7face80721a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:10.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facef778640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face81e2bf0 con 0x7facd40047c0 2023-11-28T17:48:10.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facef778640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face81f0200 con 0x7facd4003970 2023-11-28T17:48:10.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facef778640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face81f3680 con 0x7facd40051d0 2023-11-28T17:48:10.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facecd13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd40047c0 0x7face81f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:10.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facedd15640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd40051d0 msgr2=0x7face80721a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:48:10.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facedd15640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd40051d0 0x7face80721a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:10.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facecd13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd40047c0 0x7face81f3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44278/0 (socket says 172.21.15.98:44278) 2023-11-28T17:48:10.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facecd13640 1 -- 172.21.15.98:0/4149986644 learned_addr learned my addr 172.21.15.98:0/4149986644 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:10.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7faced514640 1 --2- 172.21.15.98:0/4149986644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facd4003970 0x7face81f0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:10.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facecd13640 1 -- 172.21.15.98:0/4149986644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd40051d0 msgr2=0x7face80721a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:48:10.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facecd13640 1 --2- 172.21.15.98:0/4149986644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd40051d0 0x7face80721a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:10.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facecd13640 1 -- 172.21.15.98:0/4149986644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facd4003970 msgr2=0x7face81f0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:10.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facecd13640 1 --2- 172.21.15.98:0/4149986644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facd4003970 0x7face81f0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:10.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facecd13640 1 -- 172.21.15.98:0/4149986644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face81efbd0 con 0x7facd40047c0 2023-11-28T17:48:10.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.561+0000 7facecd13640 1 --2- 172.21.15.98:0/4149986644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd40047c0 0x7face81f3140 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7facd80043a0 tx=0x7facd800d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:10.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.562+0000 7facddffb640 1 -- 172.21.15.98:0/4149986644 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd800dc30 con 0x7facd40047c0 2023-11-28T17:48:10.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.562+0000 7facddffb640 1 -- 172.21.15.98:0/4149986644 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd800dd90 con 0x7facd40047c0 2023-11-28T17:48:10.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.562+0000 7facddffb640 1 -- 172.21.15.98:0/4149986644 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8013610 con 0x7facd40047c0 2023-11-28T17:48:10.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.562+0000 7facef778640 1 -- 172.21.15.98:0/4149986644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face81f3910 con 0x7facd40047c0 2023-11-28T17:48:10.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.562+0000 7facef778640 1 -- 172.21.15.98:0/4149986644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face81f3ed0 con 0x7facd40047c0 2023-11-28T17:48:10.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.563+0000 7facef778640 1 -- 172.21.15.98:0/4149986644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face8067bf0 con 0x7facd40047c0 2023-11-28T17:48:10.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.564+0000 7facddffb640 1 -- 172.21.15.98:0/4149986644 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7facd8013770 con 0x7facd40047c0 2023-11-28T17:48:10.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.565+0000 7facddffb640 1 --2- 172.21.15.98:0/4149986644 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7faccc06ba70 0x7faccc06df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:10.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.565+0000 7facddffb640 1 -- 172.21.15.98:0/4149986644 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7facd8089ee0 con 0x7facd40047c0 2023-11-28T17:48:10.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.566+0000 7faced514640 1 --2- 172.21.15.98:0/4149986644 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7faccc06ba70 0x7faccc06df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:10.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.567+0000 7faced514640 1 --2- 172.21.15.98:0/4149986644 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7faccc06ba70 0x7faccc06df30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7face4006e00 tx=0x7face4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:10.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.571+0000 7facddffb640 1 -- 172.21.15.98:0/4149986644 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7facd8054c40 con 0x7facd40047c0 2023-11-28T17:48:10.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.838+0000 7facef778640 1 -- 172.21.15.98:0/4149986644 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7face806b2d0 con 0x7facd40047c0 2023-11-28T17:48:10.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.839+0000 7facddffb640 1 -- 172.21.15.98:0/4149986644 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7facd8017070 con 0x7facd40047c0 2023-11-28T17:48:10.840 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":74,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:48:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.842+0000 7facef778640 1 -- 172.21.15.98:0/4149986644 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7faccc06ba70 msgr2=0x7faccc06df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.842+0000 7facef778640 1 --2- 172.21.15.98:0/4149986644 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7faccc06ba70 0x7faccc06df30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7face4006e00 tx=0x7face4008040 comp rx=0 tx=0).stop 2023-11-28T17:48:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.842+0000 7facef778640 1 -- 172.21.15.98:0/4149986644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd40047c0 msgr2=0x7face81f3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.842+0000 7facef778640 1 --2- 172.21.15.98:0/4149986644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd40047c0 0x7face81f3140 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7facd80043a0 tx=0x7facd800d520 comp rx=0 tx=0).stop 2023-11-28T17:48:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.842+0000 7facef778640 1 -- 172.21.15.98:0/4149986644 shutdown_connections 2023-11-28T17:48:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.842+0000 7facef778640 1 --2- 172.21.15.98:0/4149986644 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7facd40051d0 0x7face80721a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.842+0000 7facef778640 1 --2- 172.21.15.98:0/4149986644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd40047c0 0x7face81f3140 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.842+0000 7facef778640 1 --2- 172.21.15.98:0/4149986644 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7faccc06ba70 0x7faccc06df30 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.842+0000 7facef778640 1 --2- 172.21.15.98:0/4149986644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7facd4003970 0x7face81f0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:10.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.842+0000 7facef778640 1 -- 172.21.15.98:0/4149986644 >> 172.21.15.98:0/4149986644 conn(0x7face806a4d0 msgr2=0x7face80cad70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:10.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.842+0000 7facef778640 1 -- 172.21.15.98:0/4149986644 shutdown_connections 2023-11-28T17:48:10.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:10.842+0000 7facef778640 1 -- 172.21.15.98:0/4149986644 wait complete. 2023-11-28T17:48:10.861 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-11-28T17:48:10.861 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2023-11-28T17:48:10.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:48:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.063+0000 7fab65553640 1 Processor -- start 2023-11-28T17:48:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.064+0000 7fab65553640 1 -- start start 2023-11-28T17:48:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab65553640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab60070640 0x7fab600c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab65553640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab60070a20 0x7fab60154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab65553640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab60158640 con 0x7fab600c36a0 2023-11-28T17:48:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab65553640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab600c7d60 con 0x7fab60070640 2023-11-28T17:48:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab65553640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab600ce300 con 0x7fab60070a20 2023-11-28T17:48:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab567fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab60070a20 msgr2=0x7fab60154dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:48:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab5effd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab60070640 0x7fab600c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab567fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab60070a20 0x7fab60154dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab5effd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab60070640 0x7fab600c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44284/0 (socket says 172.21.15.98:44284) 2023-11-28T17:48:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab5f7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fab600c36a0 0x7fab6015b210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58164/0 (socket says 172.21.15.98:58164) 2023-11-28T17:48:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab5effd640 1 -- 172.21.15.98:0/1728166873 learned_addr learned my addr 172.21.15.98:0/1728166873 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab5effd640 1 -- 172.21.15.98:0/1728166873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab60070a20 msgr2=0x7fab60154dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:48:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab5effd640 1 --2- 172.21.15.98:0/1728166873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab60070a20 0x7fab60154dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab5effd640 1 -- 172.21.15.98:0/1728166873 >> v1:172.21.15.98:6789/0 conn(0x7fab600c36a0 legacy=0x7fab6015b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:11.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab5effd640 1 -- 172.21.15.98:0/1728166873 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab600c7f50 con 0x7fab60070640 2023-11-28T17:48:11.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.066+0000 7fab5e7fc640 1 -- 172.21.15.98:0/1728166873 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1483572221 0 0) 0x7fab60158640 con 0x7fab600c36a0 2023-11-28T17:48:11.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.067+0000 7fab5effd640 1 --2- 172.21.15.98:0/1728166873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab60070640 0x7fab600c8f00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fab50009870 tx=0x7fab5002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88a611c77b1c91e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:11.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.068+0000 7fab5e7fc640 1 -- 172.21.15.98:0/1728166873 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5002f850 con 0x7fab60070640 2023-11-28T17:48:11.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.068+0000 7fab5e7fc640 1 -- 172.21.15.98:0/1728166873 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5002f9b0 con 0x7fab60070640 2023-11-28T17:48:11.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.068+0000 7fab5e7fc640 1 -- 172.21.15.98:0/1728166873 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5002fd40 con 0x7fab60070640 2023-11-28T17:48:11.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.068+0000 7fab65553640 1 -- 172.21.15.98:0/1728166873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab60070640 msgr2=0x7fab600c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:11.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.068+0000 7fab65553640 1 --2- 172.21.15.98:0/1728166873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab60070640 0x7fab600c8f00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fab50009870 tx=0x7fab5002ba30 comp rx=0 tx=0).stop 2023-11-28T17:48:11.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.068+0000 7fab65553640 1 -- 172.21.15.98:0/1728166873 shutdown_connections 2023-11-28T17:48:11.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.068+0000 7fab65553640 1 --2- 172.21.15.98:0/1728166873 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab60070a20 0x7fab60154dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.068+0000 7fab65553640 1 --2- 172.21.15.98:0/1728166873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab60070640 0x7fab600c8f00 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.068+0000 7fab65553640 1 -- 172.21.15.98:0/1728166873 >> 172.21.15.98:0/1728166873 conn(0x7fab6006e620 msgr2=0x7fab6006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.069+0000 7fab65553640 1 -- 172.21.15.98:0/1728166873 shutdown_connections 2023-11-28T17:48:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.069+0000 7fab65553640 1 -- 172.21.15.98:0/1728166873 wait complete. 2023-11-28T17:48:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.070+0000 7fab65553640 1 Processor -- start 2023-11-28T17:48:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.070+0000 7fab65553640 1 -- start start 2023-11-28T17:48:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.071+0000 7fab65553640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab60070640 0x7fab6014e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.071+0000 7fab65553640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab60070a20 0x7fab601ee260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.071+0000 7fab65553640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab600c36a0 0x7fab600cc480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.071+0000 7fab65553640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab600c50a0 con 0x7fab60070640 2023-11-28T17:48:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.071+0000 7fab65553640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab601ee7a0 con 0x7fab60070a20 2023-11-28T17:48:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.071+0000 7fab65553640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab600b2480 con 0x7fab600c36a0 2023-11-28T17:48:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.072+0000 7fab5f7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab600c36a0 msgr2=0x7fab600cc480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:48:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.072+0000 7fab5effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab60070640 0x7fab6014e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.072+0000 7fab5f7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab600c36a0 0x7fab600cc480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.072+0000 7fab5effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab60070640 0x7fab6014e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44298/0 (socket says 172.21.15.98:44298) 2023-11-28T17:48:11.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.072+0000 7fab5effd640 1 -- 172.21.15.98:0/2117561548 learned_addr learned my addr 172.21.15.98:0/2117561548 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:11.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.072+0000 7fab567fc640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab60070a20 0x7fab601ee260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:11.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.072+0000 7fab5effd640 1 -- 172.21.15.98:0/2117561548 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab600c36a0 msgr2=0x7fab600cc480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:48:11.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.072+0000 7fab5effd640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fab600c36a0 0x7fab600cc480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.072+0000 7fab5effd640 1 -- 172.21.15.98:0/2117561548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab60070a20 msgr2=0x7fab601ee260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:11.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.072+0000 7fab5effd640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab60070a20 0x7fab601ee260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.072+0000 7fab5effd640 1 -- 172.21.15.98:0/2117561548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab500094d0 con 0x7fab60070640 2023-11-28T17:48:11.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.072+0000 7fab5effd640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab60070640 0x7fab6014e460 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fab50009870 tx=0x7fab50004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:11.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.073+0000 7fab57fff640 1 -- 172.21.15.98:0/2117561548 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5002f800 con 0x7fab60070640 2023-11-28T17:48:11.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.073+0000 7fab57fff640 1 -- 172.21.15.98:0/2117561548 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab50041400 con 0x7fab60070640 2023-11-28T17:48:11.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.073+0000 7fab65553640 1 -- 172.21.15.98:0/2117561548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab600b32a0 con 0x7fab60070640 2023-11-28T17:48:11.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.073+0000 7fab57fff640 1 -- 172.21.15.98:0/2117561548 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab50041790 con 0x7fab60070640 2023-11-28T17:48:11.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.073+0000 7fab65553640 1 -- 172.21.15.98:0/2117561548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab600b2a00 con 0x7fab60070640 2023-11-28T17:48:11.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.074+0000 7fab65553640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600b2bf0 0x7fab600b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.074+0000 7fab65553640 1 -- 172.21.15.98:0/2117561548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6006af60 con 0x7fab600b2bf0 2023-11-28T17:48:11.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.074+0000 7fab567fc640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600b2bf0 0x7fab600b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:11.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.074+0000 7fab567fc640 1 -- 172.21.15.98:0/2117561548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab6006ea00 con 0x7fab600b2bf0 2023-11-28T17:48:11.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.075+0000 7fab567fc640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600b2bf0 0x7fab600b0f60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fab6014eab0 tx=0x7fab4800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:11.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.075+0000 7fab57fff640 1 -- 172.21.15.98:0/2117561548 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4800ded0 con 0x7fab600b2bf0 2023-11-28T17:48:11.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.075+0000 7fab57fff640 1 -- 172.21.15.98:0/2117561548 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fab500418f0 con 0x7fab60070640 2023-11-28T17:48:11.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.076+0000 7fab57fff640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fab4006b340 0x7fab4006d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.076+0000 7fab57fff640 1 -- 172.21.15.98:0/2117561548 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fab5003c070 con 0x7fab60070640 2023-11-28T17:48:11.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.076+0000 7fab5f7fe640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fab4006b340 0x7fab4006d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:11.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.076+0000 7fab57fff640 1 -- 172.21.15.98:0/2117561548 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fab480039b0 con 0x7fab600b2bf0 2023-11-28T17:48:11.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.076+0000 7fab57fff640 1 -- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600b2bf0 msgr2=0x7fab600b0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:11.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.076+0000 7fab57fff640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600b2bf0 0x7fab600b0f60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fab6014eab0 tx=0x7fab4800db90 comp rx=0 tx=0).stop 2023-11-28T17:48:11.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.076+0000 7fab57fff640 1 -- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600b2bf0 msgr2=0x7fab600b0f60 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:11.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.076+0000 7fab57fff640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600b2bf0 0x7fab600b0f60 secure :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fab6014eab0 tx=0x7fab4800db90 comp rx=0 tx=0).stop 2023-11-28T17:48:11.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.077+0000 7fab5f7fe640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fab4006b340 0x7fab4006d800 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fab58007fd0 tx=0x7fab58007d00 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:11.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.083+0000 7fab65553640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600b14a0 0x7fab6014d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.083+0000 7fab65553640 1 -- 172.21.15.98:0/2117561548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab60067de0 con 0x7fab600b14a0 2023-11-28T17:48:11.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.083+0000 7fab567fc640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600b14a0 0x7fab6014d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:11.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.084+0000 7fab567fc640 1 -- 172.21.15.98:0/2117561548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fab601582d0 con 0x7fab600b14a0 2023-11-28T17:48:11.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.084+0000 7fab567fc640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600b14a0 0x7fab6014d810 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fab48016000 tx=0x7fab480059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:11.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.084+0000 7fab57fff640 1 -- 172.21.15.98:0/2117561548 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab480055a0 con 0x7fab600b14a0 2023-11-28T17:48:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.085+0000 7fab57fff640 1 -- 172.21.15.98:0/2117561548 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4966 (secure 0 0 0) 0x7fab48005700 con 0x7fab600b14a0 2023-11-28T17:48:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.085+0000 7fab57fff640 1 -- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600b14a0 msgr2=0x7fab6014d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.085+0000 7fab57fff640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600b14a0 0x7fab6014d810 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fab48016000 tx=0x7fab480059d0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.085+0000 7fab57fff640 1 -- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600b14a0 msgr2=0x7fab6014d810 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:11.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.085+0000 7fab57fff640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab600b14a0 0x7fab6014d810 secure :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fab48016000 tx=0x7fab480059d0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.086 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:48:11.086 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:48:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:48:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:48:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 74, 2023-11-28T17:48:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:48:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:48:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: 1 2023-11-28T17:48:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:48:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:48:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:48:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:48:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:48:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:48:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:48:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:48:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:48:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:48:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:48:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:48:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:48:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:48:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:48:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:48:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:48:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:48:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:48:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:48:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:48:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:48:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:48:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:48:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:48:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:48:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:48:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:48:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:48:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:48:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:48:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:48:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:48:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:48:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:48:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:48:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:48:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:48:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:48:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:48:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:48:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:11.098 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:48:11.098 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.098 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:11.098 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:48:11.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.098 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:11.098 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:11.098 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:48:11.098 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.098 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:11.098 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:48:11.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.098 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:11.098 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:11.099 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 6 2023-11-28T17:48:11.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.099 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:11.099 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:48:11.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.099 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:11.099 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:11.099 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:48:11.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.099 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:11.099 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:48:11.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.099 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:11.099 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:11.100 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:48:11.100 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.100 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:11.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.100 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:48:11.100 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:48:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.085+0000 7fab65553640 1 -- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fab4006b340 msgr2=0x7fab4006d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.085+0000 7fab65553640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fab4006b340 0x7fab4006d800 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fab58007fd0 tx=0x7fab58007d00 comp rx=0 tx=0).stop 2023-11-28T17:48:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.085+0000 7fab65553640 1 -- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab60070640 msgr2=0x7fab6014e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.085+0000 7fab65553640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab60070640 0x7fab6014e460 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fab50009870 tx=0x7fab50004730 comp rx=0 tx=0).stop 2023-11-28T17:48:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.085+0000 7fab5f7fe640 1 -- 172.21.15.98:0/2117561548 reap_dead start 2023-11-28T17:48:11.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.085+0000 7fab65553640 1 -- 172.21.15.98:0/2117561548 shutdown_connections 2023-11-28T17:48:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.085+0000 7fab65553640 1 --2- 172.21.15.98:0/2117561548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab60070640 0x7fab6014e460 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.085+0000 7fab65553640 1 -- 172.21.15.98:0/2117561548 >> 172.21.15.98:0/2117561548 conn(0x7fab6006e620 msgr2=0x7fab600c2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.085+0000 7fab65553640 1 -- 172.21.15.98:0/2117561548 shutdown_connections 2023-11-28T17:48:11.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.085+0000 7fab65553640 1 -- 172.21.15.98:0/2117561548 wait complete. 2023-11-28T17:48:11.102 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:48:11.102 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-11-28T17:48:11.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:48:11.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.313+0000 7fb8ec543640 1 Processor -- start 2023-11-28T17:48:11.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.313+0000 7fb8ec543640 1 -- start start 2023-11-28T17:48:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.314+0000 7fb8ec543640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb8e40c4380 0x7fb8e40c2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.314+0000 7fb8ec543640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb8e415a0f0 0x7fb8e415a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.314+0000 7fb8ec543640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e4156980 con 0x7fb8e40c4760 2023-11-28T17:48:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.314+0000 7fb8ec543640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e40c2a60 con 0x7fb8e415a0f0 2023-11-28T17:48:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.314+0000 7fb8ec543640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e415aa60 con 0x7fb8e40c4380 2023-11-28T17:48:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.315+0000 7fb8eaae0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb8e415a0f0 0x7fb8e415a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.315+0000 7fb8eaae0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb8e415a0f0 0x7fb8e415a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44334/0 (socket says 172.21.15.98:44334) 2023-11-28T17:48:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.315+0000 7fb8eaae0640 1 -- 172.21.15.98:0/4152361068 learned_addr learned my addr 172.21.15.98:0/4152361068 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.315+0000 7fb8e9ade640 1 --1- 172.21.15.98:0/4152361068 >> v1:172.21.15.98:6789/0 conn(0x7fb8e40c4760 0x7fb8e40c6020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58168/0 (socket says 172.21.15.98:58168) 2023-11-28T17:48:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.315+0000 7fb8ea2df640 1 --2- 172.21.15.98:0/4152361068 >> v2:172.21.15.113:3300/0 conn(0x7fb8e40c4380 0x7fb8e40c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.315+0000 7fb8eaae0640 1 -- 172.21.15.98:0/4152361068 >> v2:172.21.15.113:3300/0 conn(0x7fb8e40c4380 msgr2=0x7fb8e40c2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.315+0000 7fb8eaae0640 1 --2- 172.21.15.98:0/4152361068 >> v2:172.21.15.113:3300/0 conn(0x7fb8e40c4380 0x7fb8e40c2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.315+0000 7fb8eaae0640 1 -- 172.21.15.98:0/4152361068 >> v1:172.21.15.98:6789/0 conn(0x7fb8e40c4760 legacy=0x7fb8e40c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.315+0000 7fb8eaae0640 1 -- 172.21.15.98:0/4152361068 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e4156c40 con 0x7fb8e415a0f0 2023-11-28T17:48:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.315+0000 7fb8e92dd640 1 -- 172.21.15.98:0/4152361068 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923149648 0 0) 0x7fb8e4156980 con 0x7fb8e40c4760 2023-11-28T17:48:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.316+0000 7fb8eaae0640 1 --2- 172.21.15.98:0/4152361068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8e415a0f0 0x7fb8e415a520 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4070840 tx=0x7fb8dc02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447cdeacb20bf036 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.316+0000 7fb8e92dd640 1 -- 172.21.15.98:0/4152361068 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc02cbb0 con 0x7fb8e415a0f0 2023-11-28T17:48:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.316+0000 7fb8e92dd640 1 -- 172.21.15.98:0/4152361068 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8dc02cd10 con 0x7fb8e415a0f0 2023-11-28T17:48:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.316+0000 7fb8e92dd640 1 -- 172.21.15.98:0/4152361068 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc034610 con 0x7fb8e415a0f0 2023-11-28T17:48:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.317+0000 7fb8ec543640 1 -- 172.21.15.98:0/4152361068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8e415a0f0 msgr2=0x7fb8e415a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.317+0000 7fb8ec543640 1 --2- 172.21.15.98:0/4152361068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8e415a0f0 0x7fb8e415a520 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4070840 tx=0x7fb8dc02c010 comp rx=0 tx=0).stop 2023-11-28T17:48:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.317+0000 7fb8ec543640 1 -- 172.21.15.98:0/4152361068 shutdown_connections 2023-11-28T17:48:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.317+0000 7fb8ec543640 1 --2- 172.21.15.98:0/4152361068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8e415a0f0 0x7fb8e415a520 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.317+0000 7fb8ec543640 1 --2- 172.21.15.98:0/4152361068 >> v2:172.21.15.113:3300/0 conn(0x7fb8e40c4380 0x7fb8e40c2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.317+0000 7fb8ec543640 1 -- 172.21.15.98:0/4152361068 >> 172.21.15.98:0/4152361068 conn(0x7fb8e406c0f0 msgr2=0x7fb8e406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:11.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.317+0000 7fb8ec543640 1 -- 172.21.15.98:0/4152361068 shutdown_connections 2023-11-28T17:48:11.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.317+0000 7fb8ec543640 1 -- 172.21.15.98:0/4152361068 wait complete. 2023-11-28T17:48:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.319+0000 7fb8ec543640 1 Processor -- start 2023-11-28T17:48:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.319+0000 7fb8ec543640 1 -- start start 2023-11-28T17:48:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8ec543640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8e40c4380 0x7fb8e414ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8ec543640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e40c4760 0x7fb8e4145b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8ec543640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8e415a0f0 0x7fb8e40c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8ec543640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e414c530 con 0x7fb8e40c4380 2023-11-28T17:48:11.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8ec543640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e414b5e0 con 0x7fb8e40c4760 2023-11-28T17:48:11.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8ec543640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e414d680 con 0x7fb8e415a0f0 2023-11-28T17:48:11.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8ea2df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8e40c4380 0x7fb8e414ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:11.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8eaae0640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8e415a0f0 msgr2=0x7fb8e40c9f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:48:11.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8ea2df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8e40c4380 0x7fb8e414ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44338/0 (socket says 172.21.15.98:44338) 2023-11-28T17:48:11.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8ea2df640 1 -- 172.21.15.98:0/162703949 learned_addr learned my addr 172.21.15.98:0/162703949 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:11.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8eaae0640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8e415a0f0 0x7fb8e40c9f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:11.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8ea2df640 1 -- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8e415a0f0 msgr2=0x7fb8e40c9f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:48:11.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8ea2df640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb8e415a0f0 0x7fb8e40c9f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8ea2df640 1 -- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e40c4760 msgr2=0x7fb8e4145b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:11.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8ea2df640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e40c4760 0x7fb8e4145b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.320+0000 7fb8ea2df640 1 -- 172.21.15.98:0/162703949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8dc009b30 con 0x7fb8e40c4380 2023-11-28T17:48:11.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.321+0000 7fb8ea2df640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8e40c4380 0x7fb8e414ba70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb8e000d800 tx=0x7fb8e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:11.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.321+0000 7fb8daffd640 1 -- 172.21.15.98:0/162703949 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e000b6c0 con 0x7fb8e40c4380 2023-11-28T17:48:11.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.321+0000 7fb8daffd640 1 -- 172.21.15.98:0/162703949 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e000b840 con 0x7fb8e40c4380 2023-11-28T17:48:11.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.321+0000 7fb8ec543640 1 -- 172.21.15.98:0/162703949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e414d8f0 con 0x7fb8e40c4380 2023-11-28T17:48:11.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.321+0000 7fb8daffd640 1 -- 172.21.15.98:0/162703949 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e000bbf0 con 0x7fb8e40c4380 2023-11-28T17:48:11.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.321+0000 7fb8ec543640 1 -- 172.21.15.98:0/162703949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e414acb0 con 0x7fb8e40c4380 2023-11-28T17:48:11.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.322+0000 7fb8ec543640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e4156550 0x7fb8e414aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.322+0000 7fb8ec543640 1 -- 172.21.15.98:0/162703949 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e40744d0 con 0x7fb8e4156550 2023-11-28T17:48:11.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.322+0000 7fb8e9ade640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e4156550 0x7fb8e414aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:11.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.323+0000 7fb8daffd640 1 -- 172.21.15.98:0/162703949 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb8e000bd50 con 0x7fb8e40c4380 2023-11-28T17:48:11.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.323+0000 7fb8e9ade640 1 -- 172.21.15.98:0/162703949 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8e40cbe70 con 0x7fb8e4156550 2023-11-28T17:48:11.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.324+0000 7fb8e9ade640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e4156550 0x7fb8e414aea0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb8d40011e0 tx=0x7fb8d400c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.324+0000 7fb8daffd640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb8c806b250 0x7fb8c806d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.324+0000 7fb8daffd640 1 -- 172.21.15.98:0/162703949 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb8e008a240 con 0x7fb8e40c4380 2023-11-28T17:48:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.324+0000 7fb8daffd640 1 -- 172.21.15.98:0/162703949 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d40064e0 con 0x7fb8e4156550 2023-11-28T17:48:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.324+0000 7fb8eaae0640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb8c806b250 0x7fb8c806d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.325+0000 7fb8daffd640 1 -- 172.21.15.98:0/162703949 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fb8d4006640 con 0x7fb8e4156550 2023-11-28T17:48:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.325+0000 7fb8daffd640 1 -- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e4156550 msgr2=0x7fb8e414aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.325+0000 7fb8daffd640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e4156550 0x7fb8e414aea0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb8d40011e0 tx=0x7fb8d400c7f0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.325+0000 7fb8daffd640 1 -- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e4156550 msgr2=0x7fb8e414aea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.325+0000 7fb8daffd640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e4156550 0x7fb8e414aea0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.325+0000 7fb8eaae0640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb8c806b250 0x7fb8c806d710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc035000 tx=0x7fb8dc030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.332+0000 7fb8ec543640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e414cf30 0x7fb8e40ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.332+0000 7fb8ec543640 1 -- 172.21.15.98:0/162703949 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e414b750 con 0x7fb8e414cf30 2023-11-28T17:48:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.333+0000 7fb8e9ade640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e414cf30 0x7fb8e40ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.334+0000 7fb8e9ade640 1 -- 172.21.15.98:0/162703949 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb8e415a590 con 0x7fb8e414cf30 2023-11-28T17:48:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.334+0000 7fb8e9ade640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e414cf30 0x7fb8e40ca450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4007fd0 tx=0x7fb8d4001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.335+0000 7fb8daffd640 1 -- 172.21.15.98:0/162703949 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d401f070 con 0x7fb8e414cf30 2023-11-28T17:48:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.335+0000 7fb8daffd640 1 -- 172.21.15.98:0/162703949 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4794 (secure 0 0 0) 0x7fb8d4012400 con 0x7fb8e414cf30 2023-11-28T17:48:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.335+0000 7fb8daffd640 1 -- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e414cf30 msgr2=0x7fb8e40ca450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.335+0000 7fb8daffd640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e414cf30 0x7fb8e40ca450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4007fd0 tx=0x7fb8d4001080 comp rx=0 tx=0).stop 2023-11-28T17:48:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.336+0000 7fb8daffd640 1 -- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e414cf30 msgr2=0x7fb8e40ca450 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.336+0000 7fb8daffd640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e414cf30 0x7fb8e40ca450 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4007fd0 tx=0x7fb8d4001080 comp rx=0 tx=0).stop 2023-11-28T17:48:11.341 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:48:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:48:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:48:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:48:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 74, 2023-11-28T17:48:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:48:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:48:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: 1 2023-11-28T17:48:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:48:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:48:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:48:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:48:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:48:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:48:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:48:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:48:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:48:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:48:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:48:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:48:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:48:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:48:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:48:11.344 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:48:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:48:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:48:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:48:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:48:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:48:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:48:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:48:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:48:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:48:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:48:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:48:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:48:11.349 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:48:11.349 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:11.349 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:11.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:11.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:48:11.350 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:11.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:11.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:48:11.350 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:11.351 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.351 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:11.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:48:11.351 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:48:11.351 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:11.351 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:11.351 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:11.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.352 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:48:11.352 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:48:11.352 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:11.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:11.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:11.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:48:11.352 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:11.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:11.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:48:11.353 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:11.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.353 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:11.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:48:11.353 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:48:11.353 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:11.354 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:11.354 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:11.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.354 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:11.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.354 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:48:11.354 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:48:11.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:48:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:48:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:11.355 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:48:11.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.356 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:11.356 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:48:11.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.356 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:11.356 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:11.356 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:48:11.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.356 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:11.356 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:48:11.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:11.356 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:11.357 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:11.357 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:48:11.357 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:11.357 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:11.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:11.357 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:48:11.357 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:48:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.336+0000 7fb8ec543640 1 -- 172.21.15.98:0/162703949 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb8c806b250 msgr2=0x7fb8c806d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.336+0000 7fb8ec543640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb8c806b250 0x7fb8c806d710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc035000 tx=0x7fb8dc030000 comp rx=0 tx=0).stop 2023-11-28T17:48:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.336+0000 7fb8ec543640 1 -- 172.21.15.98:0/162703949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8e40c4380 msgr2=0x7fb8e414ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.336+0000 7fb8ec543640 1 --2- 172.21.15.98:0/162703949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8e40c4380 0x7fb8e414ba70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb8e000d800 tx=0x7fb8e000dcc0 comp rx=0 tx=0).stop 2023-11-28T17:48:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.336+0000 7fb8eaae0640 1 -- 172.21.15.98:0/162703949 reap_dead start 2023-11-28T17:48:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.336+0000 7fb8eaae0640 1 -- 172.21.15.98:0/162703949 reap_dead start 2023-11-28T17:48:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.336+0000 7fb8ec543640 1 -- 172.21.15.98:0/162703949 shutdown_connections 2023-11-28T17:48:11.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.336+0000 7fb8ec543640 1 -- 172.21.15.98:0/162703949 >> 172.21.15.98:0/162703949 conn(0x7fb8e406c0f0 msgr2=0x7fb8e40c7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:11.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.336+0000 7fb8ec543640 1 -- 172.21.15.98:0/162703949 shutdown_connections 2023-11-28T17:48:11.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:11.336+0000 7fb8ec543640 1 -- 172.21.15.98:0/162703949 wait complete. 2023-11-28T17:48:11.360 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:48:11.360 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-11-28T17:48:14.109 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~203s 2023-11-28T17:48:20.821 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~209s 2023-11-28T17:48:27.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~216s 2023-11-28T17:48:31.361 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2023-11-28T17:48:31.361 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2023-11-28T17:48:31.361 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-28T17:48:31.362 DEBUG:teuthology.orchestra.run.smithi113:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-11-28T17:48:31.371 INFO:tasks.ceph.mon.c:Started 2023-11-28T17:48:31.371 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:48:31.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:48:31.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.587+0000 7fceda3a0640 1 Processor -- start 2023-11-28T17:48:31.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.587+0000 7fceda3a0640 1 -- start start 2023-11-28T17:48:31.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.589+0000 7fceda3a0640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fced415a6d0 0x7fced40cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:31.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.589+0000 7fceda3a0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fced40cf7c0 0x7fced40b3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:31.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.589+0000 7fceda3a0640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced415ab10 con 0x7fced4153bb0 2023-11-28T17:48:31.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.589+0000 7fceda3a0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced415d120 con 0x7fced40cf7c0 2023-11-28T17:48:31.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.589+0000 7fceda3a0640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced415d330 con 0x7fced415a6d0 2023-11-28T17:48:31.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.589+0000 7fced9b9f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fced40cf7c0 0x7fced40b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:31.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.589+0000 7fced9b9f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fced40cf7c0 0x7fced40b3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38314/0 (socket says 172.21.15.98:38314) 2023-11-28T17:48:31.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.589+0000 7fced9b9f640 1 -- 172.21.15.98:0/3854112150 learned_addr learned my addr 172.21.15.98:0/3854112150 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:31.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.589+0000 7fced8b9d640 1 --2- 172.21.15.98:0/3854112150 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fced415a6d0 0x7fced40cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:31.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.589+0000 7fced9b9f640 1 -- 172.21.15.98:0/3854112150 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fced415a6d0 msgr2=0x7fced40cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:31.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.589+0000 7fced9b9f640 1 --2- 172.21.15.98:0/3854112150 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fced415a6d0 0x7fced40cf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:31.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.589+0000 7fced9b9f640 1 -- 172.21.15.98:0/3854112150 >> v1:172.21.15.113:6789/0 conn(0x7fced4153bb0 legacy=0x7fced40ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:31.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.589+0000 7fced9b9f640 1 -- 172.21.15.98:0/3854112150 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced415d610 con 0x7fced40cf7c0 2023-11-28T17:48:31.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.590+0000 7fced9b9f640 1 --2- 172.21.15.98:0/3854112150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced40cf7c0 0x7fced40b3760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcecc009870 tx=0x7fcecc02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2fac75610e4e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:31.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.591+0000 7fcecbfff640 1 -- 172.21.15.98:0/3854112150 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc02f840 con 0x7fced40cf7c0 2023-11-28T17:48:31.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.591+0000 7fcecbfff640 1 -- 172.21.15.98:0/3854112150 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcecc02f9a0 con 0x7fced40cf7c0 2023-11-28T17:48:31.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.591+0000 7fcecbfff640 1 -- 172.21.15.98:0/3854112150 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc02fd30 con 0x7fced40cf7c0 2023-11-28T17:48:31.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.591+0000 7fceda3a0640 1 -- 172.21.15.98:0/3854112150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced40cf7c0 msgr2=0x7fced40b3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:31.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.591+0000 7fceda3a0640 1 --2- 172.21.15.98:0/3854112150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced40cf7c0 0x7fced40b3760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcecc009870 tx=0x7fcecc02ba20 comp rx=0 tx=0).stop 2023-11-28T17:48:31.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.591+0000 7fceda3a0640 1 -- 172.21.15.98:0/3854112150 shutdown_connections 2023-11-28T17:48:31.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.591+0000 7fceda3a0640 1 --2- 172.21.15.98:0/3854112150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced40cf7c0 0x7fced40b3760 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:31.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.591+0000 7fceda3a0640 1 --2- 172.21.15.98:0/3854112150 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fced415a6d0 0x7fced40cf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:31.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.591+0000 7fceda3a0640 1 -- 172.21.15.98:0/3854112150 >> 172.21.15.98:0/3854112150 conn(0x7fced4071c00 msgr2=0x7fced4072010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:31.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.592+0000 7fceda3a0640 1 -- 172.21.15.98:0/3854112150 shutdown_connections 2023-11-28T17:48:31.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.592+0000 7fceda3a0640 1 -- 172.21.15.98:0/3854112150 wait complete. 2023-11-28T17:48:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.593+0000 7fceda3a0640 1 Processor -- start 2023-11-28T17:48:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.593+0000 7fceda3a0640 1 -- start start 2023-11-28T17:48:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.594+0000 7fceda3a0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fced40cf7c0 0x7fced41576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.594+0000 7fceda3a0640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fced4153bb0 0x7fced4152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.594+0000 7fceda3a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced415a6d0 0x7fced4152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.594+0000 7fceda3a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced41f02f0 con 0x7fced415a6d0 2023-11-28T17:48:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.594+0000 7fceda3a0640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced4159510 con 0x7fced40cf7c0 2023-11-28T17:48:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.594+0000 7fceda3a0640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced41597f0 con 0x7fced4153bb0 2023-11-28T17:48:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.594+0000 7fced9b9f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced415a6d0 0x7fced4152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:31.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fced9b9f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced415a6d0 0x7fced4152ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38322/0 (socket says 172.21.15.98:38322) 2023-11-28T17:48:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fced9b9f640 1 -- 172.21.15.98:0/3408711493 learned_addr learned my addr 172.21.15.98:0/3408711493 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fced939e640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fced40cf7c0 0x7fced41576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fced8b9d640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fced4153bb0 0x7fced4152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fced8b9d640 1 -- 172.21.15.98:0/3408711493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fced4153bb0 msgr2=0x7fced4152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-11-28T17:48:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fced8b9d640 1 -- 172.21.15.98:0/3408711493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fced4153bb0 msgr2=0x7fced4152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:48:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fced8b9d640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fced4153bb0 0x7fced4152590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:48:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fced8b9d640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fced4153bb0 0x7fced4152590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fced9b9f640 1 -- 172.21.15.98:0/3408711493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fced4153bb0 msgr2=0x7fced4152590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:48:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fced9b9f640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fced4153bb0 0x7fced4152590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fced9b9f640 1 -- 172.21.15.98:0/3408711493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fced40cf7c0 msgr2=0x7fced41576e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fced9b9f640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fced40cf7c0 0x7fced41576e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:31.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fced9b9f640 1 -- 172.21.15.98:0/3408711493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcecc0094d0 con 0x7fced415a6d0 2023-11-28T17:48:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fced9b9f640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced415a6d0 0x7fced4152ad0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcecc009870 tx=0x7fcecc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fcec9ffb640 1 -- 172.21.15.98:0/3408711493 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc037c40 con 0x7fced415a6d0 2023-11-28T17:48:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fcec9ffb640 1 -- 172.21.15.98:0/3408711493 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcecc037dc0 con 0x7fced415a6d0 2023-11-28T17:48:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.595+0000 7fcec9ffb640 1 -- 172.21.15.98:0/3408711493 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc041640 con 0x7fced415a6d0 2023-11-28T17:48:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.596+0000 7fceda3a0640 1 -- 172.21.15.98:0/3408711493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced41efca0 con 0x7fced415a6d0 2023-11-28T17:48:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.596+0000 7fceda3a0640 1 -- 172.21.15.98:0/3408711493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced41ef990 con 0x7fced415a6d0 2023-11-28T17:48:31.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.597+0000 7fceda3a0640 1 -- 172.21.15.98:0/3408711493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced415aea0 con 0x7fced415a6d0 2023-11-28T17:48:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.598+0000 7fcec9ffb640 1 -- 172.21.15.98:0/3408711493 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fcecc0417a0 con 0x7fced415a6d0 2023-11-28T17:48:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.599+0000 7fcec9ffb640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fceb806b2a0 0x7fceb806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.599+0000 7fcec9ffb640 1 -- 172.21.15.98:0/3408711493 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcecc03c070 con 0x7fced415a6d0 2023-11-28T17:48:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.599+0000 7fced939e640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fceb806b2a0 0x7fceb806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.600+0000 7fced939e640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fceb806b2a0 0x7fceb806d760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fced00012e0 tx=0x7fced0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:31.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.603+0000 7fcec9ffb640 1 -- 172.21.15.98:0/3408711493 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcecc0777a0 con 0x7fced415a6d0 2023-11-28T17:48:31.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.861+0000 7fceda3a0640 1 -- 172.21.15.98:0/3408711493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fced40cccd0 con 0x7fced415a6d0 2023-11-28T17:48:31.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.862+0000 7fcec9ffb640 1 -- 172.21.15.98:0/3408711493 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7fcecc07adc0 con 0x7fced415a6d0 2023-11-28T17:48:31.863 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":74,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:48:31.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.864+0000 7fceda3a0640 1 -- 172.21.15.98:0/3408711493 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fceb806b2a0 msgr2=0x7fceb806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:31.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.864+0000 7fceda3a0640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fceb806b2a0 0x7fceb806d760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fced00012e0 tx=0x7fced0009040 comp rx=0 tx=0).stop 2023-11-28T17:48:31.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.864+0000 7fceda3a0640 1 -- 172.21.15.98:0/3408711493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced415a6d0 msgr2=0x7fced4152ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:31.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.864+0000 7fceda3a0640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced415a6d0 0x7fced4152ad0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcecc009870 tx=0x7fcecc004730 comp rx=0 tx=0).stop 2023-11-28T17:48:31.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.864+0000 7fceda3a0640 1 -- 172.21.15.98:0/3408711493 shutdown_connections 2023-11-28T17:48:31.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.864+0000 7fceda3a0640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced415a6d0 0x7fced4152ad0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:31.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.865+0000 7fceda3a0640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fced4153bb0 0x7fced4152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:31.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.865+0000 7fceda3a0640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fceb806b2a0 0x7fceb806d760 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:31.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.865+0000 7fceda3a0640 1 --2- 172.21.15.98:0/3408711493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fced40cf7c0 0x7fced41576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:31.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.865+0000 7fceda3a0640 1 -- 172.21.15.98:0/3408711493 >> 172.21.15.98:0/3408711493 conn(0x7fced4071c00 msgr2=0x7fced40ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:31.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.865+0000 7fceda3a0640 1 -- 172.21.15.98:0/3408711493 shutdown_connections 2023-11-28T17:48:31.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:31.865+0000 7fceda3a0640 1 -- 172.21.15.98:0/3408711493 wait complete. 2023-11-28T17:48:34.348 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~223s 2023-11-28T17:48:34.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:48:35.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.107+0000 7fede97ec640 1 Processor -- start 2023-11-28T17:48:35.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.107+0000 7fede97ec640 1 -- start start 2023-11-28T17:48:35.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.108+0000 7fede97ec640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fede4151d80 0x7fede4157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:35.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.109+0000 7fede97ec640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fede415bc10 0x7fede40c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:35.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.109+0000 7fede97ec640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede415b150 con 0x7fede40c6140 2023-11-28T17:48:35.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.109+0000 7fede97ec640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede406e1f0 con 0x7fede415bc10 2023-11-28T17:48:35.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.109+0000 7fede97ec640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede406e4d0 con 0x7fede4151d80 2023-11-28T17:48:35.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.109+0000 7fede27fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fede415bc10 0x7fede40c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:35.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.109+0000 7fede27fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fede415bc10 0x7fede40c5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38334/0 (socket says 172.21.15.98:38334) 2023-11-28T17:48:35.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.109+0000 7fede27fc640 1 -- 172.21.15.98:0/3137386323 learned_addr learned my addr 172.21.15.98:0/3137386323 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:35.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.109+0000 7fede2ffd640 1 --2- 172.21.15.98:0/3137386323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fede4151d80 0x7fede4157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:35.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.109+0000 7fede27fc640 1 -- 172.21.15.98:0/3137386323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fede4151d80 msgr2=0x7fede4157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:35.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.109+0000 7fede27fc640 1 --2- 172.21.15.98:0/3137386323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fede4151d80 0x7fede4157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:35.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.109+0000 7fede27fc640 1 -- 172.21.15.98:0/3137386323 >> v1:172.21.15.113:6789/0 conn(0x7fede40c6140 legacy=0x7fede40c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:35.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.109+0000 7fede27fc640 1 -- 172.21.15.98:0/3137386323 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede406e7b0 con 0x7fede415bc10 2023-11-28T17:48:35.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.110+0000 7fede27fc640 1 --2- 172.21.15.98:0/3137386323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede415bc10 0x7fede40c5c00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009870 tx=0x7fedd802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=908dcee159dd9181 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:35.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.111+0000 7fede1ffb640 1 -- 172.21.15.98:0/3137386323 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8030840 con 0x7fede415bc10 2023-11-28T17:48:35.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.111+0000 7fede1ffb640 1 -- 172.21.15.98:0/3137386323 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd80309a0 con 0x7fede415bc10 2023-11-28T17:48:35.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.111+0000 7fede1ffb640 1 -- 172.21.15.98:0/3137386323 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8030d30 con 0x7fede415bc10 2023-11-28T17:48:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.111+0000 7fede97ec640 1 -- 172.21.15.98:0/3137386323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede415bc10 msgr2=0x7fede40c5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.111+0000 7fede97ec640 1 --2- 172.21.15.98:0/3137386323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede415bc10 0x7fede40c5c00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009870 tx=0x7fedd802b980 comp rx=0 tx=0).stop 2023-11-28T17:48:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.112+0000 7fede97ec640 1 -- 172.21.15.98:0/3137386323 shutdown_connections 2023-11-28T17:48:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.112+0000 7fede97ec640 1 --2- 172.21.15.98:0/3137386323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede415bc10 0x7fede40c5c00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.112+0000 7fede97ec640 1 --2- 172.21.15.98:0/3137386323 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fede4151d80 0x7fede4157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.112+0000 7fede97ec640 1 -- 172.21.15.98:0/3137386323 >> 172.21.15.98:0/3137386323 conn(0x7fede406a4d0 msgr2=0x7fede406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.112+0000 7fede97ec640 1 -- 172.21.15.98:0/3137386323 shutdown_connections 2023-11-28T17:48:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.112+0000 7fede97ec640 1 -- 172.21.15.98:0/3137386323 wait complete. 2023-11-28T17:48:35.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.113+0000 7fede97ec640 1 Processor -- start 2023-11-28T17:48:35.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.113+0000 7fede97ec640 1 -- start start 2023-11-28T17:48:35.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.114+0000 7fede97ec640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fede40c6140 0x7fede40c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:35.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.114+0000 7fede97ec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede4151d80 0x7fede4156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:35.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.114+0000 7fede97ec640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fede415bc10 0x7fede41f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:35.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.114+0000 7fede97ec640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede41f3be0 con 0x7fede4151d80 2023-11-28T17:48:35.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.114+0000 7fede97ec640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede4155f20 con 0x7fede415bc10 2023-11-28T17:48:35.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.114+0000 7fede97ec640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede4072640 con 0x7fede40c6140 2023-11-28T17:48:35.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.115+0000 7fede27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede4151d80 0x7fede4156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:35.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.115+0000 7fede27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede4151d80 0x7fede4156970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38344/0 (socket says 172.21.15.98:38344) 2023-11-28T17:48:35.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.115+0000 7fede27fc640 1 -- 172.21.15.98:0/298487384 learned_addr learned my addr 172.21.15.98:0/298487384 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:35.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.115+0000 7fede27fc640 1 -- 172.21.15.98:0/298487384 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fede40c6140 msgr2=0x7fede40c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:35.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.115+0000 7fede27fc640 1 --2- 172.21.15.98:0/298487384 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fede40c6140 0x7fede40c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:35.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.115+0000 7fede27fc640 1 -- 172.21.15.98:0/298487384 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fede415bc10 msgr2=0x7fede41f36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:35.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.115+0000 7fede27fc640 1 --2- 172.21.15.98:0/298487384 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fede415bc10 0x7fede41f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:35.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.115+0000 7fede27fc640 1 -- 172.21.15.98:0/298487384 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd80094d0 con 0x7fede4151d80 2023-11-28T17:48:35.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.115+0000 7fede27fc640 1 --2- 172.21.15.98:0/298487384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede4151d80 0x7fede4156970 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fedd80090b0 tx=0x7fedd8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:35.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.115+0000 7fedcb7fe640 1 -- 172.21.15.98:0/298487384 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd803e040 con 0x7fede4151d80 2023-11-28T17:48:35.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.116+0000 7fedcb7fe640 1 -- 172.21.15.98:0/298487384 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd8039c90 con 0x7fede4151d80 2023-11-28T17:48:35.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.116+0000 7fedcb7fe640 1 -- 172.21.15.98:0/298487384 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd80443f0 con 0x7fede4151d80 2023-11-28T17:48:35.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.116+0000 7fede97ec640 1 -- 172.21.15.98:0/298487384 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede40c39f0 con 0x7fede4151d80 2023-11-28T17:48:35.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.116+0000 7fede97ec640 1 -- 172.21.15.98:0/298487384 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede40c3fb0 con 0x7fede4151d80 2023-11-28T17:48:35.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.117+0000 7fede97ec640 1 -- 172.21.15.98:0/298487384 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede4067bf0 con 0x7fede4151d80 2023-11-28T17:48:35.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.118+0000 7fedcb7fe640 1 -- 172.21.15.98:0/298487384 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fedd8043390 con 0x7fede4151d80 2023-11-28T17:48:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.118+0000 7fedcb7fe640 1 --2- 172.21.15.98:0/298487384 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fedd006b2a0 0x7fedd006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.119+0000 7fedcb7fe640 1 -- 172.21.15.98:0/298487384 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fedd80abfa0 con 0x7fede4151d80 2023-11-28T17:48:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.119+0000 7fede2ffd640 1 --2- 172.21.15.98:0/298487384 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fedd006b2a0 0x7fedd006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:35.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.119+0000 7fede2ffd640 1 --2- 172.21.15.98:0/298487384 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fedd006b2a0 0x7fedd006d760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fedd40098f0 tx=0x7fedd4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:35.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.123+0000 7fedcb7fe640 1 -- 172.21.15.98:0/298487384 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fedd80dee90 con 0x7fede4151d80 2023-11-28T17:48:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.375+0000 7fede97ec640 1 -- 172.21.15.98:0/298487384 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fede406b2d0 con 0x7fede4151d80 2023-11-28T17:48:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.375+0000 7fedcb7fe640 1 -- 172.21.15.98:0/298487384 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7fedd8076ff0 con 0x7fede4151d80 2023-11-28T17:48:35.377 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":74,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:48:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.378+0000 7fede97ec640 1 -- 172.21.15.98:0/298487384 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fedd006b2a0 msgr2=0x7fedd006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.378+0000 7fede97ec640 1 --2- 172.21.15.98:0/298487384 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fedd006b2a0 0x7fedd006d760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fedd40098f0 tx=0x7fedd4008040 comp rx=0 tx=0).stop 2023-11-28T17:48:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.378+0000 7fede97ec640 1 -- 172.21.15.98:0/298487384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede4151d80 msgr2=0x7fede4156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.378+0000 7fede97ec640 1 --2- 172.21.15.98:0/298487384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede4151d80 0x7fede4156970 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fedd80090b0 tx=0x7fedd8004730 comp rx=0 tx=0).stop 2023-11-28T17:48:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.378+0000 7fede97ec640 1 -- 172.21.15.98:0/298487384 shutdown_connections 2023-11-28T17:48:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.378+0000 7fede97ec640 1 --2- 172.21.15.98:0/298487384 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fedd006b2a0 0x7fedd006d760 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.378+0000 7fede97ec640 1 --2- 172.21.15.98:0/298487384 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fede415bc10 0x7fede41f36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.378+0000 7fede97ec640 1 --2- 172.21.15.98:0/298487384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fede4151d80 0x7fede4156970 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.378+0000 7fede97ec640 1 --2- 172.21.15.98:0/298487384 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fede40c6140 0x7fede40c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.378+0000 7fede97ec640 1 -- 172.21.15.98:0/298487384 >> 172.21.15.98:0/298487384 conn(0x7fede406a4d0 msgr2=0x7fede40cb330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.378+0000 7fede97ec640 1 -- 172.21.15.98:0/298487384 shutdown_connections 2023-11-28T17:48:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:35.378+0000 7fede97ec640 1 -- 172.21.15.98:0/298487384 wait complete. 2023-11-28T17:48:38.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:48:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.616+0000 7faba6373640 1 Processor -- start 2023-11-28T17:48:38.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.616+0000 7faba6373640 1 -- start start 2023-11-28T17:48:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.618+0000 7faba6373640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faba00c5c00 0x7faba00c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.618+0000 7faba6373640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faba00c4a20 0x7faba00c6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.618+0000 7faba6373640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba0157850 con 0x7faba0150c00 2023-11-28T17:48:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.618+0000 7faba6373640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba00c2e10 con 0x7faba00c4a20 2023-11-28T17:48:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.618+0000 7faba6373640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba0156980 con 0x7faba00c5c00 2023-11-28T17:48:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.618+0000 7fab9ffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7faba0150c00 0x7faba00c2700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33742/0 (socket says 172.21.15.98:33742) 2023-11-28T17:48:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.618+0000 7fab9ffff640 1 -- 172.21.15.98:0/2671887627 learned_addr learned my addr 172.21.15.98:0/2671887627 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:48:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.618+0000 7faba4910640 1 --2- 172.21.15.98:0/2671887627 >> v2:172.21.15.98:3300/0 conn(0x7faba00c4a20 0x7faba00c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.619+0000 7fab97fff640 1 --2- 172.21.15.98:0/2671887627 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faba00c5c00 0x7faba00c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.619+0000 7fab9f7fe640 1 -- 172.21.15.98:0/2671887627 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643968228 0 0) 0x7faba0157850 con 0x7faba0150c00 2023-11-28T17:48:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.619+0000 7fab9f7fe640 1 -- 172.21.15.98:0/2671887627 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab8c003520 con 0x7faba0150c00 2023-11-28T17:48:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.619+0000 7faba4910640 1 -- 172.21.15.98:0/2671887627 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faba00c5c00 msgr2=0x7faba00c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.619+0000 7faba4910640 1 --2- 172.21.15.98:0/2671887627 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faba00c5c00 0x7faba00c44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.619+0000 7faba4910640 1 -- 172.21.15.98:0/2671887627 >> v1:172.21.15.98:6789/0 conn(0x7faba0150c00 legacy=0x7faba00c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.619+0000 7faba4910640 1 -- 172.21.15.98:0/2671887627 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba0156c40 con 0x7faba00c4a20 2023-11-28T17:48:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.619+0000 7fab97fff640 1 --2- 172.21.15.98:0/2671887627 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faba00c5c00 0x7faba00c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:48:38.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.620+0000 7faba4910640 1 --2- 172.21.15.98:0/2671887627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba00c4a20 0x7faba00c6600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fab98009d90 tx=0x7fab9802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eaef7c80760b197 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.620+0000 7fab9f7fe640 1 -- 172.21.15.98:0/2671887627 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9802cc50 con 0x7faba00c4a20 2023-11-28T17:48:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.620+0000 7fab9f7fe640 1 -- 172.21.15.98:0/2671887627 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9802cdb0 con 0x7faba00c4a20 2023-11-28T17:48:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.620+0000 7fab9f7fe640 1 -- 172.21.15.98:0/2671887627 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98034660 con 0x7faba00c4a20 2023-11-28T17:48:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.621+0000 7faba6373640 1 -- 172.21.15.98:0/2671887627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba00c4a20 msgr2=0x7faba00c6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.621+0000 7faba6373640 1 --2- 172.21.15.98:0/2671887627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba00c4a20 0x7faba00c6600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fab98009d90 tx=0x7fab9802bea0 comp rx=0 tx=0).stop 2023-11-28T17:48:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.621+0000 7faba6373640 1 -- 172.21.15.98:0/2671887627 shutdown_connections 2023-11-28T17:48:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.621+0000 7faba6373640 1 --2- 172.21.15.98:0/2671887627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba00c4a20 0x7faba00c6600 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.621+0000 7faba6373640 1 --2- 172.21.15.98:0/2671887627 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faba00c5c00 0x7faba00c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.621+0000 7faba6373640 1 -- 172.21.15.98:0/2671887627 >> 172.21.15.98:0/2671887627 conn(0x7faba006a4d0 msgr2=0x7faba006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.621+0000 7faba6373640 1 -- 172.21.15.98:0/2671887627 shutdown_connections 2023-11-28T17:48:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.621+0000 7faba6373640 1 -- 172.21.15.98:0/2671887627 wait complete. 2023-11-28T17:48:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.622+0000 7faba6373640 1 Processor -- start 2023-11-28T17:48:38.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.623+0000 7faba6373640 1 -- start start 2023-11-28T17:48:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.623+0000 7faba6373640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba00c4a20 0x7faba0152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.623+0000 7faba6373640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faba00c5c00 0x7faba00c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.623+0000 7faba6373640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faba0150c00 0x7faba01e10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.623+0000 7faba6373640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba01efff0 con 0x7faba00c4a20 2023-11-28T17:48:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.623+0000 7faba6373640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba00c8610 con 0x7faba0150c00 2023-11-28T17:48:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.623+0000 7faba6373640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba01f0480 con 0x7faba00c5c00 2023-11-28T17:48:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.624+0000 7fab9ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba00c4a20 0x7faba0152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.624+0000 7fab9ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba00c4a20 0x7faba0152300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38360/0 (socket says 172.21.15.98:38360) 2023-11-28T17:48:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.624+0000 7fab9ffff640 1 -- 172.21.15.98:0/1529447615 learned_addr learned my addr 172.21.15.98:0/1529447615 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.624+0000 7fab97fff640 1 --2- 172.21.15.98:0/1529447615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faba00c5c00 0x7faba00c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.624+0000 7fab9ffff640 1 -- 172.21.15.98:0/1529447615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faba00c5c00 msgr2=0x7faba00c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.624+0000 7fab9ffff640 1 --2- 172.21.15.98:0/1529447615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faba00c5c00 0x7faba00c7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.624+0000 7fab9ffff640 1 -- 172.21.15.98:0/1529447615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faba0150c00 msgr2=0x7faba01e10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.624+0000 7fab9ffff640 1 --2- 172.21.15.98:0/1529447615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faba0150c00 0x7faba01e10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.624+0000 7fab9ffff640 1 -- 172.21.15.98:0/1529447615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab98009a70 con 0x7faba00c4a20 2023-11-28T17:48:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.624+0000 7fab9ffff640 1 --2- 172.21.15.98:0/1529447615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba00c4a20 0x7faba0152300 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fab9000c390 tx=0x7fab9000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.625+0000 7fab9d7fa640 1 -- 172.21.15.98:0/1529447615 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab90005ce0 con 0x7faba00c4a20 2023-11-28T17:48:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.625+0000 7fab9d7fa640 1 -- 172.21.15.98:0/1529447615 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab90005e40 con 0x7faba00c4a20 2023-11-28T17:48:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.625+0000 7fab9d7fa640 1 -- 172.21.15.98:0/1529447615 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab900072f0 con 0x7faba00c4a20 2023-11-28T17:48:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.625+0000 7faba6373640 1 -- 172.21.15.98:0/1529447615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba00c8330 con 0x7faba00c4a20 2023-11-28T17:48:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.625+0000 7faba6373640 1 -- 172.21.15.98:0/1529447615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba01ef9a0 con 0x7faba00c4a20 2023-11-28T17:48:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.626+0000 7faba6373640 1 -- 172.21.15.98:0/1529447615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba0067bf0 con 0x7faba00c4a20 2023-11-28T17:48:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.627+0000 7fab9d7fa640 1 -- 172.21.15.98:0/1529447615 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fab90007450 con 0x7faba00c4a20 2023-11-28T17:48:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.628+0000 7fab9d7fa640 1 --2- 172.21.15.98:0/1529447615 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fab8006b2d0 0x7fab8006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.628+0000 7fab9d7fa640 1 -- 172.21.15.98:0/1529447615 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fab90089230 con 0x7faba00c4a20 2023-11-28T17:48:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.628+0000 7fab97fff640 1 --2- 172.21.15.98:0/1529447615 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fab8006b2d0 0x7fab8006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.629+0000 7fab97fff640 1 --2- 172.21.15.98:0/1529447615 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fab8006b2d0 0x7fab8006d790 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faba014d860 tx=0x7fab88008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.632+0000 7fab9d7fa640 1 -- 172.21.15.98:0/1529447615 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab90054020 con 0x7faba00c4a20 2023-11-28T17:48:38.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.894+0000 7faba6373640 1 -- 172.21.15.98:0/1529447615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7faba0074f60 con 0x7faba00c4a20 2023-11-28T17:48:38.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.895+0000 7fab9d7fa640 1 -- 172.21.15.98:0/1529447615 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7fab90057640 con 0x7faba00c4a20 2023-11-28T17:48:38.896 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:48:38.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.898+0000 7faba6373640 1 -- 172.21.15.98:0/1529447615 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fab8006b2d0 msgr2=0x7fab8006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:38.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.898+0000 7faba6373640 1 --2- 172.21.15.98:0/1529447615 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fab8006b2d0 0x7fab8006d790 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faba014d860 tx=0x7fab88008040 comp rx=0 tx=0).stop 2023-11-28T17:48:38.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.898+0000 7faba6373640 1 -- 172.21.15.98:0/1529447615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba00c4a20 msgr2=0x7faba0152300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:38.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.898+0000 7faba6373640 1 --2- 172.21.15.98:0/1529447615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba00c4a20 0x7faba0152300 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fab9000c390 tx=0x7fab9000c850 comp rx=0 tx=0).stop 2023-11-28T17:48:38.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.898+0000 7faba6373640 1 -- 172.21.15.98:0/1529447615 shutdown_connections 2023-11-28T17:48:38.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.898+0000 7faba6373640 1 --2- 172.21.15.98:0/1529447615 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fab8006b2d0 0x7fab8006d790 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:38.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.898+0000 7faba6373640 1 --2- 172.21.15.98:0/1529447615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faba0150c00 0x7faba01e10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:38.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.898+0000 7faba6373640 1 --2- 172.21.15.98:0/1529447615 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7faba00c5c00 0x7faba00c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:38.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.898+0000 7faba6373640 1 --2- 172.21.15.98:0/1529447615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba00c4a20 0x7faba0152300 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:38.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.898+0000 7faba6373640 1 -- 172.21.15.98:0/1529447615 >> 172.21.15.98:0/1529447615 conn(0x7faba006a4d0 msgr2=0x7faba0155970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:38.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.898+0000 7faba6373640 1 -- 172.21.15.98:0/1529447615 shutdown_connections 2023-11-28T17:48:38.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:38.898+0000 7faba6373640 1 -- 172.21.15.98:0/1529447615 wait complete. 2023-11-28T17:48:38.917 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:48:38.917 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:48:38.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:48:39.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.131+0000 7fdbed333640 1 Processor -- start 2023-11-28T17:48:39.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.132+0000 7fdbed333640 1 -- start start 2023-11-28T17:48:39.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.133+0000 7fdbed333640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdbe80c4380 0x7fdbe80c2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.133+0000 7fdbed333640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdbe80c4760 0x7fdbe80c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.133+0000 7fdbed333640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe80c6560 con 0x7fdbe815a0f0 2023-11-28T17:48:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.133+0000 7fdbed333640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe80c2a60 con 0x7fdbe80c4380 2023-11-28T17:48:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.133+0000 7fdbed333640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe8156980 con 0x7fdbe80c4760 2023-11-28T17:48:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.134+0000 7fdbe6ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdbe80c4380 0x7fdbe80c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.134+0000 7fdbe77fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdbe815a0f0 0x7fdbe815a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33758/0 (socket says 172.21.15.98:33758) 2023-11-28T17:48:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.134+0000 7fdbe6ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdbe80c4380 0x7fdbe80c2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38368/0 (socket says 172.21.15.98:38368) 2023-11-28T17:48:39.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.134+0000 7fdbe77fe640 1 -- 172.21.15.98:0/3197707374 learned_addr learned my addr 172.21.15.98:0/3197707374 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:48:39.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.134+0000 7fdbe5ffb640 1 -- 172.21.15.98:0/3197707374 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624478268 0 0) 0x7fdbe80c6560 con 0x7fdbe815a0f0 2023-11-28T17:48:39.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.134+0000 7fdbe6ffd640 1 -- 172.21.15.98:0/3197707374 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdbe80c4760 msgr2=0x7fdbe80c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:39.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.134+0000 7fdbe67fc640 1 --2- 172.21.15.98:0/3197707374 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdbe80c4760 0x7fdbe80c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.134+0000 7fdbe6ffd640 1 --2- 172.21.15.98:0/3197707374 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdbe80c4760 0x7fdbe80c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.134+0000 7fdbe6ffd640 1 -- 172.21.15.98:0/3197707374 >> v1:172.21.15.98:6789/0 conn(0x7fdbe815a0f0 legacy=0x7fdbe815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.134+0000 7fdbe6ffd640 1 -- 172.21.15.98:0/3197707374 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe8156c40 con 0x7fdbe80c4380 2023-11-28T17:48:39.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.134+0000 7fdbe67fc640 1 --2- 172.21.15.98:0/3197707374 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdbe80c4760 0x7fdbe80c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:48:39.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.136+0000 7fdbe6ffd640 1 --2- 172.21.15.98:0/3197707374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe80c4380 0x7fdbe80c2520 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009d90 tx=0x7fdbd802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162b176d7572268b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.136+0000 7fdbe5ffb640 1 -- 172.21.15.98:0/3197707374 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd802cce0 con 0x7fdbe80c4380 2023-11-28T17:48:39.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.136+0000 7fdbe5ffb640 1 -- 172.21.15.98:0/3197707374 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd802ce40 con 0x7fdbe80c4380 2023-11-28T17:48:39.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.136+0000 7fdbe5ffb640 1 -- 172.21.15.98:0/3197707374 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8034660 con 0x7fdbe80c4380 2023-11-28T17:48:39.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.136+0000 7fdbed333640 1 -- 172.21.15.98:0/3197707374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe80c4380 msgr2=0x7fdbe80c2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.136+0000 7fdbed333640 1 --2- 172.21.15.98:0/3197707374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe80c4380 0x7fdbe80c2520 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009d90 tx=0x7fdbd802bf30 comp rx=0 tx=0).stop 2023-11-28T17:48:39.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.137+0000 7fdbed333640 1 -- 172.21.15.98:0/3197707374 shutdown_connections 2023-11-28T17:48:39.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.137+0000 7fdbed333640 1 --2- 172.21.15.98:0/3197707374 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdbe80c4760 0x7fdbe80c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.137+0000 7fdbed333640 1 --2- 172.21.15.98:0/3197707374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe80c4380 0x7fdbe80c2520 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.137+0000 7fdbed333640 1 -- 172.21.15.98:0/3197707374 >> 172.21.15.98:0/3197707374 conn(0x7fdbe806c0f0 msgr2=0x7fdbe806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:39.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.137+0000 7fdbed333640 1 -- 172.21.15.98:0/3197707374 shutdown_connections 2023-11-28T17:48:39.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.137+0000 7fdbed333640 1 -- 172.21.15.98:0/3197707374 wait complete. 2023-11-28T17:48:39.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.139+0000 7fdbed333640 1 Processor -- start 2023-11-28T17:48:39.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.139+0000 7fdbed333640 1 -- start start 2023-11-28T17:48:39.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.140+0000 7fdbed333640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdbe80c4380 0x7fdbe8155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.140+0000 7fdbed333640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbe80c4760 0x7fdbe8145290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.140+0000 7fdbed333640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe815a0f0 0x7fdbe81f0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.140+0000 7fdbed333640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe8154c20 con 0x7fdbe815a0f0 2023-11-28T17:48:39.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.140+0000 7fdbed333640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe81f2610 con 0x7fdbe80c4760 2023-11-28T17:48:39.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.140+0000 7fdbed333640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe806afc0 con 0x7fdbe80c4380 2023-11-28T17:48:39.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.140+0000 7fdbe77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe815a0f0 0x7fdbe81f0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.140+0000 7fdbe77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe815a0f0 0x7fdbe81f0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38376/0 (socket says 172.21.15.98:38376) 2023-11-28T17:48:39.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.140+0000 7fdbe77fe640 1 -- 172.21.15.98:0/178395084 learned_addr learned my addr 172.21.15.98:0/178395084 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:39.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.140+0000 7fdbe6ffd640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdbe80c4380 0x7fdbe8155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.141+0000 7fdbe77fe640 1 -- 172.21.15.98:0/178395084 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdbe80c4380 msgr2=0x7fdbe8155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.141+0000 7fdbe77fe640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdbe80c4380 0x7fdbe8155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.141+0000 7fdbe77fe640 1 -- 172.21.15.98:0/178395084 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbe80c4760 msgr2=0x7fdbe8145290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.141+0000 7fdbe77fe640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbe80c4760 0x7fdbe8145290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.141+0000 7fdbe77fe640 1 -- 172.21.15.98:0/178395084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd8009a70 con 0x7fdbe815a0f0 2023-11-28T17:48:39.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.141+0000 7fdbe77fe640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe815a0f0 0x7fdbe81f0510 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0001eb0 tx=0x7fdbe000c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.141+0000 7fdbdf7fe640 1 -- 172.21.15.98:0/178395084 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe00074e0 con 0x7fdbe815a0f0 2023-11-28T17:48:39.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.141+0000 7fdbdf7fe640 1 -- 172.21.15.98:0/178395084 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe0007640 con 0x7fdbe815a0f0 2023-11-28T17:48:39.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.141+0000 7fdbdf7fe640 1 -- 172.21.15.98:0/178395084 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0004a00 con 0x7fdbe815a0f0 2023-11-28T17:48:39.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.141+0000 7fdbed333640 1 -- 172.21.15.98:0/178395084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe8156380 con 0x7fdbe815a0f0 2023-11-28T17:48:39.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.141+0000 7fdbed333640 1 -- 172.21.15.98:0/178395084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe81f28b0 con 0x7fdbe815a0f0 2023-11-28T17:48:39.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.142+0000 7fdbed333640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe81f2ad0 0x7fdbe8141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.142+0000 7fdbed333640 1 -- 172.21.15.98:0/178395084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe80744d0 con 0x7fdbe81f2ad0 2023-11-28T17:48:39.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.143+0000 7fdbe6ffd640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe81f2ad0 0x7fdbe8141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.143+0000 7fdbe6ffd640 1 -- 172.21.15.98:0/178395084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdbe81574e0 con 0x7fdbe81f2ad0 2023-11-28T17:48:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.143+0000 7fdbe6ffd640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe81f2ad0 0x7fdbe8141be0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdbe80c8f70 tx=0x7fdbd8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.144+0000 7fdbdf7fe640 1 -- 172.21.15.98:0/178395084 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd803e850 con 0x7fdbe81f2ad0 2023-11-28T17:48:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.144+0000 7fdbdf7fe640 1 -- 172.21.15.98:0/178395084 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fdbd802cc80 con 0x7fdbe81f2ad0 2023-11-28T17:48:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.144+0000 7fdbdf7fe640 1 -- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe81f2ad0 msgr2=0x7fdbe8141be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.144+0000 7fdbdf7fe640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe81f2ad0 0x7fdbe8141be0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdbe80c8f70 tx=0x7fdbd8004a40 comp rx=0 tx=0).stop 2023-11-28T17:48:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.144+0000 7fdbdf7fe640 1 -- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe81f2ad0 msgr2=0x7fdbe8141be0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.144+0000 7fdbdf7fe640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe81f2ad0 0x7fdbe8141be0 secure :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdbe80c8f70 tx=0x7fdbd8004a40 comp rx=0 tx=0).stop 2023-11-28T17:48:39.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.144+0000 7fdbdf7fe640 1 -- 172.21.15.98:0/178395084 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fdbe0004b60 con 0x7fdbe815a0f0 2023-11-28T17:48:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.145+0000 7fdbdf7fe640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdbc806b020 0x7fdbc806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.145+0000 7fdbdf7fe640 1 -- 172.21.15.98:0/178395084 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdbe0089b40 con 0x7fdbe815a0f0 2023-11-28T17:48:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.145+0000 7fdbe6ffd640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdbc806b020 0x7fdbc806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.146+0000 7fdbe6ffd640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdbc806b020 0x7fdbc806d4e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdbe80c7a70 tx=0x7fdbd8034f90 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.151+0000 7fdbed333640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe8142120 0x7fdbe8142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.151+0000 7fdbed333640 1 -- 172.21.15.98:0/178395084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe8157200 con 0x7fdbe8142120 2023-11-28T17:48:39.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.152+0000 7fdbe67fc640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe8142120 0x7fdbe8142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.152+0000 7fdbe67fc640 1 -- 172.21.15.98:0/178395084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdbe81577c0 con 0x7fdbe8142120 2023-11-28T17:48:39.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.152+0000 7fdbe67fc640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe8142120 0x7fdbe8142530 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdbe806e7b0 tx=0x7fdbd000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.152+0000 7fdbdf7fe640 1 -- 172.21.15.98:0/178395084 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0007890 con 0x7fdbe8142120 2023-11-28T17:48:39.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.153+0000 7fdbdf7fe640 1 -- 172.21.15.98:0/178395084 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4977 (secure 0 0 0) 0x7fdbd000cec0 con 0x7fdbe8142120 2023-11-28T17:48:39.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.153+0000 7fdbdf7fe640 1 -- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe8142120 msgr2=0x7fdbe8142530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.153+0000 7fdbdf7fe640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe8142120 0x7fdbe8142530 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdbe806e7b0 tx=0x7fdbd000cb80 comp rx=0 tx=0).stop 2023-11-28T17:48:39.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.153+0000 7fdbdf7fe640 1 -- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe8142120 msgr2=0x7fdbe8142530 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:39.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.153+0000 7fdbdf7fe640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe8142120 0x7fdbe8142530 secure :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdbe806e7b0 tx=0x7fdbd000cb80 comp rx=0 tx=0).stop 2023-11-28T17:48:39.154 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:48:39.154 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:48:39.155 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:48:39.155 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:48:39.155 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2023-11-28T17:48:39.155 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:48:39.155 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:48:39.155 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:48:39.155 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:48:39.155 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.155 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2023-11-28T17:48:39.155 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:39.155 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:48:39.155 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:48:39.156 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:39.156 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:39.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:39.156 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:39.156 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:39.156 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:39.156 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:39.156 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:39.156 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:39.156 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:39.156 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.156 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:48:39.157 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:48:39.157 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:39.157 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:39.157 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:39.157 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:39.157 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:39.157 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:39.157 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:39.157 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:39.157 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:39.157 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:39.157 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.158 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:48:39.158 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:48:39.158 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:48:39.158 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:48:39.158 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:48:39.158 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:48:39.158 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:39.158 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:39.158 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:48:39.158 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:48:39.158 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:48:39.158 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:48:39.159 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:48:39.159 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:48:39.159 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:48:39.159 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:39.159 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:48:39.159 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:39.159 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:39.159 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:39.159 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:39.159 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:39.159 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:39.159 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:39.160 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:39.160 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:39.160 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:39.160 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.160 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:48:39.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.160 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:48:39.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.160 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:48:39.160 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:48:39.160 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:39.160 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:39.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:39.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:48:39.161 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:39.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:48:39.161 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.161 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.161 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:48:39.162 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:48:39.162 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:39.162 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:39.162 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:39.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.162 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:48:39.162 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:48:39.162 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:39.162 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:39.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:39.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:48:39.163 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:39.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:48:39.163 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.163 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.163 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:48:39.163 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:48:39.163 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:39.164 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:39.164 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:39.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.164 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:48:39.164 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:48:39.164 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:39.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:39.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:39.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:48:39.164 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:39.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:48:39.165 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.165 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.165 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:48:39.165 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:48:39.165 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:39.165 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:39.165 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:39.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.166 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.166 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:48:39.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:48:39.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.166 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:39.166 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:39.166 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:48:39.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.166 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:48:39.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.167 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:39.167 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:39.167 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:48:39.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.167 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.167 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:48:39.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.167 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:39.167 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:39.167 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 6 2023-11-28T17:48:39.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.167 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.168 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:48:39.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.168 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:39.168 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:39.168 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:48:39.168 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.168 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.168 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:48:39.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.168 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:39.168 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:39.168 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:48:39.168 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.169 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.169 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:48:39.169 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:48:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.153+0000 7fdbed333640 1 -- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdbc806b020 msgr2=0x7fdbc806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.153+0000 7fdbed333640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdbc806b020 0x7fdbc806d4e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdbe80c7a70 tx=0x7fdbd8034f90 comp rx=0 tx=0).stop 2023-11-28T17:48:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.153+0000 7fdbed333640 1 -- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe815a0f0 msgr2=0x7fdbe81f0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.153+0000 7fdbed333640 1 --2- 172.21.15.98:0/178395084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbe815a0f0 0x7fdbe81f0510 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0001eb0 tx=0x7fdbe000c870 comp rx=0 tx=0).stop 2023-11-28T17:48:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.153+0000 7fdbe77fe640 1 -- 172.21.15.98:0/178395084 reap_dead start 2023-11-28T17:48:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.153+0000 7fdbe77fe640 1 -- 172.21.15.98:0/178395084 reap_dead start 2023-11-28T17:48:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.153+0000 7fdbed333640 1 -- 172.21.15.98:0/178395084 shutdown_connections 2023-11-28T17:48:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.153+0000 7fdbed333640 1 -- 172.21.15.98:0/178395084 >> 172.21.15.98:0/178395084 conn(0x7fdbe806c0f0 msgr2=0x7fdbe806dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.153+0000 7fdbed333640 1 -- 172.21.15.98:0/178395084 shutdown_connections 2023-11-28T17:48:39.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.153+0000 7fdbed333640 1 -- 172.21.15.98:0/178395084 wait complete. 2023-11-28T17:48:39.171 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:48:39.171 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:48:39.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:48:39.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.366+0000 7f4309db0640 1 Processor -- start 2023-11-28T17:48:39.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.366+0000 7f4309db0640 1 -- start start 2023-11-28T17:48:39.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.368+0000 7f4309db0640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4304151be0 0x7f43040c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.368+0000 7f4309db0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43041584d0 0x7f430406dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.368+0000 7f4309db0640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43040c8150 con 0x7f43040c5a60 2023-11-28T17:48:39.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.368+0000 7f4309db0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43040c8850 con 0x7f43041584d0 2023-11-28T17:48:39.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.368+0000 7f4309db0640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43040c5f50 con 0x7f4304151be0 2023-11-28T17:48:39.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.368+0000 7f4303fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43041584d0 0x7f430406dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.368+0000 7f4303fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43041584d0 0x7f430406dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38420/0 (socket says 172.21.15.98:38420) 2023-11-28T17:48:39.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.368+0000 7f4303fff640 1 -- 172.21.15.98:0/139365158 learned_addr learned my addr 172.21.15.98:0/139365158 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:39.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.369+0000 7f4303fff640 1 -- 172.21.15.98:0/139365158 >> v2:172.21.15.113:3300/0 conn(0x7f4304151be0 msgr2=0x7f43040c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:39.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.369+0000 7f4303fff640 1 --2- 172.21.15.98:0/139365158 >> v2:172.21.15.113:3300/0 conn(0x7f4304151be0 0x7f43040c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.369+0000 7f4303fff640 1 -- 172.21.15.98:0/139365158 >> v1:172.21.15.113:6789/0 conn(0x7f43040c5a60 legacy=0x7f43040c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:39.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.369+0000 7f4303fff640 1 -- 172.21.15.98:0/139365158 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43040c7b90 con 0x7f43041584d0 2023-11-28T17:48:39.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.370+0000 7f4303fff640 1 --2- 172.21.15.98:0/139365158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43041584d0 0x7f430406dd80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f43040c5330 tx=0x7f42fc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ecbfedf4a8b66f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.370+0000 7f4302ffd640 1 -- 172.21.15.98:0/139365158 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc02ca80 con 0x7f43041584d0 2023-11-28T17:48:39.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.370+0000 7f4302ffd640 1 -- 172.21.15.98:0/139365158 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42fc02cbe0 con 0x7f43041584d0 2023-11-28T17:48:39.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.370+0000 7f4302ffd640 1 -- 172.21.15.98:0/139365158 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc0343f0 con 0x7f43041584d0 2023-11-28T17:48:39.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.371+0000 7f4309db0640 1 -- 172.21.15.98:0/139365158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43041584d0 msgr2=0x7f430406dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.371+0000 7f4309db0640 1 --2- 172.21.15.98:0/139365158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43041584d0 0x7f430406dd80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f43040c5330 tx=0x7f42fc02bee0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.371+0000 7f4309db0640 1 -- 172.21.15.98:0/139365158 shutdown_connections 2023-11-28T17:48:39.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.371+0000 7f4309db0640 1 --2- 172.21.15.98:0/139365158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43041584d0 0x7f430406dd80 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.371+0000 7f4309db0640 1 --2- 172.21.15.98:0/139365158 >> v2:172.21.15.113:3300/0 conn(0x7f4304151be0 0x7f43040c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.371+0000 7f4309db0640 1 -- 172.21.15.98:0/139365158 >> 172.21.15.98:0/139365158 conn(0x7f430406c0f0 msgr2=0x7f430406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:39.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.371+0000 7f4309db0640 1 -- 172.21.15.98:0/139365158 shutdown_connections 2023-11-28T17:48:39.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.371+0000 7f4309db0640 1 -- 172.21.15.98:0/139365158 wait complete. 2023-11-28T17:48:39.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.372+0000 7f4309db0640 1 Processor -- start 2023-11-28T17:48:39.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.373+0000 7f4309db0640 1 -- start start 2023-11-28T17:48:39.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.373+0000 7f4309db0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43040c5a60 0x7f43040ca210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.373+0000 7f4309db0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4304151be0 0x7f4304145f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.373+0000 7f4309db0640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43041584d0 0x7f43041464b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.373+0000 7f4309db0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43041ee1f0 con 0x7f4304151be0 2023-11-28T17:48:39.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.373+0000 7f4309db0640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4304159110 con 0x7f43040c5a60 2023-11-28T17:48:39.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.373+0000 7f4309db0640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4304159c50 con 0x7f43041584d0 2023-11-28T17:48:39.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.374+0000 7f42fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4304151be0 0x7f4304145f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.374+0000 7f42fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4304151be0 0x7f4304145f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38426/0 (socket says 172.21.15.98:38426) 2023-11-28T17:48:39.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.374+0000 7f42fbfff640 1 -- 172.21.15.98:0/3392422408 learned_addr learned my addr 172.21.15.98:0/3392422408 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:39.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.374+0000 7f43037fe640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43040c5a60 0x7f43040ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.374+0000 7f4303fff640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43041584d0 0x7f43041464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.374+0000 7f42fbfff640 1 -- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43041584d0 msgr2=0x7f43041464b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.374+0000 7f42fbfff640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f43041584d0 0x7f43041464b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.374+0000 7f42fbfff640 1 -- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43040c5a60 msgr2=0x7f43040ca210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.374+0000 7f42fbfff640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43040c5a60 0x7f43040ca210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.374+0000 7f42fbfff640 1 -- 172.21.15.98:0/3392422408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42fc009b30 con 0x7f4304151be0 2023-11-28T17:48:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.374+0000 7f43037fe640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43040c5a60 0x7f43040ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:48:39.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.374+0000 7f42fbfff640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4304151be0 0x7f4304145f70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00b3e0 tx=0x7f42ec00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.375+0000 7f4300ff9640 1 -- 172.21.15.98:0/3392422408 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec011830 con 0x7f4304151be0 2023-11-28T17:48:39.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.375+0000 7f4300ff9640 1 -- 172.21.15.98:0/3392422408 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ec0119b0 con 0x7f4304151be0 2023-11-28T17:48:39.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.375+0000 7f4300ff9640 1 -- 172.21.15.98:0/3392422408 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec011d60 con 0x7f4304151be0 2023-11-28T17:48:39.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.375+0000 7f4309db0640 1 -- 172.21.15.98:0/3392422408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f430414b050 con 0x7f4304151be0 2023-11-28T17:48:39.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.375+0000 7f4309db0640 1 -- 172.21.15.98:0/3392422408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43041edf10 con 0x7f4304151be0 2023-11-28T17:48:39.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.376+0000 7f4309db0640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4304146aa0 0x7f430414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.376+0000 7f4309db0640 1 -- 172.21.15.98:0/3392422408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43040744d0 con 0x7f4304146aa0 2023-11-28T17:48:39.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.376+0000 7f43037fe640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4304146aa0 0x7f430414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.377+0000 7f4300ff9640 1 -- 172.21.15.98:0/3392422408 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f42ec01a400 con 0x7f4304151be0 2023-11-28T17:48:39.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.377+0000 7f43037fe640 1 -- 172.21.15.98:0/3392422408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4304158920 con 0x7f4304146aa0 2023-11-28T17:48:39.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.377+0000 7f43037fe640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4304146aa0 0x7f430414b4c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42f400c720 tx=0x7f42f400cbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.377+0000 7f4300ff9640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f42e406b2d0 0x7f42e406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.378+0000 7f4300ff9640 1 -- 172.21.15.98:0/3392422408 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f42ec089e70 con 0x7f4304151be0 2023-11-28T17:48:39.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.378+0000 7f4303fff640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f42e406b2d0 0x7f42e406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.378+0000 7f4300ff9640 1 -- 172.21.15.98:0/3392422408 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f4007890 con 0x7f4304146aa0 2023-11-28T17:48:39.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.378+0000 7f4303fff640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f42e406b2d0 0x7f42e406d790 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f42fc035000 tx=0x7f42fc030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.379+0000 7f4300ff9640 1 -- 172.21.15.98:0/3392422408 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f42f40079f0 con 0x7f4304146aa0 2023-11-28T17:48:39.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.379+0000 7f4300ff9640 1 -- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4304146aa0 msgr2=0x7f430414b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.379+0000 7f4300ff9640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4304146aa0 0x7f430414b4c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42f400c720 tx=0x7f42f400cbe0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.379+0000 7f4300ff9640 1 -- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4304146aa0 msgr2=0x7f430414b4c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:39.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.379+0000 7f4300ff9640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4304146aa0 0x7f430414b4c0 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42f400c720 tx=0x7f42f400cbe0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.386+0000 7f4309db0640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f430414ba00 0x7f430414c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.386+0000 7f4309db0640 1 -- 172.21.15.98:0/3392422408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43040c84e0 con 0x7f430414ba00 2023-11-28T17:48:39.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.386+0000 7f43037fe640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f430414ba00 0x7f430414c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.387+0000 7f43037fe640 1 -- 172.21.15.98:0/3392422408 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f43040c4e40 con 0x7f430414ba00 2023-11-28T17:48:39.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.387+0000 7f43037fe640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f430414ba00 0x7f430414c440 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f42f400c720 tx=0x7f42f40047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.388+0000 7f4300ff9640 1 -- 172.21.15.98:0/3392422408 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f4028070 con 0x7f430414ba00 2023-11-28T17:48:39.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.388+0000 7f4300ff9640 1 -- 172.21.15.98:0/3392422408 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4805 (secure 0 0 0) 0x7f42f4020040 con 0x7f430414ba00 2023-11-28T17:48:39.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.388+0000 7f4300ff9640 1 -- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f430414ba00 msgr2=0x7f430414c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.388+0000 7f4300ff9640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f430414ba00 0x7f430414c440 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f42f400c720 tx=0x7f42f40047d0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.388+0000 7f4300ff9640 1 -- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f430414ba00 msgr2=0x7f430414c440 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:39.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.388+0000 7f4300ff9640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f430414ba00 0x7f430414c440 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f42f400c720 tx=0x7f42f40047d0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.389 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:48:39.390 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:48:39.390 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:48:39.390 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:48:39.390 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2023-11-28T17:48:39.390 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:48:39.390 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:48:39.390 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:48:39.390 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:48:39.390 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.390 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2023-11-28T17:48:39.390 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:39.390 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:48:39.391 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:48:39.391 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:39.391 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:39.391 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:39.391 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:39.391 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:39.391 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:39.391 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:39.391 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:39.391 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:39.391 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:39.391 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.392 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:48:39.392 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:48:39.392 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:39.392 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:39.392 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:39.392 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:39.392 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:39.392 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:39.392 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:39.392 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:39.392 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:39.392 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:39.393 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.393 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:48:39.393 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:48:39.393 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:48:39.393 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:48:39.393 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:48:39.393 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:48:39.393 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:39.393 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:39.393 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:48:39.393 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:48:39.394 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:48:39.394 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:48:39.394 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:48:39.394 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:48:39.394 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:48:39.394 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:39.394 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:48:39.394 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:39.394 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:39.394 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:39.394 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:39.394 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:39.395 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:39.395 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:39.395 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:39.395 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:39.395 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:39.395 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.395 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:48:39.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.395 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:48:39.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.395 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:48:39.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:48:39.395 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:39.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:39.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:39.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:48:39.396 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:39.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:48:39.396 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.396 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.397 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:48:39.397 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:48:39.397 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:39.397 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:39.397 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:39.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.397 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:48:39.397 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:48:39.397 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:39.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:39.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:39.398 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:48:39.398 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:39.398 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:48:39.398 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.398 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.399 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:48:39.399 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:48:39.399 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:39.399 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:39.399 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:39.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.399 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:48:39.399 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:48:39.399 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:39.399 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:39.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:39.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:48:39.400 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:39.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:48:39.400 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.400 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.400 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:48:39.400 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:48:39.401 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:39.401 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:39.401 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:39.401 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.401 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.401 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:48:39.401 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:48:39.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.401 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:39.401 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:39.401 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:48:39.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.402 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.402 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:48:39.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.402 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:39.402 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:39.402 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:48:39.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.402 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.402 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:48:39.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.402 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:39.403 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:39.403 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:48:39.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.403 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.403 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:48:39.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.403 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:39.403 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:39.403 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:48:39.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.403 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.403 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:48:39.404 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:48:39.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.389+0000 7f4309db0640 1 -- 172.21.15.98:0/3392422408 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f42e406b2d0 msgr2=0x7f42e406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.389+0000 7f4309db0640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f42e406b2d0 0x7f42e406d790 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f42fc035000 tx=0x7f42fc030000 comp rx=0 tx=0).stop 2023-11-28T17:48:39.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.389+0000 7f4309db0640 1 -- 172.21.15.98:0/3392422408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4304151be0 msgr2=0x7f4304145f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.389+0000 7f4309db0640 1 --2- 172.21.15.98:0/3392422408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4304151be0 0x7f4304145f70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00b3e0 tx=0x7f42ec00b8a0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.389+0000 7f4303fff640 1 -- 172.21.15.98:0/3392422408 reap_dead start 2023-11-28T17:48:39.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.389+0000 7f4303fff640 1 -- 172.21.15.98:0/3392422408 reap_dead start 2023-11-28T17:48:39.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.389+0000 7f4309db0640 1 -- 172.21.15.98:0/3392422408 shutdown_connections 2023-11-28T17:48:39.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.389+0000 7f4309db0640 1 -- 172.21.15.98:0/3392422408 >> 172.21.15.98:0/3392422408 conn(0x7f430406c0f0 msgr2=0x7f4304156e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:39.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.389+0000 7f4309db0640 1 -- 172.21.15.98:0/3392422408 shutdown_connections 2023-11-28T17:48:39.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.389+0000 7f4309db0640 1 -- 172.21.15.98:0/3392422408 wait complete. 2023-11-28T17:48:39.406 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:48:39.406 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:48:39.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:48:39.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.604+0000 7f3322783640 1 Processor -- start 2023-11-28T17:48:39.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.604+0000 7f3322783640 1 -- start start 2023-11-28T17:48:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.606+0000 7f3322783640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f331c0c24e0 0x7f331c0c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.606+0000 7f3322783640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c28c0 0x7f331c162390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.606+0000 7f3322783640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f331c0c6300 con 0x7f331c06b160 2023-11-28T17:48:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.606+0000 7f3322783640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f331c0c2d40 con 0x7f331c0c24e0 2023-11-28T17:48:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.606+0000 7f3322783640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331c0c7430 con 0x7f331c0c28c0 2023-11-28T17:48:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.606+0000 7f3321f82640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c28c0 0x7f331c162390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.606+0000 7f3321f82640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c28c0 0x7f331c162390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:47132/0 (socket says 172.21.15.98:47132) 2023-11-28T17:48:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.606+0000 7f3321f82640 1 -- 172.21.15.98:0/1097238977 learned_addr learned my addr 172.21.15.98:0/1097238977 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:39.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.607+0000 7f3321781640 1 --2- 172.21.15.98:0/1097238977 >> v2:172.21.15.113:3300/0 conn(0x7f331c0c24e0 0x7f331c0c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.607+0000 7f3321f82640 1 -- 172.21.15.98:0/1097238977 >> v2:172.21.15.113:3300/0 conn(0x7f331c0c24e0 msgr2=0x7f331c0c6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:39.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.607+0000 7f3321f82640 1 --2- 172.21.15.98:0/1097238977 >> v2:172.21.15.113:3300/0 conn(0x7f331c0c24e0 0x7f331c0c6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.607+0000 7f3321f82640 1 -- 172.21.15.98:0/1097238977 >> v1:172.21.15.113:6789/0 conn(0x7f331c06b160 legacy=0x7f331c157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.607+0000 7f3321f82640 1 -- 172.21.15.98:0/1097238977 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331c0c7710 con 0x7f331c0c28c0 2023-11-28T17:48:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.607+0000 7f3321781640 1 --2- 172.21.15.98:0/1097238977 >> v2:172.21.15.113:3300/0 conn(0x7f331c0c24e0 0x7f331c0c6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:48:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.608+0000 7f3321f82640 1 --2- 172.21.15.98:0/1097238977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c28c0 0x7f331c162390 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3314009e50 tx=0x7f331402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cf186de35d599a1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.609+0000 7f3313fff640 1 -- 172.21.15.98:0/1097238977 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331402c9e0 con 0x7f331c0c28c0 2023-11-28T17:48:39.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.609+0000 7f3313fff640 1 -- 172.21.15.98:0/1097238977 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331402cb40 con 0x7f331c0c28c0 2023-11-28T17:48:39.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.609+0000 7f3313fff640 1 -- 172.21.15.98:0/1097238977 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33140343f0 con 0x7f331c0c28c0 2023-11-28T17:48:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.609+0000 7f3322783640 1 -- 172.21.15.98:0/1097238977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c28c0 msgr2=0x7f331c162390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.609+0000 7f3322783640 1 --2- 172.21.15.98:0/1097238977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c28c0 0x7f331c162390 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3314009e50 tx=0x7f331402be40 comp rx=0 tx=0).stop 2023-11-28T17:48:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.610+0000 7f3322783640 1 -- 172.21.15.98:0/1097238977 shutdown_connections 2023-11-28T17:48:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.610+0000 7f3322783640 1 --2- 172.21.15.98:0/1097238977 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c28c0 0x7f331c162390 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.610+0000 7f3322783640 1 --2- 172.21.15.98:0/1097238977 >> v2:172.21.15.113:3300/0 conn(0x7f331c0c24e0 0x7f331c0c6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.610+0000 7f3322783640 1 -- 172.21.15.98:0/1097238977 >> 172.21.15.98:0/1097238977 conn(0x7f331c0cafe0 msgr2=0x7f331c0cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.610+0000 7f3322783640 1 -- 172.21.15.98:0/1097238977 shutdown_connections 2023-11-28T17:48:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.610+0000 7f3322783640 1 -- 172.21.15.98:0/1097238977 wait complete. 2023-11-28T17:48:39.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.612+0000 7f3322783640 1 Processor -- start 2023-11-28T17:48:39.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.612+0000 7f3322783640 1 -- start start 2023-11-28T17:48:39.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.613+0000 7f3322783640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f331c06b160 0x7f331c0c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.613+0000 7f3322783640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c24e0 0x7f331c147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.613+0000 7f3322783640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c0c28c0 0x7f331c147a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.613+0000 7f3322783640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c14cf90 con 0x7f331c0c28c0 2023-11-28T17:48:39.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.613+0000 7f3322783640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331c153810 con 0x7f331c06b160 2023-11-28T17:48:39.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.613+0000 7f3322783640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331c153bb0 con 0x7f331c0c24e0 2023-11-28T17:48:39.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.614+0000 7f3321f82640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c0c28c0 0x7f331c147a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.614+0000 7f3321f82640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c0c28c0 0x7f331c147a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38440/0 (socket says 172.21.15.98:38440) 2023-11-28T17:48:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.614+0000 7f3321f82640 1 -- 172.21.15.98:0/397242463 learned_addr learned my addr 172.21.15.98:0/397242463 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.614+0000 7f3321781640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f331c06b160 0x7f331c0c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.614+0000 7f3320f80640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c24e0 0x7f331c147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.614+0000 7f3321f82640 1 -- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c24e0 msgr2=0x7f331c147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.614+0000 7f3321f82640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c24e0 0x7f331c147510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.614+0000 7f3321f82640 1 -- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f331c06b160 msgr2=0x7f331c0c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.614+0000 7f3321f82640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f331c06b160 0x7f331c0c4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.614+0000 7f3321f82640 1 -- 172.21.15.98:0/397242463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3314009b30 con 0x7f331c0c28c0 2023-11-28T17:48:39.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.614+0000 7f3321f82640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c0c28c0 0x7f331c147a50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3314036040 tx=0x7f3314004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.615+0000 7f3311ffb640 1 -- 172.21.15.98:0/397242463 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314040670 con 0x7f331c0c28c0 2023-11-28T17:48:39.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.615+0000 7f3311ffb640 1 -- 172.21.15.98:0/397242463 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33140407f0 con 0x7f331c0c28c0 2023-11-28T17:48:39.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.615+0000 7f3322783640 1 -- 172.21.15.98:0/397242463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331c0ca7a0 con 0x7f331c0c28c0 2023-11-28T17:48:39.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.615+0000 7f3311ffb640 1 -- 172.21.15.98:0/397242463 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314040ba0 con 0x7f331c0c28c0 2023-11-28T17:48:39.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.615+0000 7f3322783640 1 -- 172.21.15.98:0/397242463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331c14c5f0 con 0x7f331c0c28c0 2023-11-28T17:48:39.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.616+0000 7f3322783640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c9e50 0x7f331c14c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.616+0000 7f3322783640 1 -- 172.21.15.98:0/397242463 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331c074310 con 0x7f331c0c9e50 2023-11-28T17:48:39.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.616+0000 7f3321781640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c9e50 0x7f331c14c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.617+0000 7f3311ffb640 1 -- 172.21.15.98:0/397242463 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3314040d00 con 0x7f331c0c28c0 2023-11-28T17:48:39.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.617+0000 7f3321781640 1 -- 172.21.15.98:0/397242463 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f331c0c68a0 con 0x7f331c0c9e50 2023-11-28T17:48:39.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.618+0000 7f3321781640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c9e50 0x7f331c14c7e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3318010790 tx=0x7f3318010c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.618+0000 7f3311ffb640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f330006b2d0 0x7f330006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.618+0000 7f3311ffb640 1 -- 172.21.15.98:0/397242463 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f33140accc0 con 0x7f331c0c28c0 2023-11-28T17:48:39.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.618+0000 7f3320f80640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f330006b2d0 0x7f330006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.618+0000 7f3311ffb640 1 -- 172.21.15.98:0/397242463 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331800e9d0 con 0x7f331c0c9e50 2023-11-28T17:48:39.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.618+0000 7f3320f80640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f330006b2d0 0x7f330006d790 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f331c14d4e0 tx=0x7f330c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.619+0000 7f3311ffb640 1 -- 172.21.15.98:0/397242463 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f331800c050 con 0x7f331c0c9e50 2023-11-28T17:48:39.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.619+0000 7f3311ffb640 1 -- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c9e50 msgr2=0x7f331c14c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.619+0000 7f3311ffb640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c9e50 0x7f331c14c7e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3318010790 tx=0x7f3318010c50 comp rx=0 tx=0).stop 2023-11-28T17:48:39.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.619+0000 7f3311ffb640 1 -- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c9e50 msgr2=0x7f331c14c7e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:39.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.619+0000 7f3311ffb640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c0c9e50 0x7f331c14c7e0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3318010790 tx=0x7f3318010c50 comp rx=0 tx=0).stop 2023-11-28T17:48:39.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.626+0000 7f3322783640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c152400 0x7f331c1f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.626+0000 7f3322783640 1 -- 172.21.15.98:0/397242463 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331c14d0d0 con 0x7f331c152400 2023-11-28T17:48:39.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.627+0000 7f3321781640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c152400 0x7f331c1f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.627+0000 7f3321781640 1 -- 172.21.15.98:0/397242463 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f331c150950 con 0x7f331c152400 2023-11-28T17:48:39.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.628+0000 7f3321781640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c152400 0x7f331c1f29e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3318010790 tx=0x7f3318004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.628+0000 7f3311ffb640 1 -- 172.21.15.98:0/397242463 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3318004440 con 0x7f331c152400 2023-11-28T17:48:39.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.629+0000 7f3311ffb640 1 -- 172.21.15.98:0/397242463 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4465 (secure 0 0 0) 0x7f33180212e0 con 0x7f331c152400 2023-11-28T17:48:39.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.629+0000 7f3311ffb640 1 -- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c152400 msgr2=0x7f331c1f29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.629+0000 7f3311ffb640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c152400 0x7f331c1f29e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3318010790 tx=0x7f3318004730 comp rx=0 tx=0).stop 2023-11-28T17:48:39.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.629+0000 7f3311ffb640 1 -- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c152400 msgr2=0x7f331c1f29e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:39.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.629+0000 7f3311ffb640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f331c152400 0x7f331c1f29e0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3318010790 tx=0x7f3318004730 comp rx=0 tx=0).stop 2023-11-28T17:48:39.633 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:48:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:48:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:48:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:48:39.633 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2023-11-28T17:48:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:48:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:48:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:48:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:48:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2023-11-28T17:48:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:48:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:48:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:39.635 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.637 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:48:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:48:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:39.638 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:39.639 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.640 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:48:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:48:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:48:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:48:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:48:39.641 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:48:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:48:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:48:39.642 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:48:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:48:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:48:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:48:39.643 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:48:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:48:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:39.644 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:39.645 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:39.645 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:39.645 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:39.645 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:39.646 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:39.646 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:39.646 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:39.646 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.647 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:48:39.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.647 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:48:39.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.647 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:48:39.648 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:48:39.648 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:39.648 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:39.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:39.649 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:48:39.649 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:39.650 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:48:39.650 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.650 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.651 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.651 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:48:39.651 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:48:39.651 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:39.652 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:39.652 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:39.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.652 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:48:39.652 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:48:39.653 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:39.653 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:39.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:39.653 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:48:39.653 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.654 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:39.654 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:48:39.654 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.657 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.658 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.658 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:48:39.659 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:48:39.659 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:39.659 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:39.659 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:39.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.660 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:48:39.660 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:48:39.660 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:39.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:39.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.661 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:39.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:48:39.661 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:39.663 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:48:39.663 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:39.663 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.663 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.663 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:48:39.663 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:48:39.663 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:39.664 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:39.664 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:39.664 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.664 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.664 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:48:39.664 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:48:39.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.664 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:39.664 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:39.664 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:48:39.664 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.664 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:39.665 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:48:39.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:39.665 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:39.665 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:39.665 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:48:39.665 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:39.665 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:39.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:39.665 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:48:39.665 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:48:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.629+0000 7f3322783640 1 -- 172.21.15.98:0/397242463 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f330006b2d0 msgr2=0x7f330006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.629+0000 7f3322783640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f330006b2d0 0x7f330006d790 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f331c14d4e0 tx=0x7f330c009040 comp rx=0 tx=0).stop 2023-11-28T17:48:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.629+0000 7f3322783640 1 -- 172.21.15.98:0/397242463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c0c28c0 msgr2=0x7f331c147a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.629+0000 7f3321f82640 1 -- 172.21.15.98:0/397242463 reap_dead start 2023-11-28T17:48:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.630+0000 7f3322783640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c0c28c0 0x7f331c147a50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3314036040 tx=0x7f3314004a40 comp rx=0 tx=0).stop 2023-11-28T17:48:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.630+0000 7f3322783640 1 -- 172.21.15.98:0/397242463 shutdown_connections 2023-11-28T17:48:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.630+0000 7f3322783640 1 --2- 172.21.15.98:0/397242463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f331c0c28c0 0x7f331c147a50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.630+0000 7f3322783640 1 -- 172.21.15.98:0/397242463 >> 172.21.15.98:0/397242463 conn(0x7f331c0cafe0 msgr2=0x7f331c0cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.630+0000 7f3322783640 1 -- 172.21.15.98:0/397242463 shutdown_connections 2023-11-28T17:48:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.630+0000 7f3322783640 1 -- 172.21.15.98:0/397242463 wait complete. 2023-11-28T17:48:39.668 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:48:39.668 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-11-28T17:48:39.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-11-28T17:48:39.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.888+0000 7f4f8ca39640 1 Processor -- start 2023-11-28T17:48:39.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.888+0000 7f4f8ca39640 1 -- start start 2023-11-28T17:48:39.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.889+0000 7f4f8ca39640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f8815a6d0 0x7f4f880cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.889+0000 7f4f8ca39640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8815ab10 con 0x7f4f880cf7c0 2023-11-28T17:48:39.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.889+0000 7f4f8ca39640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8815d330 con 0x7f4f88153bb0 2023-11-28T17:48:39.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.889+0000 7f4f8ca39640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8815d120 con 0x7f4f8815a6d0 2023-11-28T17:48:39.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.890+0000 7f4f87fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4f880cf7c0 0x7f4f880b3760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33762/0 (socket says 172.21.15.98:33762) 2023-11-28T17:48:39.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.890+0000 7f4f87fff640 1 -- 172.21.15.98:0/3823542854 learned_addr learned my addr 172.21.15.98:0/3823542854 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:48:39.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.890+0000 7f4f86ffd640 1 --2- 172.21.15.98:0/3823542854 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f8815a6d0 0x7f4f880cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.890+0000 7f4f867fc640 1 -- 172.21.15.98:0/3823542854 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646626562 0 0) 0x7f4f8815ab10 con 0x7f4f880cf7c0 2023-11-28T17:48:39.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.890+0000 7f4f867fc640 1 -- 172.21.15.98:0/3823542854 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f68003520 con 0x7f4f880cf7c0 2023-11-28T17:48:39.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.890+0000 7f4f867fc640 1 -- 172.21.15.98:0/3823542854 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1088910405 0 0) 0x7f4f68003520 con 0x7f4f880cf7c0 2023-11-28T17:48:39.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.890+0000 7f4f867fc640 1 -- 172.21.15.98:0/3823542854 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f8815a6d0 msgr2=0x7f4f880cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:39.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.890+0000 7f4f867fc640 1 --2- 172.21.15.98:0/3823542854 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f8815a6d0 0x7f4f880cf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.890+0000 7f4f867fc640 1 -- 172.21.15.98:0/3823542854 >> v1:172.21.15.113:6789/0 conn(0x7f4f88153bb0 legacy=0x7f4f880ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.890+0000 7f4f867fc640 1 -- 172.21.15.98:0/3823542854 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8815d610 con 0x7f4f880cf7c0 2023-11-28T17:48:39.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f867fc640 1 -- 172.21.15.98:0/3823542854 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f4f800019f0 con 0x7f4f880cf7c0 2023-11-28T17:48:39.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f867fc640 1 -- 172.21.15.98:0/3823542854 >> v1:172.21.15.98:6789/0 conn(0x7f4f880cf7c0 legacy=0x7f4f880b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f867fc640 1 --2- 172.21.15.98:0/3823542854 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f8815a6d0 0x7f4f680039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f867fc640 1 --2- 172.21.15.98:0/3823542854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f68003f20 0x7f4f680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f867fc640 1 --2- 172.21.15.98:0/3823542854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f68004d00 0x7f4f68005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f867fc640 1 -- 172.21.15.98:0/3823542854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8815d120 con 0x7f4f68004d00 2023-11-28T17:48:39.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f867fc640 1 -- 172.21.15.98:0/3823542854 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6800baf0 con 0x7f4f68003f20 2023-11-28T17:48:39.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f867fc640 1 -- 172.21.15.98:0/3823542854 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6800bcb0 con 0x7f4f8815a6d0 2023-11-28T17:48:39.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f87fff640 1 --2- 172.21.15.98:0/3823542854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f68004d00 0x7f4f68005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f877fe640 1 --2- 172.21.15.98:0/3823542854 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f8815a6d0 0x7f4f680039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f86ffd640 1 --2- 172.21.15.98:0/3823542854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f68003f20 0x7f4f680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f87fff640 1 -- 172.21.15.98:0/3823542854 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f8815a6d0 msgr2=0x7f4f680039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:39.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f87fff640 1 --2- 172.21.15.98:0/3823542854 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f8815a6d0 0x7f4f680039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f87fff640 1 -- 172.21.15.98:0/3823542854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f68003f20 msgr2=0x7f4f680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:39.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f87fff640 1 --2- 172.21.15.98:0/3823542854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f68003f20 0x7f4f680047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f87fff640 1 -- 172.21.15.98:0/3823542854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f68003520 con 0x7f4f68004d00 2023-11-28T17:48:39.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.891+0000 7f4f86ffd640 1 --2- 172.21.15.98:0/3823542854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f68003f20 0x7f4f680047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:48:39.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.892+0000 7f4f87fff640 1 --2- 172.21.15.98:0/3823542854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f68004d00 0x7f4f68005180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000a820 tx=0x7f4f680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1799a1ea7077f43 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.893+0000 7f4f867fc640 1 -- 172.21.15.98:0/3823542854 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f800072d0 con 0x7f4f68004d00 2023-11-28T17:48:39.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.893+0000 7f4f867fc640 1 -- 172.21.15.98:0/3823542854 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8002f070 con 0x7f4f68004d00 2023-11-28T17:48:39.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.893+0000 7f4f867fc640 1 -- 172.21.15.98:0/3823542854 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f800076f0 con 0x7f4f68004d00 2023-11-28T17:48:39.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.893+0000 7f4f8ca39640 1 -- 172.21.15.98:0/3823542854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f68004d00 msgr2=0x7f4f68005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.893+0000 7f4f8ca39640 1 --2- 172.21.15.98:0/3823542854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f68004d00 0x7f4f68005180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000a820 tx=0x7f4f680057f0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.893+0000 7f4f87fff640 1 -- 172.21.15.98:0/3823542854 reap_dead start 2023-11-28T17:48:39.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.893+0000 7f4f8ca39640 1 -- 172.21.15.98:0/3823542854 shutdown_connections 2023-11-28T17:48:39.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.893+0000 7f4f8ca39640 1 -- 172.21.15.98:0/3823542854 >> 172.21.15.98:0/3823542854 conn(0x7f4f88071c00 msgr2=0x7f4f88072010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:39.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.894+0000 7f4f8ca39640 1 -- 172.21.15.98:0/3823542854 shutdown_connections 2023-11-28T17:48:39.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.894+0000 7f4f8ca39640 1 -- 172.21.15.98:0/3823542854 wait complete. 2023-11-28T17:48:39.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.895+0000 7f4f8ca39640 1 Processor -- start 2023-11-28T17:48:39.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.895+0000 7f4f8ca39640 1 -- start start 2023-11-28T17:48:39.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.896+0000 7f4f8ca39640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f68004d00 0x7f4f88155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.896+0000 7f4f8ca39640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f88156f60 0x7f4f880c5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.896+0000 7f4f8ca39640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f881f2ed0 0x7f4f881f3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.896+0000 7f4f8ca39640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f88154400 con 0x7f4f68004d00 2023-11-28T17:48:39.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.896+0000 7f4f8ca39640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8814d200 con 0x7f4f88156f60 2023-11-28T17:48:39.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.896+0000 7f4f8ca39640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f88155d90 con 0x7f4f881f2ed0 2023-11-28T17:48:39.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.897+0000 7f4f877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f68004d00 0x7f4f88155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.897+0000 7f4f877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f68004d00 0x7f4f88155620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38466/0 (socket says 172.21.15.98:38466) 2023-11-28T17:48:39.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.897+0000 7f4f877fe640 1 -- 172.21.15.98:0/2363876131 learned_addr learned my addr 172.21.15.98:0/2363876131 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:39.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.897+0000 7f4f877fe640 1 -- 172.21.15.98:0/2363876131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f881f2ed0 msgr2=0x7f4f881f3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.897+0000 7f4f877fe640 1 --2- 172.21.15.98:0/2363876131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f881f2ed0 0x7f4f881f3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.897+0000 7f4f877fe640 1 -- 172.21.15.98:0/2363876131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f88156f60 msgr2=0x7f4f880c5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:39.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.897+0000 7f4f877fe640 1 --2- 172.21.15.98:0/2363876131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f88156f60 0x7f4f880c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:39.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.897+0000 7f4f877fe640 1 -- 172.21.15.98:0/2363876131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f881f3890 con 0x7f4f68004d00 2023-11-28T17:48:39.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.897+0000 7f4f877fe640 1 --2- 172.21.15.98:0/2363876131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f68004d00 0x7f4f88155620 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800e4d0 tx=0x7f4f7800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.897+0000 7f4f7ffff640 1 -- 172.21.15.98:0/2363876131 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f780052e0 con 0x7f4f68004d00 2023-11-28T17:48:39.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.897+0000 7f4f7ffff640 1 -- 172.21.15.98:0/2363876131 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f78005440 con 0x7f4f68004d00 2023-11-28T17:48:39.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.897+0000 7f4f7ffff640 1 -- 172.21.15.98:0/2363876131 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f78003810 con 0x7f4f68004d00 2023-11-28T17:48:39.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.897+0000 7f4f8ca39640 1 -- 172.21.15.98:0/2363876131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f881545f0 con 0x7f4f68004d00 2023-11-28T17:48:39.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.898+0000 7f4f8ca39640 1 -- 172.21.15.98:0/2363876131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f8814d450 con 0x7f4f68004d00 2023-11-28T17:48:39.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.899+0000 7f4f8ca39640 1 -- 172.21.15.98:0/2363876131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f88155410 con 0x7f4f68004d00 2023-11-28T17:48:39.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.900+0000 7f4f7ffff640 1 -- 172.21.15.98:0/2363876131 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4f78003970 con 0x7f4f68004d00 2023-11-28T17:48:39.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.900+0000 7f4f7ffff640 1 --2- 172.21.15.98:0/2363876131 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f6c06b2a0 0x7f4f6c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:39.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.901+0000 7f4f7ffff640 1 -- 172.21.15.98:0/2363876131 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4f780898f0 con 0x7f4f68004d00 2023-11-28T17:48:39.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.901+0000 7f4f86ffd640 1 --2- 172.21.15.98:0/2363876131 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f6c06b2a0 0x7f4f6c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:39.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.901+0000 7f4f86ffd640 1 --2- 172.21.15.98:0/2363876131 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f6c06b2a0 0x7f4f6c06d760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4f880cada0 tx=0x7f4f70008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:39.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:39.905+0000 7f4f7ffff640 1 -- 172.21.15.98:0/2363876131 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f88155410 con 0x7f4f68004d00 2023-11-28T17:48:40.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:48:40.014 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:48:40.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:40.169+0000 7f4f8ca39640 1 -- 172.21.15.98:0/2363876131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f4f88155410 con 0x7f4f68004d00 2023-11-28T17:48:40.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:40.170+0000 7f4f7ffff640 1 -- 172.21.15.98:0/2363876131 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f4f88155410 con 0x7f4f68004d00 2023-11-28T17:48:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:40.173+0000 7f4f8ca39640 1 -- 172.21.15.98:0/2363876131 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f6c06b2a0 msgr2=0x7f4f6c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:40.173+0000 7f4f8ca39640 1 --2- 172.21.15.98:0/2363876131 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f6c06b2a0 0x7f4f6c06d760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4f880cada0 tx=0x7f4f70008040 comp rx=0 tx=0).stop 2023-11-28T17:48:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:40.173+0000 7f4f8ca39640 1 -- 172.21.15.98:0/2363876131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f68004d00 msgr2=0x7f4f88155620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:40.173+0000 7f4f8ca39640 1 --2- 172.21.15.98:0/2363876131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f68004d00 0x7f4f88155620 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800e4d0 tx=0x7f4f7800e990 comp rx=0 tx=0).stop 2023-11-28T17:48:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:40.174+0000 7f4f8ca39640 1 -- 172.21.15.98:0/2363876131 shutdown_connections 2023-11-28T17:48:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:40.174+0000 7f4f8ca39640 1 --2- 172.21.15.98:0/2363876131 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4f881f2ed0 0x7f4f881f3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:40.174+0000 7f4f8ca39640 1 --2- 172.21.15.98:0/2363876131 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4f6c06b2a0 0x7f4f6c06d760 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:40.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:40.174+0000 7f4f8ca39640 1 --2- 172.21.15.98:0/2363876131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f88156f60 0x7f4f880c5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:40.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:40.174+0000 7f4f8ca39640 1 --2- 172.21.15.98:0/2363876131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f68004d00 0x7f4f88155620 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:40.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:40.174+0000 7f4f8ca39640 1 -- 172.21.15.98:0/2363876131 >> 172.21.15.98:0/2363876131 conn(0x7f4f88071c00 msgr2=0x7f4f880ce260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:40.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:40.174+0000 7f4f8ca39640 1 -- 172.21.15.98:0/2363876131 shutdown_connections 2023-11-28T17:48:40.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:40.174+0000 7f4f8ca39640 1 -- 172.21.15.98:0/2363876131 wait complete. 2023-11-28T17:48:40.196 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2023-11-28T17:48:41.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~229s 2023-11-28T17:48:47.992 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~236s 2023-11-28T17:48:50.197 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:48:50.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:48:50.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.413+0000 7f291589f640 1 Processor -- start 2023-11-28T17:48:50.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.413+0000 7f291589f640 1 -- start start 2023-11-28T17:48:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.415+0000 7f291589f640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2910150ac0 0x7f29100716e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.415+0000 7f291589f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f291006b1e0 0x7f2910071c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.415+0000 7f291589f640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2910150ea0 con 0x7f29100c71f0 2023-11-28T17:48:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.415+0000 7f291589f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f291015ad90 con 0x7f291006b1e0 2023-11-28T17:48:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.415+0000 7f291589f640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29101567a0 con 0x7f2910150ac0 2023-11-28T17:48:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.415+0000 7f290ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f291006b1e0 0x7f2910071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.415+0000 7f290ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f291006b1e0 0x7f2910071c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49476/0 (socket says 172.21.15.98:49476) 2023-11-28T17:48:50.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.415+0000 7f290ffff640 1 -- 172.21.15.98:0/278327166 learned_addr learned my addr 172.21.15.98:0/278327166 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:50.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.416+0000 7f290ffff640 1 -- 172.21.15.98:0/278327166 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2910150ac0 msgr2=0x7f29100716e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.416+0000 7f290ffff640 1 --2- 172.21.15.98:0/278327166 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2910150ac0 0x7f29100716e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.416+0000 7f290ffff640 1 -- 172.21.15.98:0/278327166 >> v1:172.21.15.113:6789/0 conn(0x7f29100c71f0 legacy=0x7f2910156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.416+0000 7f290ffff640 1 -- 172.21.15.98:0/278327166 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2910159f30 con 0x7f291006b1e0 2023-11-28T17:48:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.417+0000 7f290ffff640 1 --2- 172.21.15.98:0/278327166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f291006b1e0 0x7f2910071c20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f28fc009870 tx=0x7f28fc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef55b9fc0cf77e7a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.417+0000 7f290f7fe640 1 -- 172.21.15.98:0/278327166 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28fc030840 con 0x7f291006b1e0 2023-11-28T17:48:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.417+0000 7f290f7fe640 1 -- 172.21.15.98:0/278327166 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28fc0309a0 con 0x7f291006b1e0 2023-11-28T17:48:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.417+0000 7f290f7fe640 1 -- 172.21.15.98:0/278327166 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28fc030d30 con 0x7f291006b1e0 2023-11-28T17:48:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.418+0000 7f291589f640 1 -- 172.21.15.98:0/278327166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f291006b1e0 msgr2=0x7f2910071c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.418+0000 7f291589f640 1 --2- 172.21.15.98:0/278327166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f291006b1e0 0x7f2910071c20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f28fc009870 tx=0x7f28fc02b980 comp rx=0 tx=0).stop 2023-11-28T17:48:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.418+0000 7f291589f640 1 -- 172.21.15.98:0/278327166 shutdown_connections 2023-11-28T17:48:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.418+0000 7f291589f640 1 --2- 172.21.15.98:0/278327166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f291006b1e0 0x7f2910071c20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.418+0000 7f291589f640 1 --2- 172.21.15.98:0/278327166 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2910150ac0 0x7f29100716e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.418+0000 7f291589f640 1 -- 172.21.15.98:0/278327166 >> 172.21.15.98:0/278327166 conn(0x7f2910066230 msgr2=0x7f291006a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:50.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.418+0000 7f291589f640 1 -- 172.21.15.98:0/278327166 shutdown_connections 2023-11-28T17:48:50.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.418+0000 7f291589f640 1 -- 172.21.15.98:0/278327166 wait complete. 2023-11-28T17:48:50.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.420+0000 7f291589f640 1 Processor -- start 2023-11-28T17:48:50.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.420+0000 7f291589f640 1 -- start start 2023-11-28T17:48:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f291589f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f291006b1e0 0x7f29100cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f291589f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29100c71f0 0x7f2910148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f291589f640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2910150ac0 0x7f29101491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f291589f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29101ee620 con 0x7f29100c71f0 2023-11-28T17:48:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f291589f640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29101ed7c0 con 0x7f291006b1e0 2023-11-28T17:48:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f291589f640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29101edaa0 con 0x7f2910150ac0 2023-11-28T17:48:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f290ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29100c71f0 0x7f2910148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f290ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29100c71f0 0x7f2910148c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49490/0 (socket says 172.21.15.98:49490) 2023-11-28T17:48:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f290ffff640 1 -- 172.21.15.98:0/2497596260 learned_addr learned my addr 172.21.15.98:0/2497596260 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f291489d640 1 --2- 172.21.15.98:0/2497596260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f291006b1e0 0x7f29100cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f291509e640 1 --2- 172.21.15.98:0/2497596260 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2910150ac0 0x7f29101491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f290ffff640 1 -- 172.21.15.98:0/2497596260 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2910150ac0 msgr2=0x7f29101491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f290ffff640 1 --2- 172.21.15.98:0/2497596260 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2910150ac0 0x7f29101491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f290ffff640 1 -- 172.21.15.98:0/2497596260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f291006b1e0 msgr2=0x7f29100cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f290ffff640 1 --2- 172.21.15.98:0/2497596260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f291006b1e0 0x7f29100cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.421+0000 7f290ffff640 1 -- 172.21.15.98:0/2497596260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc0094d0 con 0x7f29100c71f0 2023-11-28T17:48:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.422+0000 7f290ffff640 1 --2- 172.21.15.98:0/2497596260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29100c71f0 0x7f2910148c60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f28fc009870 tx=0x7f28fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.422+0000 7f290d7fa640 1 -- 172.21.15.98:0/2497596260 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28fc039b60 con 0x7f29100c71f0 2023-11-28T17:48:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.422+0000 7f290d7fa640 1 -- 172.21.15.98:0/2497596260 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28fc039cc0 con 0x7f29100c71f0 2023-11-28T17:48:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.422+0000 7f290d7fa640 1 -- 172.21.15.98:0/2497596260 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28fc043630 con 0x7f29100c71f0 2023-11-28T17:48:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.422+0000 7f291589f640 1 -- 172.21.15.98:0/2497596260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29101edd80 con 0x7f29100c71f0 2023-11-28T17:48:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.422+0000 7f291589f640 1 -- 172.21.15.98:0/2497596260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29101ee340 con 0x7f29100c71f0 2023-11-28T17:48:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.424+0000 7f291589f640 1 -- 172.21.15.98:0/2497596260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2910074db0 con 0x7f29100c71f0 2023-11-28T17:48:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.425+0000 7f290d7fa640 1 -- 172.21.15.98:0/2497596260 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f28fc042390 con 0x7f29100c71f0 2023-11-28T17:48:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.425+0000 7f290d7fa640 1 --2- 172.21.15.98:0/2497596260 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f28f406b2a0 0x7f28f406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.426+0000 7f290d7fa640 1 -- 172.21.15.98:0/2497596260 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f28fc0ac020 con 0x7f29100c71f0 2023-11-28T17:48:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.426+0000 7f291489d640 1 --2- 172.21.15.98:0/2497596260 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f28f406b2a0 0x7f28f406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:50.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.427+0000 7f291489d640 1 --2- 172.21.15.98:0/2497596260 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f28f406b2a0 0x7f28f406d760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2904009890 tx=0x7f2904008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.430+0000 7f290d7fa640 1 -- 172.21.15.98:0/2497596260 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f28fc076d80 con 0x7f29100c71f0 2023-11-28T17:48:50.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.691+0000 7f291589f640 1 -- 172.21.15.98:0/2497596260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f29101477e0 con 0x7f29100c71f0 2023-11-28T17:48:50.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.694+0000 7f290d7fa640 1 -- 172.21.15.98:0/2497596260 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f28fc034620 con 0x7f29100c71f0 2023-11-28T17:48:50.695 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:48:50.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.697+0000 7f291589f640 1 -- 172.21.15.98:0/2497596260 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f28f406b2a0 msgr2=0x7f28f406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:50.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.697+0000 7f291589f640 1 --2- 172.21.15.98:0/2497596260 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f28f406b2a0 0x7f28f406d760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2904009890 tx=0x7f2904008040 comp rx=0 tx=0).stop 2023-11-28T17:48:50.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.697+0000 7f291589f640 1 -- 172.21.15.98:0/2497596260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29100c71f0 msgr2=0x7f2910148c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:50.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.697+0000 7f291589f640 1 --2- 172.21.15.98:0/2497596260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29100c71f0 0x7f2910148c60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f28fc009870 tx=0x7f28fc004730 comp rx=0 tx=0).stop 2023-11-28T17:48:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.697+0000 7f291589f640 1 -- 172.21.15.98:0/2497596260 shutdown_connections 2023-11-28T17:48:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.697+0000 7f291589f640 1 --2- 172.21.15.98:0/2497596260 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2910150ac0 0x7f29101491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.697+0000 7f291589f640 1 --2- 172.21.15.98:0/2497596260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29100c71f0 0x7f2910148c60 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.697+0000 7f291589f640 1 --2- 172.21.15.98:0/2497596260 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f28f406b2a0 0x7f28f406d760 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.697+0000 7f291589f640 1 --2- 172.21.15.98:0/2497596260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f291006b1e0 0x7f29100cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.697+0000 7f291589f640 1 -- 172.21.15.98:0/2497596260 >> 172.21.15.98:0/2497596260 conn(0x7f2910066230 msgr2=0x7f29100c5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.697+0000 7f291589f640 1 -- 172.21.15.98:0/2497596260 shutdown_connections 2023-11-28T17:48:50.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.697+0000 7f291589f640 1 -- 172.21.15.98:0/2497596260 wait complete. 2023-11-28T17:48:50.717 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:48:50.717 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2023-11-28T17:48:50.717 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:48:50.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:48:50.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.916+0000 7f0922a59640 1 Processor -- start 2023-11-28T17:48:50.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.917+0000 7f0922a59640 1 -- start start 2023-11-28T17:48:50.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.918+0000 7f0922a59640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f091c0c69b0 0x7f091c0c7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.918+0000 7f0922a59640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f091c0c7c60 0x7f091c164440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.918+0000 7f0922a59640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091c0c80f0 con 0x7f091c153c80 2023-11-28T17:48:50.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.918+0000 7f0922a59640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091c15e070 con 0x7f091c0c7c60 2023-11-28T17:48:50.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.918+0000 7f0922a59640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091c15d230 con 0x7f091c0c69b0 2023-11-28T17:48:50.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.918+0000 7f0921a57640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f091c153c80 0x7f091c06b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53442/0 (socket says 172.21.15.98:53442) 2023-11-28T17:48:50.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.918+0000 7f0921a57640 1 -- 172.21.15.98:0/1785114247 learned_addr learned my addr 172.21.15.98:0/1785114247 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:48:50.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 -- 172.21.15.98:0/1785114247 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542276828 0 0) 0x7f091c0c80f0 con 0x7f091c153c80 2023-11-28T17:48:50.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 -- 172.21.15.98:0/1785114247 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f090c003520 con 0x7f091c153c80 2023-11-28T17:48:50.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0922258640 1 --2- 172.21.15.98:0/1785114247 >> v2:172.21.15.113:3300/0 conn(0x7f091c0c7c60 0x7f091c164440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:50.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0921256640 1 --2- 172.21.15.98:0/1785114247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f091c0c69b0 0x7f091c0c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:50.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 -- 172.21.15.98:0/1785114247 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 154614645 0 0) 0x7f090c003520 con 0x7f091c153c80 2023-11-28T17:48:50.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 -- 172.21.15.98:0/1785114247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f091c0c69b0 msgr2=0x7f091c0c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:50.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 --2- 172.21.15.98:0/1785114247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f091c0c69b0 0x7f091c0c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 -- 172.21.15.98:0/1785114247 >> v2:172.21.15.113:3300/0 conn(0x7f091c0c7c60 msgr2=0x7f091c164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:50.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 --2- 172.21.15.98:0/1785114247 >> v2:172.21.15.113:3300/0 conn(0x7f091c0c7c60 0x7f091c164440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 -- 172.21.15.98:0/1785114247 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c15d4f0 con 0x7f091c153c80 2023-11-28T17:48:50.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 -- 172.21.15.98:0/1785114247 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f09100019f0 con 0x7f091c153c80 2023-11-28T17:48:50.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0921256640 1 --2- 172.21.15.98:0/1785114247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f091c0c69b0 0x7f091c0c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:48:50.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 -- 172.21.15.98:0/1785114247 >> v1:172.21.15.98:6789/0 conn(0x7f091c153c80 legacy=0x7f091c06b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:50.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 --2- 172.21.15.98:0/1785114247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f091c0c69b0 0x7f090c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 --2- 172.21.15.98:0/1785114247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003fb0 0x7f090c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 --2- 172.21.15.98:0/1785114247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f090c004d00 0x7f090c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 -- 172.21.15.98:0/1785114247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c15e070 con 0x7f090c003fb0 2023-11-28T17:48:50.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 -- 172.21.15.98:0/1785114247 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c15d230 con 0x7f090c004d00 2023-11-28T17:48:50.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.919+0000 7f0920a55640 1 -- 172.21.15.98:0/1785114247 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090c00bb40 con 0x7f091c0c69b0 2023-11-28T17:48:50.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.920+0000 7f0921256640 1 --2- 172.21.15.98:0/1785114247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003fb0 0x7f090c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:50.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.920+0000 7f0921a57640 1 --2- 172.21.15.98:0/1785114247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f091c0c69b0 0x7f090c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:50.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.920+0000 7f0921256640 1 -- 172.21.15.98:0/1785114247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f091c0c69b0 msgr2=0x7f090c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:50.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.920+0000 7f0921256640 1 --2- 172.21.15.98:0/1785114247 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f091c0c69b0 0x7f090c003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.920+0000 7f0921256640 1 -- 172.21.15.98:0/1785114247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f090c004d00 msgr2=0x7f090c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:50.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.920+0000 7f0921256640 1 --2- 172.21.15.98:0/1785114247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f090c004d00 0x7f090c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.920+0000 7f0921256640 1 -- 172.21.15.98:0/1785114247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090c003520 con 0x7f090c003fb0 2023-11-28T17:48:50.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.921+0000 7f0921256640 1 --2- 172.21.15.98:0/1785114247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003fb0 0x7f090c0047c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f09080098a0 tx=0x7f090c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d6e38e448fb7a85 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:50.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.921+0000 7f0920a55640 1 -- 172.21.15.98:0/1785114247 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908004230 con 0x7f090c003fb0 2023-11-28T17:48:50.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.921+0000 7f0920a55640 1 -- 172.21.15.98:0/1785114247 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0908004390 con 0x7f090c003fb0 2023-11-28T17:48:50.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.921+0000 7f0920a55640 1 -- 172.21.15.98:0/1785114247 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908003830 con 0x7f090c003fb0 2023-11-28T17:48:50.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.922+0000 7f0922a59640 1 -- 172.21.15.98:0/1785114247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003fb0 msgr2=0x7f090c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:50.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.922+0000 7f0922a59640 1 --2- 172.21.15.98:0/1785114247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003fb0 0x7f090c0047c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f09080098a0 tx=0x7f090c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.922+0000 7f0922258640 1 -- 172.21.15.98:0/1785114247 reap_dead start 2023-11-28T17:48:50.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.922+0000 7f0922a59640 1 -- 172.21.15.98:0/1785114247 shutdown_connections 2023-11-28T17:48:50.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.922+0000 7f0922a59640 1 -- 172.21.15.98:0/1785114247 >> 172.21.15.98:0/1785114247 conn(0x7f091c0cf300 msgr2=0x7f091c0cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:50.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.922+0000 7f0922a59640 1 -- 172.21.15.98:0/1785114247 shutdown_connections 2023-11-28T17:48:50.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.922+0000 7f0922a59640 1 -- 172.21.15.98:0/1785114247 wait complete. 2023-11-28T17:48:50.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.924+0000 7f0922a59640 1 Processor -- start 2023-11-28T17:48:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.924+0000 7f0922a59640 1 -- start start 2023-11-28T17:48:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.925+0000 7f0922a59640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003fb0 0x7f091c15a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.925+0000 7f0922a59640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f091c0c2e50 0x7f091c1f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.925+0000 7f0922a59640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f091c0ce4d0 0x7f091c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.925+0000 7f0922a59640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c1f3020 con 0x7f090c003fb0 2023-11-28T17:48:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.925+0000 7f0922a59640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c159ad0 con 0x7f091c0ce4d0 2023-11-28T17:48:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.925+0000 7f0922a59640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091c151a90 con 0x7f091c0c2e50 2023-11-28T17:48:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.925+0000 7f0921a57640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003fb0 0x7f091c15a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.925+0000 7f0921a57640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003fb0 0x7f091c15a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49506/0 (socket says 172.21.15.98:49506) 2023-11-28T17:48:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.925+0000 7f0921a57640 1 -- 172.21.15.98:0/2535973362 learned_addr learned my addr 172.21.15.98:0/2535973362 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.926+0000 7f0921a57640 1 -- 172.21.15.98:0/2535973362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f091c0c2e50 msgr2=0x7f091c1f2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:50.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.926+0000 7f0921a57640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f091c0c2e50 0x7f091c1f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.926+0000 7f0922258640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f091c0ce4d0 0x7f091c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.926+0000 7f0921a57640 1 -- 172.21.15.98:0/2535973362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f091c0ce4d0 msgr2=0x7f091c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.926+0000 7f0921a57640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f091c0ce4d0 0x7f091c15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.926+0000 7f0921a57640 1 -- 172.21.15.98:0/2535973362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c159250 con 0x7f090c003fb0 2023-11-28T17:48:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.926+0000 7f0921a57640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003fb0 0x7f091c15a630 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f09100021c0 tx=0x7f09100021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.926+0000 7f091a7fc640 1 -- 172.21.15.98:0/2535973362 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0910004d60 con 0x7f090c003fb0 2023-11-28T17:48:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.926+0000 7f091a7fc640 1 -- 172.21.15.98:0/2535973362 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09100070d0 con 0x7f090c003fb0 2023-11-28T17:48:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.926+0000 7f0922a59640 1 -- 172.21.15.98:0/2535973362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091c159510 con 0x7f090c003fb0 2023-11-28T17:48:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.926+0000 7f091a7fc640 1 -- 172.21.15.98:0/2535973362 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0910007460 con 0x7f090c003fb0 2023-11-28T17:48:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.926+0000 7f0922a59640 1 -- 172.21.15.98:0/2535973362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091c159e30 con 0x7f090c003fb0 2023-11-28T17:48:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.927+0000 7f0922a59640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f091c0ce8b0 0x7f091c15c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.927+0000 7f0922a59640 1 -- 172.21.15.98:0/2535973362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c0c3f70 con 0x7f091c0ce8b0 2023-11-28T17:48:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.928+0000 7f0921256640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f091c0ce8b0 0x7f091c15c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.928+0000 7f0921256640 1 -- 172.21.15.98:0/2535973362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f091c15dd90 con 0x7f091c0ce8b0 2023-11-28T17:48:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.928+0000 7f0921256640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f091c0ce8b0 0x7f091c15c420 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0908006fd0 tx=0x7f0908003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.929+0000 7f091a7fc640 1 -- 172.21.15.98:0/2535973362 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908030630 con 0x7f091c0ce8b0 2023-11-28T17:48:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.929+0000 7f091a7fc640 1 -- 172.21.15.98:0/2535973362 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f09100075c0 con 0x7f090c003fb0 2023-11-28T17:48:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.930+0000 7f091a7fc640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f090006b370 0x7f090006d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.930+0000 7f091a7fc640 1 -- 172.21.15.98:0/2535973362 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f091008a090 con 0x7f090c003fb0 2023-11-28T17:48:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.930+0000 7f091a7fc640 1 -- 172.21.15.98:0/2535973362 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f0908006e10 con 0x7f091c0ce8b0 2023-11-28T17:48:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.930+0000 7f091a7fc640 1 -- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f091c0ce8b0 msgr2=0x7f091c15c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.930+0000 7f0922258640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f090006b370 0x7f090006d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:50.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.930+0000 7f091a7fc640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f091c0ce8b0 0x7f091c15c420 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0908006fd0 tx=0x7f0908003f50 comp rx=0 tx=0).stop 2023-11-28T17:48:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.930+0000 7f091a7fc640 1 -- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f091c0ce8b0 msgr2=0x7f091c15c420 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.930+0000 7f091a7fc640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f091c0ce8b0 0x7f091c15c420 secure :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0908006fd0 tx=0x7f0908003f50 comp rx=0 tx=0).stop 2023-11-28T17:48:50.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.931+0000 7f0922258640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f090006b370 0x7f090006d830 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f091c0c8040 tx=0x7f0914008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.937+0000 7f0922a59640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f091c15c960 0x7f091c14b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.937+0000 7f0922a59640 1 -- 172.21.15.98:0/2535973362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c068170 con 0x7f091c15c960 2023-11-28T17:48:50.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.938+0000 7f0921256640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f091c15c960 0x7f091c14b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.938+0000 7f0921256640 1 -- 172.21.15.98:0/2535973362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f091c15dab0 con 0x7f091c15c960 2023-11-28T17:48:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.938+0000 7f0921256640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f091c15c960 0x7f091c14b080 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0908000c00 tx=0x7f0908032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.939+0000 7f091a7fc640 1 -- 172.21.15.98:0/2535973362 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090804ce30 con 0x7f091c15c960 2023-11-28T17:48:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.939+0000 7f091a7fc640 1 -- 172.21.15.98:0/2535973362 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4978 (secure 0 0 0) 0x7f090804b2a0 con 0x7f091c15c960 2023-11-28T17:48:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.939+0000 7f091a7fc640 1 -- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f091c15c960 msgr2=0x7f091c14b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:50.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.939+0000 7f091a7fc640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f091c15c960 0x7f091c14b080 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0908000c00 tx=0x7f0908032000 comp rx=0 tx=0).stop 2023-11-28T17:48:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.939+0000 7f091a7fc640 1 -- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f091c15c960 msgr2=0x7f091c14b080 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:50.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.939+0000 7f091a7fc640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f091c15c960 0x7f091c14b080 secure :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0908000c00 tx=0x7f0908032000 comp rx=0 tx=0).stop 2023-11-28T17:48:50.941 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:48:50.941 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:48:50.941 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:48:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:48:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2023-11-28T17:48:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:48:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:48:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:48:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:48:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 14, 2023-11-28T17:48:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:48:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:48:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:48:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:48:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:48:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:48:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:48:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:48:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:48:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:48:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:48:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:48:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:48:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:48:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:48:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:48:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:48:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:48:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:48:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:48:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:48:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:48:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:48:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:48:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:48:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:48:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:48:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:48:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:48:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:48:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:48:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:48:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:48:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:48:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:48:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:48:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:48:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:48:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:48:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:48:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:48:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:48:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:48:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:48:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 6 2023-11-28T17:48:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:48:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:48:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:48:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:48:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:48:50.967 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:48:50.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.939+0000 7f0922a59640 1 -- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f090006b370 msgr2=0x7f090006d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.939+0000 7f0922a59640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f090006b370 0x7f090006d830 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f091c0c8040 tx=0x7f0914008040 comp rx=0 tx=0).stop 2023-11-28T17:48:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.939+0000 7f0922a59640 1 -- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003fb0 msgr2=0x7f091c15a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.939+0000 7f0922a59640 1 --2- 172.21.15.98:0/2535973362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f090c003fb0 0x7f091c15a630 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f09100021c0 tx=0x7f09100021f0 comp rx=0 tx=0).stop 2023-11-28T17:48:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.939+0000 7f0922258640 1 -- 172.21.15.98:0/2535973362 reap_dead start 2023-11-28T17:48:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.940+0000 7f0922258640 1 -- 172.21.15.98:0/2535973362 reap_dead start 2023-11-28T17:48:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.940+0000 7f0922a59640 1 -- 172.21.15.98:0/2535973362 shutdown_connections 2023-11-28T17:48:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.940+0000 7f0922a59640 1 -- 172.21.15.98:0/2535973362 >> 172.21.15.98:0/2535973362 conn(0x7f091c0cf300 msgr2=0x7f091c0cab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.940+0000 7f0922a59640 1 -- 172.21.15.98:0/2535973362 shutdown_connections 2023-11-28T17:48:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:50.940+0000 7f0922a59640 1 -- 172.21.15.98:0/2535973362 wait complete. 2023-11-28T17:48:50.970 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:48:50.970 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:48:50.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:48:51.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.178+0000 7f73755a9640 1 Processor -- start 2023-11-28T17:48:51.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.179+0000 7f73755a9640 1 -- start start 2023-11-28T17:48:51.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.180+0000 7f73755a9640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7370151be0 0x7f73700c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.180+0000 7f73755a9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73700c8150 con 0x7f73700c5a60 2023-11-28T17:48:51.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.180+0000 7f73755a9640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73700c8850 con 0x7f73701584d0 2023-11-28T17:48:51.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.180+0000 7f73755a9640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73700c5f50 con 0x7f7370151be0 2023-11-28T17:48:51.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.180+0000 7f736e7fc640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f73700c5a60 0x7f73700c4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53450/0 (socket says 172.21.15.98:53450) 2023-11-28T17:48:51.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.180+0000 7f736e7fc640 1 -- 172.21.15.98:0/1274582386 learned_addr learned my addr 172.21.15.98:0/1274582386 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:48:51.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736dffb640 1 -- 172.21.15.98:0/1274582386 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649777168 0 0) 0x7f73700c8150 con 0x7f73700c5a60 2023-11-28T17:48:51.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736effd640 1 --2- 172.21.15.98:0/1274582386 >> v2:172.21.15.113:3300/0 conn(0x7f7370151be0 0x7f73700c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736dffb640 1 -- 172.21.15.98:0/1274582386 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f735c003520 con 0x7f73700c5a60 2023-11-28T17:48:51.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736dffb640 1 -- 172.21.15.98:0/1274582386 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3901404255 0 0) 0x7f735c003520 con 0x7f73700c5a60 2023-11-28T17:48:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736dffb640 1 -- 172.21.15.98:0/1274582386 >> v2:172.21.15.113:3300/0 conn(0x7f7370151be0 msgr2=0x7f73700c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736dffb640 1 --2- 172.21.15.98:0/1274582386 >> v2:172.21.15.113:3300/0 conn(0x7f7370151be0 0x7f73700c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736dffb640 1 -- 172.21.15.98:0/1274582386 >> v1:172.21.15.113:6789/0 conn(0x7f73701584d0 legacy=0x7f737006dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736dffb640 1 -- 172.21.15.98:0/1274582386 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73700c7b90 con 0x7f73700c5a60 2023-11-28T17:48:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736dffb640 1 -- 172.21.15.98:0/1274582386 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f7358001670 con 0x7f73700c5a60 2023-11-28T17:48:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736dffb640 1 -- 172.21.15.98:0/1274582386 >> v1:172.21.15.98:6789/0 conn(0x7f73700c5a60 legacy=0x7f73700c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736dffb640 1 --2- 172.21.15.98:0/1274582386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f735c003970 0x7f735c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736dffb640 1 --2- 172.21.15.98:0/1274582386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735c0047c0 0x7f735c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736dffb640 1 --2- 172.21.15.98:0/1274582386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f735c005180 0x7f735c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736dffb640 1 -- 172.21.15.98:0/1274582386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73700c5f50 con 0x7f735c0047c0 2023-11-28T17:48:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736dffb640 1 -- 172.21.15.98:0/1274582386 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c00c0b0 con 0x7f735c003970 2023-11-28T17:48:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.181+0000 7f736dffb640 1 -- 172.21.15.98:0/1274582386 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c00c270 con 0x7f735c005180 2023-11-28T17:48:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.182+0000 7f736e7fc640 1 --2- 172.21.15.98:0/1274582386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735c0047c0 0x7f735c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.182+0000 7f736f7fe640 1 --2- 172.21.15.98:0/1274582386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f735c005180 0x7f735c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.182+0000 7f736e7fc640 1 -- 172.21.15.98:0/1274582386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f735c005180 msgr2=0x7f735c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.182+0000 7f736e7fc640 1 --2- 172.21.15.98:0/1274582386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f735c005180 0x7f735c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.182+0000 7f736e7fc640 1 -- 172.21.15.98:0/1274582386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f735c003970 msgr2=0x7f735c003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.182+0000 7f736e7fc640 1 --2- 172.21.15.98:0/1274582386 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f735c003970 0x7f735c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.182+0000 7f736e7fc640 1 -- 172.21.15.98:0/1274582386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c003520 con 0x7f735c0047c0 2023-11-28T17:48:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.182+0000 7f736f7fe640 1 --2- 172.21.15.98:0/1274582386 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f735c005180 0x7f735c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:48:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.182+0000 7f736f7fe640 1 -- 172.21.15.98:0/1274582386 reap_dead start 2023-11-28T17:48:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.183+0000 7f736e7fc640 1 --2- 172.21.15.98:0/1274582386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735c0047c0 0x7f735c004c40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f735800c210 tx=0x7f735c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450d8f1d662882cf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.183+0000 7f736dffb640 1 -- 172.21.15.98:0/1274582386 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7358005ad0 con 0x7f735c0047c0 2023-11-28T17:48:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.183+0000 7f736dffb640 1 -- 172.21.15.98:0/1274582386 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7358005c30 con 0x7f735c0047c0 2023-11-28T17:48:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.183+0000 7f736dffb640 1 -- 172.21.15.98:0/1274582386 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735802c870 con 0x7f735c0047c0 2023-11-28T17:48:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.184+0000 7f73755a9640 1 -- 172.21.15.98:0/1274582386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735c0047c0 msgr2=0x7f735c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.184+0000 7f73755a9640 1 --2- 172.21.15.98:0/1274582386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735c0047c0 0x7f735c004c40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f735800c210 tx=0x7f735c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.184+0000 7f73755a9640 1 -- 172.21.15.98:0/1274582386 shutdown_connections 2023-11-28T17:48:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.184+0000 7f73755a9640 1 --2- 172.21.15.98:0/1274582386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f735c0047c0 0x7f735c004c40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.184+0000 7f73755a9640 1 -- 172.21.15.98:0/1274582386 >> 172.21.15.98:0/1274582386 conn(0x7f737006c0f0 msgr2=0x7f737006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.184+0000 7f73755a9640 1 -- 172.21.15.98:0/1274582386 shutdown_connections 2023-11-28T17:48:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.184+0000 7f73755a9640 1 -- 172.21.15.98:0/1274582386 wait complete. 2023-11-28T17:48:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.186+0000 7f73755a9640 1 Processor -- start 2023-11-28T17:48:51.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.186+0000 7f73755a9640 1 -- start start 2023-11-28T17:48:51.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.187+0000 7f73755a9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f735c0047c0 0x7f73701f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.187+0000 7f73755a9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73701f0b20 0x7f7370153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.187+0000 7f73755a9640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7370153c90 0x7f7370145760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.187+0000 7f73755a9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73701f12f0 con 0x7f73701f0b20 2023-11-28T17:48:51.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.187+0000 7f73755a9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7370148b50 con 0x7f735c0047c0 2023-11-28T17:48:51.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.187+0000 7f73755a9640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73701475c0 con 0x7f7370153c90 2023-11-28T17:48:51.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.187+0000 7f736e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73701f0b20 0x7f7370153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.187+0000 7f736e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73701f0b20 0x7f7370153750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49554/0 (socket says 172.21.15.98:49554) 2023-11-28T17:48:51.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.187+0000 7f736e7fc640 1 -- 172.21.15.98:0/4040685880 learned_addr learned my addr 172.21.15.98:0/4040685880 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:51.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.188+0000 7f736f7fe640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7370153c90 0x7f7370145760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.188+0000 7f736e7fc640 1 -- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7370153c90 msgr2=0x7f7370145760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.188+0000 7f736e7fc640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7370153c90 0x7f7370145760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.188+0000 7f736e7fc640 1 -- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f735c0047c0 msgr2=0x7f73701f05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.188+0000 7f736effd640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f735c0047c0 0x7f73701f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.188+0000 7f736e7fc640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f735c0047c0 0x7f73701f05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.188+0000 7f736e7fc640 1 -- 172.21.15.98:0/4040685880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73701f1010 con 0x7f73701f0b20 2023-11-28T17:48:51.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.188+0000 7f736e7fc640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73701f0b20 0x7f7370153750 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7358001930 tx=0x7f735802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:51.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.188+0000 7f73677fe640 1 -- 172.21.15.98:0/4040685880 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735803c040 con 0x7f73701f0b20 2023-11-28T17:48:51.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.188+0000 7f73677fe640 1 -- 172.21.15.98:0/4040685880 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735802cd40 con 0x7f73701f0b20 2023-11-28T17:48:51.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.188+0000 7f73755a9640 1 -- 172.21.15.98:0/4040685880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73701487e0 con 0x7f73701f0b20 2023-11-28T17:48:51.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.188+0000 7f73677fe640 1 -- 172.21.15.98:0/4040685880 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73580072b0 con 0x7f73701f0b20 2023-11-28T17:48:51.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.188+0000 7f73755a9640 1 -- 172.21.15.98:0/4040685880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7370156380 con 0x7f73701f0b20 2023-11-28T17:48:51.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.189+0000 7f73755a9640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7370152db0 0x7f7370145ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.189+0000 7f73755a9640 1 -- 172.21.15.98:0/4040685880 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73700744d0 con 0x7f7370152db0 2023-11-28T17:48:51.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.190+0000 7f736effd640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7370152db0 0x7f7370145ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.191+0000 7f73677fe640 1 -- 172.21.15.98:0/4040685880 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f7358032070 con 0x7f73701f0b20 2023-11-28T17:48:51.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.191+0000 7f736effd640 1 -- 172.21.15.98:0/4040685880 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7370158920 con 0x7f7370152db0 2023-11-28T17:48:51.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.191+0000 7f736effd640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7370152db0 0x7f7370145ca0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7360007d40 tx=0x7f736000f520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:51.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.191+0000 7f73677fe640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f735006b2a0 0x7f735006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.192+0000 7f73677fe640 1 -- 172.21.15.98:0/4040685880 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f73580ab060 con 0x7f73701f0b20 2023-11-28T17:48:51.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.192+0000 7f736f7fe640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f735006b2a0 0x7f735006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.192+0000 7f73677fe640 1 -- 172.21.15.98:0/4040685880 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73600145f0 con 0x7f7370152db0 2023-11-28T17:48:51.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.192+0000 7f736f7fe640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f735006b2a0 0x7f735006d760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f73700c5240 tx=0x7f7368009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:51.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.192+0000 7f73677fe640 1 -- 172.21.15.98:0/4040685880 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f736000feb0 con 0x7f7370152db0 2023-11-28T17:48:51.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.193+0000 7f73677fe640 1 -- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7370152db0 msgr2=0x7f7370145ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.193+0000 7f73677fe640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7370152db0 0x7f7370145ca0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7360007d40 tx=0x7f736000f520 comp rx=0 tx=0).stop 2023-11-28T17:48:51.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.193+0000 7f73677fe640 1 -- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7370152db0 msgr2=0x7f7370145ca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:51.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.193+0000 7f73677fe640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7370152db0 0x7f7370145ca0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7360007d40 tx=0x7f736000f520 comp rx=0 tx=0).stop 2023-11-28T17:48:51.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.200+0000 7f73755a9640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73700c1730 0x7f73701461e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.200+0000 7f73755a9640 1 -- 172.21.15.98:0/4040685880 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73700c84e0 con 0x7f73700c1730 2023-11-28T17:48:51.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.200+0000 7f736effd640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73700c1730 0x7f73701461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.201+0000 7f736effd640 1 -- 172.21.15.98:0/4040685880 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7370151fc0 con 0x7f73700c1730 2023-11-28T17:48:51.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.202+0000 7f736effd640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73700c1730 0x7f73701461e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7360007d40 tx=0x7f7360007040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:51.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.202+0000 7f73677fe640 1 -- 172.21.15.98:0/4040685880 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360014bc0 con 0x7f73700c1730 2023-11-28T17:48:51.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.203+0000 7f73677fe640 1 -- 172.21.15.98:0/4040685880 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4806 (secure 0 0 0) 0x7f7360014d20 con 0x7f73700c1730 2023-11-28T17:48:51.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.203+0000 7f73677fe640 1 -- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73700c1730 msgr2=0x7f73701461e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.203+0000 7f73677fe640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73700c1730 0x7f73701461e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7360007d40 tx=0x7f7360007040 comp rx=0 tx=0).stop 2023-11-28T17:48:51.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.203+0000 7f73677fe640 1 -- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73700c1730 msgr2=0x7f73701461e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:51.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.203+0000 7f73677fe640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73700c1730 0x7f73701461e0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7360007d40 tx=0x7f7360007040 comp rx=0 tx=0).stop 2023-11-28T17:48:51.204 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:48:51.204 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:48:51.204 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:48:51.204 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:48:51.204 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2023-11-28T17:48:51.205 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:48:51.205 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:48:51.205 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:48:51.205 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:48:51.205 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:51.205 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 15, 2023-11-28T17:48:51.205 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:51.205 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:48:51.205 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:48:51.205 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:51.205 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:51.205 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:51.205 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:51.205 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:51.206 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:51.206 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:51.206 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:51.206 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:51.206 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:51.206 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:51.206 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:48:51.206 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:48:51.206 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:51.206 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:51.206 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:51.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:51.206 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:51.206 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:51.207 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:51.207 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:51.207 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:51.207 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:51.207 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:51.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.207 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:48:51.207 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:48:51.207 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:48:51.207 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:48:51.207 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:48:51.207 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:48:51.207 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:51.208 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:51.208 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:48:51.208 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:48:51.208 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:48:51.208 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:48:51.208 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:48:51.208 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:48:51.208 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:48:51.208 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:51.208 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:48:51.208 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:51.208 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:51.208 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:51.208 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:51.209 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:51.209 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:51.209 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:51.209 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:51.209 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:51.209 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:51.209 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:51.209 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:48:51.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.209 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:48:51.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.209 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:48:51.209 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:48:51.209 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:51.210 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:51.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:51.210 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:48:51.210 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:51.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:51.210 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:48:51.210 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:51.210 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:51.210 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:48:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:48:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:48:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:48:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:51.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:51.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:48:51.212 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:51.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:51.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:48:51.212 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:51.212 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:51.213 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:51.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.213 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:48:51.213 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:48:51.213 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:51.213 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:51.213 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:51.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.213 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:48:51.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:48:51.213 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:51.214 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:51.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:51.214 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:48:51.214 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:51.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:51.214 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:48:51.214 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:51.214 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:51.214 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:51.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.215 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:48:51.215 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:48:51.215 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:51.215 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:51.215 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:51.215 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:51.215 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:51.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.215 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:48:51.215 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:48:51.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.216 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:51.216 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:51.216 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:48:51.216 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:51.216 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:51.216 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:48:51.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.216 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:51.216 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:51.216 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:48:51.216 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:51.216 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:51.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:48:51.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.217 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:51.217 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:51.217 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:48:51.217 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:51.217 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:51.217 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:48:51.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.217 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:51.217 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:51.217 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:48:51.217 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:51.218 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:51.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.218 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:48:51.218 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:48:51.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.203+0000 7f73755a9640 1 -- 172.21.15.98:0/4040685880 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f735006b2a0 msgr2=0x7f735006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.203+0000 7f73755a9640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f735006b2a0 0x7f735006d760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f73700c5240 tx=0x7f7368009040 comp rx=0 tx=0).stop 2023-11-28T17:48:51.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.203+0000 7f73755a9640 1 -- 172.21.15.98:0/4040685880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73701f0b20 msgr2=0x7f7370153750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.203+0000 7f73755a9640 1 --2- 172.21.15.98:0/4040685880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73701f0b20 0x7f7370153750 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7358001930 tx=0x7f735802e070 comp rx=0 tx=0).stop 2023-11-28T17:48:51.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.203+0000 7f736f7fe640 1 -- 172.21.15.98:0/4040685880 reap_dead start 2023-11-28T17:48:51.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.203+0000 7f736f7fe640 1 -- 172.21.15.98:0/4040685880 reap_dead start 2023-11-28T17:48:51.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.204+0000 7f73755a9640 1 -- 172.21.15.98:0/4040685880 shutdown_connections 2023-11-28T17:48:51.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.204+0000 7f73755a9640 1 -- 172.21.15.98:0/4040685880 >> 172.21.15.98:0/4040685880 conn(0x7f737006c0f0 msgr2=0x7f73700710b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:51.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.204+0000 7f73755a9640 1 -- 172.21.15.98:0/4040685880 shutdown_connections 2023-11-28T17:48:51.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.204+0000 7f73755a9640 1 -- 172.21.15.98:0/4040685880 wait complete. 2023-11-28T17:48:51.220 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:48:51.221 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:48:51.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:48:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.427+0000 7f0d53fff640 1 Processor -- start 2023-11-28T17:48:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.428+0000 7f0d53fff640 1 -- start start 2023-11-28T17:48:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.429+0000 7f0d53fff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f0d54153c80 0x7f0d5406b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.429+0000 7f0d53fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d540c69b0 0x7f0d5406be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.429+0000 7f0d53fff640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d5415d230 con 0x7f0d540c7720 2023-11-28T17:48:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.429+0000 7f0d53fff640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5415e070 con 0x7f0d540c69b0 2023-11-28T17:48:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.429+0000 7f0d53fff640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5415d4f0 con 0x7f0d54153c80 2023-11-28T17:48:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.429+0000 7f0d527fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d540c69b0 0x7f0d5406be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.429+0000 7f0d527fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d540c69b0 0x7f0d5406be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49560/0 (socket says 172.21.15.98:49560) 2023-11-28T17:48:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.429+0000 7f0d527fc640 1 -- 172.21.15.98:0/3769023492 learned_addr learned my addr 172.21.15.98:0/3769023492 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.430+0000 7f0d527fc640 1 -- 172.21.15.98:0/3769023492 >> v2:172.21.15.113:3300/0 conn(0x7f0d54153c80 msgr2=0x7f0d5406b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.430+0000 7f0d527fc640 1 --2- 172.21.15.98:0/3769023492 >> v2:172.21.15.113:3300/0 conn(0x7f0d54153c80 0x7f0d5406b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.430+0000 7f0d527fc640 1 -- 172.21.15.98:0/3769023492 >> v1:172.21.15.113:6789/0 conn(0x7f0d540c7720 legacy=0x7f0d540c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.430+0000 7f0d527fc640 1 -- 172.21.15.98:0/3769023492 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5415d7d0 con 0x7f0d540c69b0 2023-11-28T17:48:51.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.431+0000 7f0d527fc640 1 --2- 172.21.15.98:0/3769023492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d540c69b0 0x7f0d5406be80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0d5406c630 tx=0x7f0d3c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a014ae2edb28d6a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.431+0000 7f0d51ffb640 1 -- 172.21.15.98:0/3769023492 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c030860 con 0x7f0d540c69b0 2023-11-28T17:48:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.431+0000 7f0d51ffb640 1 -- 172.21.15.98:0/3769023492 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3c0309c0 con 0x7f0d540c69b0 2023-11-28T17:48:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.431+0000 7f0d51ffb640 1 -- 172.21.15.98:0/3769023492 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c030d50 con 0x7f0d540c69b0 2023-11-28T17:48:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.432+0000 7f0d53fff640 1 -- 172.21.15.98:0/3769023492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d540c69b0 msgr2=0x7f0d5406be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.432+0000 7f0d53fff640 1 --2- 172.21.15.98:0/3769023492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d540c69b0 0x7f0d5406be80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0d5406c630 tx=0x7f0d3c02b9a0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.432+0000 7f0d53fff640 1 -- 172.21.15.98:0/3769023492 shutdown_connections 2023-11-28T17:48:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.432+0000 7f0d53fff640 1 --2- 172.21.15.98:0/3769023492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d540c69b0 0x7f0d5406be80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.432+0000 7f0d53fff640 1 --2- 172.21.15.98:0/3769023492 >> v2:172.21.15.113:3300/0 conn(0x7f0d54153c80 0x7f0d5406b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.432+0000 7f0d53fff640 1 -- 172.21.15.98:0/3769023492 >> 172.21.15.98:0/3769023492 conn(0x7f0d540cf300 msgr2=0x7f0d540cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.432+0000 7f0d53fff640 1 -- 172.21.15.98:0/3769023492 shutdown_connections 2023-11-28T17:48:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.432+0000 7f0d53fff640 1 -- 172.21.15.98:0/3769023492 wait complete. 2023-11-28T17:48:51.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.434+0000 7f0d53fff640 1 Processor -- start 2023-11-28T17:48:51.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.434+0000 7f0d53fff640 1 -- start start 2023-11-28T17:48:51.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.435+0000 7f0d53fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d540c69b0 0x7f0d54157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.435+0000 7f0d53fff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d540c7720 0x7f0d54154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.435+0000 7f0d53fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d54153c80 0x7f0d541efd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.435+0000 7f0d53fff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d540ce800 con 0x7f0d54153c80 2023-11-28T17:48:51.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.435+0000 7f0d53fff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d541ef8d0 con 0x7f0d540c69b0 2023-11-28T17:48:51.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.435+0000 7f0d53fff640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d54158d20 con 0x7f0d540c7720 2023-11-28T17:48:51.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.435+0000 7f0d537fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d54153c80 0x7f0d541efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.435+0000 7f0d537fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d54153c80 0x7f0d541efd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49572/0 (socket says 172.21.15.98:49572) 2023-11-28T17:48:51.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.435+0000 7f0d537fe640 1 -- 172.21.15.98:0/1549733073 learned_addr learned my addr 172.21.15.98:0/1549733073 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:51.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.436+0000 7f0d527fc640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d540c7720 0x7f0d54154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.436+0000 7f0d52ffd640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d540c69b0 0x7f0d54157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.436+0000 7f0d537fe640 1 -- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d540c7720 msgr2=0x7f0d54154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.436+0000 7f0d537fe640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d540c7720 0x7f0d54154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.436+0000 7f0d537fe640 1 -- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d540c69b0 msgr2=0x7f0d54157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.436+0000 7f0d537fe640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d540c69b0 0x7f0d54157880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.436+0000 7f0d537fe640 1 -- 172.21.15.98:0/1549733073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c0094d0 con 0x7f0d54153c80 2023-11-28T17:48:51.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.436+0000 7f0d527fc640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d540c7720 0x7f0d54154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:48:51.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.436+0000 7f0d537fe640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d54153c80 0x7f0d541efd60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00c3c0 tx=0x7f0d4c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:51.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.436+0000 7f0d4b7fe640 1 -- 172.21.15.98:0/1549733073 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c0035f0 con 0x7f0d54153c80 2023-11-28T17:48:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.436+0000 7f0d4b7fe640 1 -- 172.21.15.98:0/1549733073 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d4c003750 con 0x7f0d54153c80 2023-11-28T17:48:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.436+0000 7f0d4b7fe640 1 -- 172.21.15.98:0/1549733073 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c003b00 con 0x7f0d54153c80 2023-11-28T17:48:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.436+0000 7f0d53fff640 1 -- 172.21.15.98:0/1549733073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d541f37f0 con 0x7f0d54153c80 2023-11-28T17:48:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.436+0000 7f0d53fff640 1 -- 172.21.15.98:0/1549733073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d540cea50 con 0x7f0d54153c80 2023-11-28T17:48:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.437+0000 7f0d53fff640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d5415a960 0x7f0d541f02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.437+0000 7f0d53fff640 1 -- 172.21.15.98:0/1549733073 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d54068170 con 0x7f0d5415a960 2023-11-28T17:48:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.438+0000 7f0d52ffd640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d5415a960 0x7f0d541f02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.438+0000 7f0d4b7fe640 1 -- 172.21.15.98:0/1549733073 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0d4c003c60 con 0x7f0d54153c80 2023-11-28T17:48:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.439+0000 7f0d52ffd640 1 -- 172.21.15.98:0/1549733073 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d540cbb10 con 0x7f0d5415a960 2023-11-28T17:48:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.439+0000 7f0d52ffd640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d5415a960 0x7f0d541f02a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400cad0 tx=0x7f0d4400cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.439+0000 7f0d4b7fe640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d3406b2d0 0x7f0d3406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.440+0000 7f0d4b7fe640 1 -- 172.21.15.98:0/1549733073 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0d4c088d10 con 0x7f0d54153c80 2023-11-28T17:48:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.440+0000 7f0d527fc640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d3406b2d0 0x7f0d3406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.440+0000 7f0d4b7fe640 1 -- 172.21.15.98:0/1549733073 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4400ac30 con 0x7f0d5415a960 2023-11-28T17:48:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.440+0000 7f0d4b7fe640 1 -- 172.21.15.98:0/1549733073 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f0d4400ad90 con 0x7f0d5415a960 2023-11-28T17:48:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.440+0000 7f0d4b7fe640 1 -- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d5415a960 msgr2=0x7f0d541f02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.440+0000 7f0d4b7fe640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d5415a960 0x7f0d541f02a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400cad0 tx=0x7f0d4400cf90 comp rx=0 tx=0).stop 2023-11-28T17:48:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.440+0000 7f0d4b7fe640 1 -- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d5415a960 msgr2=0x7f0d541f02a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:51.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.440+0000 7f0d4b7fe640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d5415a960 0x7f0d541f02a0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400cad0 tx=0x7f0d4400cf90 comp rx=0 tx=0).stop 2023-11-28T17:48:51.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.440+0000 7f0d527fc640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d3406b2d0 0x7f0d3406d790 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c036000 tx=0x7f0d3c03b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:51.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.447+0000 7f0d53fff640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d540cec40 0x7f0d5414b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.447+0000 7f0d53fff640 1 -- 172.21.15.98:0/1549733073 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5415dd90 con 0x7f0d540cec40 2023-11-28T17:48:51.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.448+0000 7f0d52ffd640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d540cec40 0x7f0d5414b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.448+0000 7f0d52ffd640 1 -- 172.21.15.98:0/1549733073 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0d540b50d0 con 0x7f0d540cec40 2023-11-28T17:48:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.449+0000 7f0d52ffd640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d540cec40 0x7f0d5414b3d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0d44017000 tx=0x7f0d44004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:51.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.449+0000 7f0d4b7fe640 1 -- 172.21.15.98:0/1549733073 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44028070 con 0x7f0d540cec40 2023-11-28T17:48:51.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.450+0000 7f0d4b7fe640 1 -- 172.21.15.98:0/1549733073 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4466 (secure 0 0 0) 0x7f0d44020040 con 0x7f0d540cec40 2023-11-28T17:48:51.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.450+0000 7f0d4b7fe640 1 -- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d540cec40 msgr2=0x7f0d5414b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.450+0000 7f0d4b7fe640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d540cec40 0x7f0d5414b3d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0d44017000 tx=0x7f0d44004ab0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.450+0000 7f0d4b7fe640 1 -- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d540cec40 msgr2=0x7f0d5414b3d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:48:51.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.450+0000 7f0d4b7fe640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f0d540cec40 0x7f0d5414b3d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.451 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:48:51.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:48:51.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:48:51.452 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:48:51.452 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2023-11-28T17:48:51.452 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:48:51.452 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:48:51.452 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:48:51.452 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:48:51.452 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:51.452 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 15, 2023-11-28T17:48:51.452 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:51.452 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:48:51.452 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:48:51.452 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:51.452 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:51.453 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:51.453 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:51.453 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:51.453 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:51.453 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:51.453 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:51.453 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:51.453 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:51.453 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:51.453 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:48:51.453 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:48:51.453 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:51.454 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:51.454 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:51.454 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:51.454 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:51.454 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:51.454 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:51.454 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:51.454 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:51.454 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:51.454 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:51.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.454 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:48:51.455 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:48:51.455 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:48:51.455 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:48:51.455 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:48:51.455 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:48:51.455 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:51.455 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:48:51.455 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:48:51.455 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:48:51.455 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:48:51.455 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:48:51.455 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:48:51.456 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:48:51.456 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:48:51.456 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:48:51.456 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:48:51.456 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:48:51.456 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:48:51.456 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:48:51.456 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:48:51.456 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:48:51.456 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:48:51.456 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:48:51.456 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:48:51.456 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:48:51.457 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:48:51.457 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:51.457 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:48:51.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.457 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:48:51.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:48:51.457 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:48:51.457 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:51.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:51.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:51.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:48:51.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:51.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:51.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:48:51.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:51.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:51.458 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:51.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:48:51.458 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:48:51.459 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:51.459 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:51.459 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:51.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:48:51.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:48:51.459 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:51.459 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:51.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:51.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:48:51.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:51.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:51.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:48:51.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:51.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:51.460 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:51.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:48:51.460 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:48:51.460 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:51.461 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:51.461 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:51.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:48:51.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:48:51.461 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:48:51.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:48:51.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:48:51.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:48:51.462 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:51.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:48:51.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:48:51.462 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:48:51.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:51.462 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:51.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:48:51.463 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:48:51.463 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:48:51.463 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:48:51.463 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:48:51.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:51.463 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:51.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.463 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:48:51.463 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:48:51.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.463 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:51.463 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:51.463 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:48:51.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:51.464 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:48:51.464 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:48:51.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:48:51.464 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:48:51.464 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:48:51.464 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:48:51.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:48:51.464 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:48:51.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:48:51.464 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:48:51.464 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:48:51.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.450+0000 7f0d53fff640 1 -- 172.21.15.98:0/1549733073 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d3406b2d0 msgr2=0x7f0d3406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.450+0000 7f0d53fff640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f0d3406b2d0 0x7f0d3406d790 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c036000 tx=0x7f0d3c03b040 comp rx=0 tx=0).stop 2023-11-28T17:48:51.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.450+0000 7f0d53fff640 1 -- 172.21.15.98:0/1549733073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d54153c80 msgr2=0x7f0d541efd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.450+0000 7f0d53fff640 1 --2- 172.21.15.98:0/1549733073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d54153c80 0x7f0d541efd60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00c3c0 tx=0x7f0d4c00c880 comp rx=0 tx=0).stop 2023-11-28T17:48:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.450+0000 7f0d537fe640 1 -- 172.21.15.98:0/1549733073 reap_dead start 2023-11-28T17:48:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.450+0000 7f0d537fe640 1 -- 172.21.15.98:0/1549733073 reap_dead start 2023-11-28T17:48:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.451+0000 7f0d53fff640 1 -- 172.21.15.98:0/1549733073 shutdown_connections 2023-11-28T17:48:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.451+0000 7f0d53fff640 1 -- 172.21.15.98:0/1549733073 >> 172.21.15.98:0/1549733073 conn(0x7f0d540cf300 msgr2=0x7f0d5406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.451+0000 7f0d53fff640 1 -- 172.21.15.98:0/1549733073 shutdown_connections 2023-11-28T17:48:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.451+0000 7f0d53fff640 1 -- 172.21.15.98:0/1549733073 wait complete. 2023-11-28T17:48:51.469 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:48:51.469 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2023-11-28T17:48:51.469 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2023-11-28T17:48:51.470 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2023-11-28T17:48:51.470 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-11-28T17:48:51.470 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-28T17:48:51.470 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:48:51.507 INFO:tasks.ceph.mon.a:Stopped 2023-11-28T17:48:51.507 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2023-11-28T17:48:51.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:48:51.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.758+0000 7f686d47e640 1 Processor -- start 2023-11-28T17:48:51.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.758+0000 7f686d47e640 1 -- start start 2023-11-28T17:48:51.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.760+0000 7f686d47e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68681573c0 0x7f686815d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.760+0000 7f686d47e640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f68680c7ed0 0x7f68680c8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.760+0000 7f686d47e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f686806a4d0 0x7f686806ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.760+0000 7f686d47e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68681577a0 con 0x7f686806a4d0 2023-11-28T17:48:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.760+0000 7f686d47e640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68680c6a10 con 0x7f68680c7ed0 2023-11-28T17:48:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.760+0000 7f686d47e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68680c84e0 con 0x7f68681573c0 2023-11-28T17:48:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.760+0000 7f686cc7d640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f686806a4d0 msgr2=0x7f686806ad70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:48:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.760+0000 7f686cc7d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f686806a4d0 0x7f686806ad70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.761+0000 7f68677fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f68680c7ed0 0x7f68680c8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.761+0000 7f6867fff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68681573c0 0x7f686815d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.761+0000 7f68677fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f68680c7ed0 0x7f68680c8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:60630/0 (socket says 172.21.15.98:60630) 2023-11-28T17:48:51.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.761+0000 7f68677fe640 1 -- 172.21.15.98:0/690493387 learned_addr learned my addr 172.21.15.98:0/690493387 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.761+0000 7f68677fe640 1 -- 172.21.15.98:0/690493387 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68681573c0 msgr2=0x7f686815d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:48:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.761+0000 7f68677fe640 1 --2- 172.21.15.98:0/690493387 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68681573c0 0x7f686815d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.761+0000 7f68677fe640 1 -- 172.21.15.98:0/690493387 >> v2:172.21.15.98:3300/0 conn(0x7f686806a4d0 msgr2=0x7f686806ad70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:48:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.761+0000 7f68677fe640 1 --2- 172.21.15.98:0/690493387 >> v2:172.21.15.98:3300/0 conn(0x7f686806a4d0 0x7f686806ad70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.761+0000 7f68677fe640 1 -- 172.21.15.98:0/690493387 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68680c8aa0 con 0x7f68680c7ed0 2023-11-28T17:48:51.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.763+0000 7f68677fe640 1 --2- 172.21.15.98:0/690493387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68680c7ed0 0x7f68680c8f30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f686806fb10 tx=0x7f685802bf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a31b7ba0365b379 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.763+0000 7f6866ffd640 1 -- 172.21.15.98:0/690493387 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685802caa0 con 0x7f68680c7ed0 2023-11-28T17:48:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.763+0000 7f6866ffd640 1 -- 172.21.15.98:0/690493387 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685802cc00 con 0x7f68680c7ed0 2023-11-28T17:48:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.763+0000 7f6866ffd640 1 -- 172.21.15.98:0/690493387 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68580343f0 con 0x7f68680c7ed0 2023-11-28T17:48:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.764+0000 7f686d47e640 1 -- 172.21.15.98:0/690493387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68680c7ed0 msgr2=0x7f68680c8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.764+0000 7f686d47e640 1 --2- 172.21.15.98:0/690493387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68680c7ed0 0x7f68680c8f30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f686806fb10 tx=0x7f685802bf00 comp rx=0 tx=0).stop 2023-11-28T17:48:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.764+0000 7f686d47e640 1 -- 172.21.15.98:0/690493387 shutdown_connections 2023-11-28T17:48:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.764+0000 7f686d47e640 1 --2- 172.21.15.98:0/690493387 >> v2:172.21.15.98:3300/0 conn(0x7f686806a4d0 0x7f686806ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.764+0000 7f686d47e640 1 --2- 172.21.15.98:0/690493387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f68680c7ed0 0x7f68680c8f30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.764+0000 7f686d47e640 1 --2- 172.21.15.98:0/690493387 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68681573c0 0x7f686815d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.764+0000 7f686d47e640 1 -- 172.21.15.98:0/690493387 >> 172.21.15.98:0/690493387 conn(0x7f686806ed10 msgr2=0x7f686806f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:51.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.764+0000 7f686d47e640 1 -- 172.21.15.98:0/690493387 shutdown_connections 2023-11-28T17:48:51.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.764+0000 7f686d47e640 1 -- 172.21.15.98:0/690493387 wait complete. 2023-11-28T17:48:51.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.766+0000 7f686d47e640 1 Processor -- start 2023-11-28T17:48:51.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.766+0000 7f686d47e640 1 -- start start 2023-11-28T17:48:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.767+0000 7f686d47e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f686806a4d0 0x7f68680712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.767+0000 7f686d47e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68680c7ed0 0x7f68681f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.767+0000 7f686d47e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68681573c0 0x7f68681f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.767+0000 7f686d47e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68681f2c30 con 0x7f68681573c0 2023-11-28T17:48:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.767+0000 7f686d47e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686815c3f0 con 0x7f686806a4d0 2023-11-28T17:48:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.767+0000 7f686d47e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68680cb9d0 con 0x7f68680c7ed0 2023-11-28T17:48:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.767+0000 7f686cc7d640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68681573c0 msgr2=0x7f68681f3600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:48:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.767+0000 7f686cc7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68681573c0 0x7f68681f3600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.767+0000 7f6867fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f686806a4d0 0x7f68680712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.767+0000 7f68677fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68680c7ed0 0x7f68681f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.767+0000 7f6867fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f686806a4d0 0x7f68680712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:60640/0 (socket says 172.21.15.98:60640) 2023-11-28T17:48:51.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.767+0000 7f6867fff640 1 -- 172.21.15.98:0/4060588358 learned_addr learned my addr 172.21.15.98:0/4060588358 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.768+0000 7f6867fff640 1 -- 172.21.15.98:0/4060588358 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68680c7ed0 msgr2=0x7f68681f30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.768+0000 7f6867fff640 1 --2- 172.21.15.98:0/4060588358 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68680c7ed0 0x7f68681f30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.768+0000 7f6867fff640 1 -- 172.21.15.98:0/4060588358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68681573c0 msgr2=0x7f68681f3600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:48:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.768+0000 7f6867fff640 1 --2- 172.21.15.98:0/4060588358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68681573c0 0x7f68681f3600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.768+0000 7f6867fff640 1 -- 172.21.15.98:0/4060588358 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6858009b30 con 0x7f686806a4d0 2023-11-28T17:48:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.768+0000 7f6867fff640 1 --2- 172.21.15.98:0/4060588358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f686806a4d0 0x7f68680712c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f685000e9b0 tx=0x7f685000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.769+0000 7f6864ff9640 1 -- 172.21.15.98:0/4060588358 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6850009670 con 0x7f686806a4d0 2023-11-28T17:48:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.769+0000 7f686d47e640 1 -- 172.21.15.98:0/4060588358 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68680cbc90 con 0x7f686806a4d0 2023-11-28T17:48:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.769+0000 7f6864ff9640 1 -- 172.21.15.98:0/4060588358 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685000c9a0 con 0x7f686806a4d0 2023-11-28T17:48:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.769+0000 7f6864ff9640 1 -- 172.21.15.98:0/4060588358 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685000cd50 con 0x7f686806a4d0 2023-11-28T17:48:51.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.769+0000 7f686d47e640 1 -- 172.21.15.98:0/4060588358 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68680c2480 con 0x7f686806a4d0 2023-11-28T17:48:51.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.770+0000 7f686d47e640 1 -- 172.21.15.98:0/4060588358 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6868074db0 con 0x7f686806a4d0 2023-11-28T17:48:51.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.771+0000 7f6864ff9640 1 -- 172.21.15.98:0/4060588358 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6850013470 con 0x7f686806a4d0 2023-11-28T17:48:51.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.772+0000 7f6864ff9640 1 --2- 172.21.15.98:0/4060588358 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f684806b250 0x7f684806d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:51.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.772+0000 7f6864ff9640 1 -- 172.21.15.98:0/4060588358 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f685008a5d0 con 0x7f686806a4d0 2023-11-28T17:48:51.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.772+0000 7f68677fe640 1 --2- 172.21.15.98:0/4060588358 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f684806b250 0x7f684806d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:51.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.773+0000 7f68677fe640 1 --2- 172.21.15.98:0/4060588358 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f684806b250 0x7f684806d710 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6858035000 tx=0x7f6858030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:51.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:51.788+0000 7f6864ff9640 1 -- 172.21.15.98:0/4060588358 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6850055470 con 0x7f686806a4d0 2023-11-28T17:48:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:52.054+0000 7f686d47e640 1 -- 172.21.15.98:0/4060588358 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f68680c72a0 con 0x7f686806a4d0 2023-11-28T17:48:52.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:52.056+0000 7f6864ff9640 1 -- 172.21.15.98:0/4060588358 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f6850058a90 con 0x7f686806a4d0 2023-11-28T17:48:52.057 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:48:52.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:52.058+0000 7f686d47e640 1 -- 172.21.15.98:0/4060588358 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f684806b250 msgr2=0x7f684806d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:52.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:52.058+0000 7f686d47e640 1 --2- 172.21.15.98:0/4060588358 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f684806b250 0x7f684806d710 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6858035000 tx=0x7f6858030000 comp rx=0 tx=0).stop 2023-11-28T17:48:52.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:52.058+0000 7f686d47e640 1 -- 172.21.15.98:0/4060588358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f686806a4d0 msgr2=0x7f68680712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:52.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:52.058+0000 7f686d47e640 1 --2- 172.21.15.98:0/4060588358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f686806a4d0 0x7f68680712c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f685000e9b0 tx=0x7f685000ee70 comp rx=0 tx=0).stop 2023-11-28T17:48:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:52.058+0000 7f686d47e640 1 -- 172.21.15.98:0/4060588358 shutdown_connections 2023-11-28T17:48:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:52.058+0000 7f686d47e640 1 --2- 172.21.15.98:0/4060588358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68681573c0 0x7f68681f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:52.058+0000 7f686d47e640 1 --2- 172.21.15.98:0/4060588358 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f68680c7ed0 0x7f68681f30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:52.058+0000 7f686d47e640 1 --2- 172.21.15.98:0/4060588358 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f684806b250 0x7f684806d710 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:52.058+0000 7f686d47e640 1 --2- 172.21.15.98:0/4060588358 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f686806a4d0 0x7f68680712c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:52.058+0000 7f686d47e640 1 -- 172.21.15.98:0/4060588358 >> 172.21.15.98:0/4060588358 conn(0x7f686806ed10 msgr2=0x7f686815d780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:52.058+0000 7f686d47e640 1 -- 172.21.15.98:0/4060588358 shutdown_connections 2023-11-28T17:48:52.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:52.058+0000 7f686d47e640 1 -- 172.21.15.98:0/4060588358 wait complete. 2023-11-28T17:48:54.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~243s 2023-11-28T17:48:55.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:48:55.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.302+0000 7f6a750d1640 1 Processor -- start 2023-11-28T17:48:55.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.303+0000 7f6a750d1640 1 -- start start 2023-11-28T17:48:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.304+0000 7f6a750d1640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6a70153e90 0x7f6a7006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.304+0000 7f6a750d1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6a700ca9b0 0x7f6a700c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.304+0000 7f6a750d1640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a7006c830 con 0x7f6a700701a0 2023-11-28T17:48:55.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.304+0000 7f6a750d1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a700c9cc0 con 0x7f6a700ca9b0 2023-11-28T17:48:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.304+0000 7f6a750d1640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a700c90d0 con 0x7f6a70153e90 2023-11-28T17:48:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.304+0000 7f6a67fff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f6a700ca9b0 msgr2=0x7f6a700c7cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:48:55.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.305+0000 7f6a67fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6a700ca9b0 0x7f6a700c7cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.305+0000 7f6a6ffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6a70153e90 0x7f6a7006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.305+0000 7f6a6ffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6a70153e90 0x7f6a7006bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:60656/0 (socket says 172.21.15.98:60656) 2023-11-28T17:48:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.305+0000 7f6a6ffff640 1 -- 172.21.15.98:0/3625647193 learned_addr learned my addr 172.21.15.98:0/3625647193 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.306+0000 7f6a6ffff640 1 -- 172.21.15.98:0/3625647193 >> v2:172.21.15.98:3300/0 conn(0x7f6a700ca9b0 msgr2=0x7f6a700c7cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:48:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.306+0000 7f6a6ffff640 1 --2- 172.21.15.98:0/3625647193 >> v2:172.21.15.98:3300/0 conn(0x7f6a700ca9b0 0x7f6a700c7cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.306+0000 7f6a6ffff640 1 -- 172.21.15.98:0/3625647193 >> v1:172.21.15.113:6789/0 conn(0x7f6a700701a0 legacy=0x7f6a70159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:55.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.306+0000 7f6a6ffff640 1 -- 172.21.15.98:0/3625647193 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7006c550 con 0x7f6a70153e90 2023-11-28T17:48:55.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.308+0000 7f6a6ffff640 1 --2- 172.21.15.98:0/3625647193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a70153e90 0x7f6a7006bf50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6a700cd250 tx=0x7f6a6002b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c62938e1e009316f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.308+0000 7f6a6f7fe640 1 -- 172.21.15.98:0/3625647193 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60031890 con 0x7f6a70153e90 2023-11-28T17:48:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.308+0000 7f6a6f7fe640 1 -- 172.21.15.98:0/3625647193 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a600319f0 con 0x7f6a70153e90 2023-11-28T17:48:55.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.308+0000 7f6a6f7fe640 1 -- 172.21.15.98:0/3625647193 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60031d80 con 0x7f6a70153e90 2023-11-28T17:48:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.309+0000 7f6a750d1640 1 -- 172.21.15.98:0/3625647193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a70153e90 msgr2=0x7f6a7006bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.309+0000 7f6a750d1640 1 --2- 172.21.15.98:0/3625647193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a70153e90 0x7f6a7006bf50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6a700cd250 tx=0x7f6a6002b930 comp rx=0 tx=0).stop 2023-11-28T17:48:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.309+0000 7f6a750d1640 1 -- 172.21.15.98:0/3625647193 shutdown_connections 2023-11-28T17:48:55.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.309+0000 7f6a750d1640 1 --2- 172.21.15.98:0/3625647193 >> v2:172.21.15.98:3300/0 conn(0x7f6a700ca9b0 0x7f6a700c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.309+0000 7f6a750d1640 1 --2- 172.21.15.98:0/3625647193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a70153e90 0x7f6a7006bf50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.309+0000 7f6a750d1640 1 -- 172.21.15.98:0/3625647193 >> 172.21.15.98:0/3625647193 conn(0x7f6a700722b0 msgr2=0x7f6a70072690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.309+0000 7f6a750d1640 1 -- 172.21.15.98:0/3625647193 shutdown_connections 2023-11-28T17:48:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.309+0000 7f6a750d1640 1 -- 172.21.15.98:0/3625647193 wait complete. 2023-11-28T17:48:55.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.311+0000 7f6a750d1640 1 Processor -- start 2023-11-28T17:48:55.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.311+0000 7f6a750d1640 1 -- start start 2023-11-28T17:48:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.312+0000 7f6a750d1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a700701a0 0x7f6a700c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.312+0000 7f6a750d1640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a700ca9b0 0x7f6a701f2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.312+0000 7f6a750d1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a70153e90 0x7f6a700cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.312+0000 7f6a750d1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a700cb570 con 0x7f6a700701a0 2023-11-28T17:48:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.312+0000 7f6a750d1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a700cbe10 con 0x7f6a70153e90 2023-11-28T17:48:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.312+0000 7f6a750d1640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a701f34e0 con 0x7f6a700ca9b0 2023-11-28T17:48:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.312+0000 7f6a6ffff640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a700701a0 msgr2=0x7f6a700c4d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:48:55.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.312+0000 7f6a6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a700701a0 0x7f6a700c4d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.313+0000 7f6a67fff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a700ca9b0 0x7f6a701f2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.313+0000 7f6a748d0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a70153e90 0x7f6a700cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.313+0000 7f6a67fff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a700ca9b0 0x7f6a701f2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:55712/0 (socket says 172.21.15.98:55712) 2023-11-28T17:48:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.313+0000 7f6a67fff640 1 -- 172.21.15.98:0/735722894 learned_addr learned my addr 172.21.15.98:0/735722894 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:55.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.313+0000 7f6a67fff640 1 -- 172.21.15.98:0/735722894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a70153e90 msgr2=0x7f6a700cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.313+0000 7f6a67fff640 1 --2- 172.21.15.98:0/735722894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a70153e90 0x7f6a700cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.313+0000 7f6a67fff640 1 -- 172.21.15.98:0/735722894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a700701a0 msgr2=0x7f6a700c4d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:48:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.313+0000 7f6a67fff640 1 --2- 172.21.15.98:0/735722894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a700701a0 0x7f6a700c4d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.313+0000 7f6a67fff640 1 -- 172.21.15.98:0/735722894 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a600094d0 con 0x7f6a700ca9b0 2023-11-28T17:48:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.314+0000 7f6a67fff640 1 --2- 172.21.15.98:0/735722894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a700ca9b0 0x7f6a701f2a20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800b550 tx=0x7f6a5800ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.314+0000 7f6a6d7fa640 1 -- 172.21.15.98:0/735722894 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58003dc0 con 0x7f6a700ca9b0 2023-11-28T17:48:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.314+0000 7f6a6d7fa640 1 -- 172.21.15.98:0/735722894 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a58005070 con 0x7f6a700ca9b0 2023-11-28T17:48:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.314+0000 7f6a750d1640 1 -- 172.21.15.98:0/735722894 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a700cb7c0 con 0x7f6a700ca9b0 2023-11-28T17:48:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.314+0000 7f6a6d7fa640 1 -- 172.21.15.98:0/735722894 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58005420 con 0x7f6a700ca9b0 2023-11-28T17:48:55.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.314+0000 7f6a750d1640 1 -- 172.21.15.98:0/735722894 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a700b0520 con 0x7f6a700ca9b0 2023-11-28T17:48:55.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.315+0000 7f6a750d1640 1 -- 172.21.15.98:0/735722894 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a70074db0 con 0x7f6a700ca9b0 2023-11-28T17:48:55.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.317+0000 7f6a6d7fa640 1 -- 172.21.15.98:0/735722894 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6a58005580 con 0x7f6a700ca9b0 2023-11-28T17:48:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.318+0000 7f6a6d7fa640 1 --2- 172.21.15.98:0/735722894 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6a5406b250 0x7f6a5406d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.318+0000 7f6a6d7fa640 1 -- 172.21.15.98:0/735722894 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6a5808a050 con 0x7f6a700ca9b0 2023-11-28T17:48:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.318+0000 7f6a6ffff640 1 --2- 172.21.15.98:0/735722894 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6a5406b250 0x7f6a5406d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:55.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.319+0000 7f6a6ffff640 1 --2- 172.21.15.98:0/735722894 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6a5406b250 0x7f6a5406d710 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6a60036000 tx=0x7f6a6003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:55.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.333+0000 7f6a6d7fa640 1 -- 172.21.15.98:0/735722894 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a58054f20 con 0x7f6a700ca9b0 2023-11-28T17:48:55.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.598+0000 7f6a750d1640 1 -- 172.21.15.98:0/735722894 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6a700c8370 con 0x7f6a700ca9b0 2023-11-28T17:48:55.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.599+0000 7f6a6d7fa640 1 -- 172.21.15.98:0/735722894 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f6a58058540 con 0x7f6a700ca9b0 2023-11-28T17:48:55.600 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:48:55.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.602+0000 7f6a750d1640 1 -- 172.21.15.98:0/735722894 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6a5406b250 msgr2=0x7f6a5406d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:55.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.602+0000 7f6a750d1640 1 --2- 172.21.15.98:0/735722894 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6a5406b250 0x7f6a5406d710 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6a60036000 tx=0x7f6a6003c040 comp rx=0 tx=0).stop 2023-11-28T17:48:55.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.602+0000 7f6a750d1640 1 -- 172.21.15.98:0/735722894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a700ca9b0 msgr2=0x7f6a701f2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:55.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.602+0000 7f6a750d1640 1 --2- 172.21.15.98:0/735722894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a700ca9b0 0x7f6a701f2a20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800b550 tx=0x7f6a5800ba10 comp rx=0 tx=0).stop 2023-11-28T17:48:55.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.602+0000 7f6a750d1640 1 -- 172.21.15.98:0/735722894 shutdown_connections 2023-11-28T17:48:55.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.602+0000 7f6a750d1640 1 --2- 172.21.15.98:0/735722894 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6a5406b250 0x7f6a5406d710 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:55.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.602+0000 7f6a750d1640 1 --2- 172.21.15.98:0/735722894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a70153e90 0x7f6a700cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:55.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.602+0000 7f6a750d1640 1 --2- 172.21.15.98:0/735722894 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6a700ca9b0 0x7f6a701f2a20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:55.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.602+0000 7f6a750d1640 1 --2- 172.21.15.98:0/735722894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a700701a0 0x7f6a700c4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:55.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.602+0000 7f6a750d1640 1 -- 172.21.15.98:0/735722894 >> 172.21.15.98:0/735722894 conn(0x7f6a700722b0 msgr2=0x7f6a70152130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:55.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.602+0000 7f6a750d1640 1 -- 172.21.15.98:0/735722894 shutdown_connections 2023-11-28T17:48:55.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:55.602+0000 7f6a750d1640 1 -- 172.21.15.98:0/735722894 wait complete. 2023-11-28T17:48:58.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:48:58.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.842+0000 7fbf44133640 1 Processor -- start 2023-11-28T17:48:58.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.842+0000 7fbf44133640 1 -- start start 2023-11-28T17:48:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.844+0000 7fbf44133640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbf3c0afa00 0x7fbf3c15dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.844+0000 7fbf44133640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbf3c159df0 0x7fbf3c164690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.844+0000 7fbf44133640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3c15da10 con 0x7fbf3c1572e0 2023-11-28T17:48:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.844+0000 7fbf44133640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3c0c84a0 con 0x7fbf3c159df0 2023-11-28T17:48:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.844+0000 7fbf44133640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3c06b680 con 0x7fbf3c0afa00 2023-11-28T17:48:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.844+0000 7fbf426d0640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7fbf3c159df0 msgr2=0x7fbf3c164690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:48:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.844+0000 7fbf426d0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbf3c159df0 0x7fbf3c164690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:58.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.845+0000 7fbf416ce640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbf3c0afa00 0x7fbf3c15dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:58.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.845+0000 7fbf416ce640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbf3c0afa00 0x7fbf3c15dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:60680/0 (socket says 172.21.15.98:60680) 2023-11-28T17:48:58.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.845+0000 7fbf416ce640 1 -- 172.21.15.98:0/1826816936 learned_addr learned my addr 172.21.15.98:0/1826816936 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.845+0000 7fbf416ce640 1 -- 172.21.15.98:0/1826816936 >> v2:172.21.15.98:3300/0 conn(0x7fbf3c159df0 msgr2=0x7fbf3c164690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:48:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.845+0000 7fbf416ce640 1 --2- 172.21.15.98:0/1826816936 >> v2:172.21.15.98:3300/0 conn(0x7fbf3c159df0 0x7fbf3c164690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.845+0000 7fbf416ce640 1 -- 172.21.15.98:0/1826816936 >> v1:172.21.15.113:6789/0 conn(0x7fbf3c1572e0 legacy=0x7fbf3c06e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.845+0000 7fbf416ce640 1 -- 172.21.15.98:0/1826816936 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c0aff50 con 0x7fbf3c0afa00 2023-11-28T17:48:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.847+0000 7fbf416ce640 1 --2- 172.21.15.98:0/1826816936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3c0afa00 0x7fbf3c15dea0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c0c9fb0 tx=0x7fbf3002ba60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7af985a6bb86dda9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:58.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.847+0000 7fbf40ecd640 1 -- 172.21.15.98:0/1826816936 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3002d820 con 0x7fbf3c0afa00 2023-11-28T17:48:58.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.847+0000 7fbf40ecd640 1 -- 172.21.15.98:0/1826816936 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3002d980 con 0x7fbf3c0afa00 2023-11-28T17:48:58.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.847+0000 7fbf40ecd640 1 -- 172.21.15.98:0/1826816936 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3002dd10 con 0x7fbf3c0afa00 2023-11-28T17:48:58.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.848+0000 7fbf44133640 1 -- 172.21.15.98:0/1826816936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3c0afa00 msgr2=0x7fbf3c15dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:58.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.848+0000 7fbf44133640 1 --2- 172.21.15.98:0/1826816936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3c0afa00 0x7fbf3c15dea0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c0c9fb0 tx=0x7fbf3002ba60 comp rx=0 tx=0).stop 2023-11-28T17:48:58.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.848+0000 7fbf44133640 1 -- 172.21.15.98:0/1826816936 shutdown_connections 2023-11-28T17:48:58.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.848+0000 7fbf44133640 1 --2- 172.21.15.98:0/1826816936 >> v2:172.21.15.98:3300/0 conn(0x7fbf3c159df0 0x7fbf3c164690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.848+0000 7fbf44133640 1 --2- 172.21.15.98:0/1826816936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3c0afa00 0x7fbf3c15dea0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.848+0000 7fbf44133640 1 -- 172.21.15.98:0/1826816936 >> 172.21.15.98:0/1826816936 conn(0x7fbf3c066230 msgr2=0x7fbf3c066610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.848+0000 7fbf44133640 1 -- 172.21.15.98:0/1826816936 shutdown_connections 2023-11-28T17:48:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.848+0000 7fbf44133640 1 -- 172.21.15.98:0/1826816936 wait complete. 2023-11-28T17:48:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.850+0000 7fbf44133640 1 Processor -- start 2023-11-28T17:48:58.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.850+0000 7fbf44133640 1 -- start start 2023-11-28T17:48:58.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.851+0000 7fbf44133640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf3c0afa00 0x7fbf3c152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:58.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.851+0000 7fbf44133640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf3c1572e0 0x7fbf3c155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:58.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.851+0000 7fbf44133640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3c159df0 0x7fbf3c1560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:58.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.851+0000 7fbf44133640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c1588d0 con 0x7fbf3c0afa00 2023-11-28T17:48:58.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.851+0000 7fbf44133640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c1f3840 con 0x7fbf3c159df0 2023-11-28T17:48:58.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.851+0000 7fbf44133640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c1509c0 con 0x7fbf3c1572e0 2023-11-28T17:48:58.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.851+0000 7fbf41ecf640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf3c0afa00 msgr2=0x7fbf3c152210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:48:58.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.851+0000 7fbf41ecf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf3c0afa00 0x7fbf3c152210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:48:58.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.851+0000 7fbf426d0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3c159df0 0x7fbf3c1560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:58.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.852+0000 7fbf416ce640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf3c1572e0 0x7fbf3c155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:58.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.852+0000 7fbf426d0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3c159df0 0x7fbf3c1560b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:60682/0 (socket says 172.21.15.98:60682) 2023-11-28T17:48:58.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.852+0000 7fbf426d0640 1 -- 172.21.15.98:0/2161254937 learned_addr learned my addr 172.21.15.98:0/2161254937 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:48:58.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.852+0000 7fbf426d0640 1 -- 172.21.15.98:0/2161254937 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf3c1572e0 msgr2=0x7fbf3c155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:58.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.852+0000 7fbf426d0640 1 --2- 172.21.15.98:0/2161254937 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf3c1572e0 0x7fbf3c155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:58.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.852+0000 7fbf426d0640 1 -- 172.21.15.98:0/2161254937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf3c0afa00 msgr2=0x7fbf3c152210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:48:58.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.852+0000 7fbf426d0640 1 --2- 172.21.15.98:0/2161254937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf3c0afa00 0x7fbf3c152210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:58.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.852+0000 7fbf426d0640 1 -- 172.21.15.98:0/2161254937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf300094d0 con 0x7fbf3c159df0 2023-11-28T17:48:58.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.853+0000 7fbf426d0640 1 --2- 172.21.15.98:0/2161254937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3c159df0 0x7fbf3c1560b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbf3400b3e0 tx=0x7fbf3400b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:58.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.853+0000 7fbf3a7fc640 1 -- 172.21.15.98:0/2161254937 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34003dc0 con 0x7fbf3c159df0 2023-11-28T17:48:58.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.853+0000 7fbf3a7fc640 1 -- 172.21.15.98:0/2161254937 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf340060d0 con 0x7fbf3c159df0 2023-11-28T17:48:58.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.853+0000 7fbf3a7fc640 1 -- 172.21.15.98:0/2161254937 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34006480 con 0x7fbf3c159df0 2023-11-28T17:48:58.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.853+0000 7fbf44133640 1 -- 172.21.15.98:0/2161254937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3c1f2f10 con 0x7fbf3c159df0 2023-11-28T17:48:58.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.853+0000 7fbf44133640 1 -- 172.21.15.98:0/2161254937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3c1f34d0 con 0x7fbf3c159df0 2023-11-28T17:48:58.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.854+0000 7fbf44133640 1 -- 172.21.15.98:0/2161254937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3c074f60 con 0x7fbf3c159df0 2023-11-28T17:48:58.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.857+0000 7fbf3a7fc640 1 -- 172.21.15.98:0/2161254937 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fbf340065e0 con 0x7fbf3c159df0 2023-11-28T17:48:58.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.857+0000 7fbf3a7fc640 1 --2- 172.21.15.98:0/2161254937 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbf2806b280 0x7fbf2806d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:48:58.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.858+0000 7fbf3a7fc640 1 -- 172.21.15.98:0/2161254937 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbf34088f20 con 0x7fbf3c159df0 2023-11-28T17:48:58.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.858+0000 7fbf41ecf640 1 --2- 172.21.15.98:0/2161254937 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbf2806b280 0x7fbf2806d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:48:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.858+0000 7fbf41ecf640 1 --2- 172.21.15.98:0/2161254937 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbf2806b280 0x7fbf2806d740 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0013b0 tx=0x7fbf2c007db0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:48:58.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:58.870+0000 7fbf3a7fc640 1 -- 172.21.15.98:0/2161254937 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf34053ea0 con 0x7fbf3c159df0 2023-11-28T17:48:59.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:59.129+0000 7fbf44133640 1 -- 172.21.15.98:0/2161254937 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbf3c1f3980 con 0x7fbf3c159df0 2023-11-28T17:48:59.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:59.130+0000 7fbf3a7fc640 1 -- 172.21.15.98:0/2161254937 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7fbf3c1f3980 con 0x7fbf3c159df0 2023-11-28T17:48:59.132 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:48:59.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:59.132+0000 7fbf44133640 1 -- 172.21.15.98:0/2161254937 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbf2806b280 msgr2=0x7fbf2806d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:59.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:59.132+0000 7fbf44133640 1 --2- 172.21.15.98:0/2161254937 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbf2806b280 0x7fbf2806d740 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0013b0 tx=0x7fbf2c007db0 comp rx=0 tx=0).stop 2023-11-28T17:48:59.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:59.132+0000 7fbf44133640 1 -- 172.21.15.98:0/2161254937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3c159df0 msgr2=0x7fbf3c1560b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:48:59.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:59.132+0000 7fbf44133640 1 --2- 172.21.15.98:0/2161254937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3c159df0 0x7fbf3c1560b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbf3400b3e0 tx=0x7fbf3400b8a0 comp rx=0 tx=0).stop 2023-11-28T17:48:59.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:59.132+0000 7fbf44133640 1 -- 172.21.15.98:0/2161254937 shutdown_connections 2023-11-28T17:48:59.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:59.132+0000 7fbf44133640 1 --2- 172.21.15.98:0/2161254937 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbf2806b280 0x7fbf2806d740 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:59.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:59.132+0000 7fbf44133640 1 --2- 172.21.15.98:0/2161254937 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf3c159df0 0x7fbf3c1560b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:59.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:59.132+0000 7fbf44133640 1 --2- 172.21.15.98:0/2161254937 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbf3c1572e0 0x7fbf3c155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:59.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:59.132+0000 7fbf44133640 1 --2- 172.21.15.98:0/2161254937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf3c0afa00 0x7fbf3c152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:48:59.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:59.132+0000 7fbf44133640 1 -- 172.21.15.98:0/2161254937 >> 172.21.15.98:0/2161254937 conn(0x7fbf3c066230 msgr2=0x7fbf3c0c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:48:59.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:59.132+0000 7fbf44133640 1 -- 172.21.15.98:0/2161254937 shutdown_connections 2023-11-28T17:48:59.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:48:59.133+0000 7fbf44133640 1 -- 172.21.15.98:0/2161254937 wait complete. 2023-11-28T17:49:01.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~250s 2023-11-28T17:49:02.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:49:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.373+0000 7f81dfbd5640 1 Processor -- start 2023-11-28T17:49:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.373+0000 7f81dfbd5640 1 -- start start 2023-11-28T17:49:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.375+0000 7f81dfbd5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f81d8157b60 0x7f81d815bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.375+0000 7f81dfbd5640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 0x7f81d80c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.375+0000 7f81dfbd5640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d815b150 con 0x7f81d8151d80 2023-11-28T17:49:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.375+0000 7f81dfbd5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d8157f90 con 0x7f81d8157b60 2023-11-28T17:49:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.375+0000 7f81dfbd5640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d815c150 con 0x7f81d80c4910 2023-11-28T17:49:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.375+0000 7f81dd170640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f81d8157b60 msgr2=0x7f81d815bc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:49:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.375+0000 7f81dd170640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f81d8157b60 0x7f81d815bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.376+0000 7f81de172640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 0x7f81d80c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.376+0000 7f81dd971640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f81d8151d80 0x7f81d80c5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.98:49454/0 (socket says 172.21.15.98:49454) 2023-11-28T17:49:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.376+0000 7f81dd971640 1 -- 172.21.15.98:0/2171880713 learned_addr learned my addr 172.21.15.98:0/2171880713 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:49:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.376+0000 7f81de172640 1 -- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 msgr2=0x7f81d80c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:49:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.376+0000 7f81de172640 1 -- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 msgr2=0x7f81d80c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:49:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.376+0000 7f81de172640 1 --2- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 0x7f81d80c4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.376+0000 7f81de172640 1 --2- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 0x7f81d80c4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:02.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.576+0000 7f81dd170640 1 -- 172.21.15.98:0/2171880713 >> v2:172.21.15.98:3300/0 conn(0x7f81d8157b60 msgr2=0x7f81d815bc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:49:02.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.576+0000 7f81dd170640 1 --2- 172.21.15.98:0/2171880713 >> v2:172.21.15.98:3300/0 conn(0x7f81d8157b60 0x7f81d815bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:49:02.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.577+0000 7f81de172640 1 --2- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 0x7f81d80c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:02.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.578+0000 7f81de172640 1 -- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 msgr2=0x7f81d80c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:49:02.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.578+0000 7f81de172640 1 -- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 msgr2=0x7f81d80c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:49:02.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.578+0000 7f81de172640 1 --2- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 0x7f81d80c4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:02.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.578+0000 7f81de172640 1 --2- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 0x7f81d80c4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:49:02.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.977+0000 7f81dd170640 1 -- 172.21.15.98:0/2171880713 >> v2:172.21.15.98:3300/0 conn(0x7f81d8157b60 msgr2=0x7f81d815bc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:49:02.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.977+0000 7f81dd170640 1 --2- 172.21.15.98:0/2171880713 >> v2:172.21.15.98:3300/0 conn(0x7f81d8157b60 0x7f81d815bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:49:02.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.980+0000 7f81de172640 1 --2- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 0x7f81d80c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:02.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.980+0000 7f81de172640 1 -- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 msgr2=0x7f81d80c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:49:02.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.980+0000 7f81de172640 1 -- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 msgr2=0x7f81d80c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:49:02.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.980+0000 7f81de172640 1 --2- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 0x7f81d80c4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:02.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:02.980+0000 7f81de172640 1 --2- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 0x7f81d80c4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:49:03.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:03.778+0000 7f81dd170640 1 -- 172.21.15.98:0/2171880713 >> v2:172.21.15.98:3300/0 conn(0x7f81d8157b60 msgr2=0x7f81d815bc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:49:03.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:03.778+0000 7f81dd170640 1 --2- 172.21.15.98:0/2171880713 >> v2:172.21.15.98:3300/0 conn(0x7f81d8157b60 0x7f81d815bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:49:03.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:03.782+0000 7f81de172640 1 --2- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 0x7f81d80c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:03.782+0000 7f81de172640 1 -- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 msgr2=0x7f81d80c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:49:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:03.782+0000 7f81de172640 1 -- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 msgr2=0x7f81d80c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:49:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:03.782+0000 7f81de172640 1 --2- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 0x7f81d80c4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:03.782+0000 7f81de172640 1 --2- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 0x7f81d80c4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:49:05.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.376+0000 7f81cf7fe640 1 -- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 msgr2=0x7f81d80c4d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:49:05.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.376+0000 7f81cf7fe640 1 --2- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d80c4910 0x7f81d80c4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:05.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.376+0000 7f81cf7fe640 1 -- 172.21.15.98:0/2171880713 >> v2:172.21.15.98:3300/0 conn(0x7f81d8157b60 msgr2=0x7f81d815bc10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:49:05.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.376+0000 7f81cf7fe640 1 --2- 172.21.15.98:0/2171880713 >> v2:172.21.15.98:3300/0 conn(0x7f81d8157b60 0x7f81d815bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.376+0000 7f81cf7fe640 1 -- 172.21.15.98:0/2171880713 >> v1:172.21.15.113:6789/0 conn(0x7f81d8151d80 legacy=0x7f81d80c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.376+0000 7f81cf7fe640 1 --2- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 0x7f81bc0098a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.376+0000 7f81cf7fe640 1 -- 172.21.15.98:0/2171880713 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d806e1f0 con 0x7f81bc0036c0 2023-11-28T17:49:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.376+0000 7f81cf7fe640 1 -- 172.21.15.98:0/2171880713 --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81bc00af70 con 0x7f81bc007400 2023-11-28T17:49:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.376+0000 7f81dd971640 1 -- 172.21.15.98:0/2171880713 >> v1:172.21.15.98:6789/0 conn(0x7f81bc0036c0 legacy=0x7f81bc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:49:05.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.376+0000 7f81dd170640 1 --2- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 0x7f81bc0098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:05.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.377+0000 7f81dd170640 1 -- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 msgr2=0x7f81bc0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:49:05.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.377+0000 7f81dd170640 1 -- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 msgr2=0x7f81bc0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:49:05.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.377+0000 7f81dd170640 1 --2- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 0x7f81bc0098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:05.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.377+0000 7f81dd170640 1 --2- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 0x7f81bc0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:05.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.577+0000 7f81dd971640 1 -- 172.21.15.98:0/2171880713 >> v1:172.21.15.98:6789/0 conn(0x7f81bc0036c0 legacy=0x7f81bc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:49:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.578+0000 7f81dd170640 1 --2- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 0x7f81bc0098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.579+0000 7f81dd170640 1 -- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 msgr2=0x7f81bc0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:49:05.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.579+0000 7f81dd170640 1 -- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 msgr2=0x7f81bc0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:49:05.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.579+0000 7f81dd170640 1 --2- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 0x7f81bc0098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:05.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.579+0000 7f81dd170640 1 --2- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 0x7f81bc0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:49:05.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.978+0000 7f81dd971640 1 -- 172.21.15.98:0/2171880713 >> v1:172.21.15.98:6789/0 conn(0x7f81bc0036c0 legacy=0x7f81bc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:49:05.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.980+0000 7f81dd170640 1 --2- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 0x7f81bc0098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:05.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.981+0000 7f81dd170640 1 -- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 msgr2=0x7f81bc0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:49:05.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.981+0000 7f81dd170640 1 -- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 msgr2=0x7f81bc0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:49:05.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.981+0000 7f81dd170640 1 --2- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 0x7f81bc0098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:05.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:05.981+0000 7f81dd170640 1 --2- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 0x7f81bc0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:49:06.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.779+0000 7f81dd971640 1 -- 172.21.15.98:0/2171880713 >> v1:172.21.15.98:6789/0 conn(0x7f81bc0036c0 legacy=0x7f81bc005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:49:06.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.783+0000 7f81dd170640 1 --2- 172.21.15.98:0/2171880713 >> v2:172.21.15.113:3300/0 conn(0x7f81bc007400 0x7f81bc0098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:06.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.784+0000 7f81dd170640 1 -- 172.21.15.98:0/2171880713 >> v1:172.21.15.98:6789/0 conn(0x7f81bc0036c0 legacy=0x7f81bc005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:49:06.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.784+0000 7f81dd170640 1 -- 172.21.15.98:0/2171880713 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81bc00b0e0 con 0x7f81bc007400 2023-11-28T17:49:06.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.787+0000 7f81dd170640 1 --2- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81bc007400 0x7f81bc0098a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f81c8003980 tx=0x7f81c802c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86592475f36f4176 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:06.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.788+0000 7f81dc96f640 1 -- 172.21.15.98:0/2171880713 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c8033890 con 0x7f81bc007400 2023-11-28T17:49:06.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.788+0000 7f81dc96f640 1 -- 172.21.15.98:0/2171880713 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c8006e90 con 0x7f81bc007400 2023-11-28T17:49:06.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.788+0000 7f81dc96f640 1 -- 172.21.15.98:0/2171880713 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c8033c30 con 0x7f81bc007400 2023-11-28T17:49:06.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.789+0000 7f81dfbd5640 1 -- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81bc007400 msgr2=0x7f81bc0098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:06.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.789+0000 7f81dfbd5640 1 --2- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81bc007400 0x7f81bc0098a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f81c8003980 tx=0x7f81c802c9c0 comp rx=0 tx=0).stop 2023-11-28T17:49:06.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.789+0000 7f81de172640 1 -- 172.21.15.98:0/2171880713 reap_dead start 2023-11-28T17:49:06.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.789+0000 7f81dfbd5640 1 -- 172.21.15.98:0/2171880713 shutdown_connections 2023-11-28T17:49:06.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.789+0000 7f81dfbd5640 1 --2- 172.21.15.98:0/2171880713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81bc007400 0x7f81bc0098a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:06.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.789+0000 7f81dfbd5640 1 -- 172.21.15.98:0/2171880713 >> 172.21.15.98:0/2171880713 conn(0x7f81d806a4d0 msgr2=0x7f81d806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:06.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.790+0000 7f81dfbd5640 1 -- 172.21.15.98:0/2171880713 shutdown_connections 2023-11-28T17:49:06.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.790+0000 7f81dfbd5640 1 -- 172.21.15.98:0/2171880713 wait complete. 2023-11-28T17:49:06.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.793+0000 7f81dfbd5640 1 Processor -- start 2023-11-28T17:49:06.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.793+0000 7f81dfbd5640 1 -- start start 2023-11-28T17:49:06.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.795+0000 7f81dfbd5640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81bc007400 0x7f81d815a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:06.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.795+0000 7f81dfbd5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81d8157330 0x7f81d81557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:06.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.795+0000 7f81dfbd5640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d8146c20 0x7f81d81470a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:06.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.795+0000 7f81dfbd5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d813c450 con 0x7f81d8157330 2023-11-28T17:49:06.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.795+0000 7f81dfbd5640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d81475e0 con 0x7f81bc007400 2023-11-28T17:49:06.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.795+0000 7f81dfbd5640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d81447e0 con 0x7f81d8146c20 2023-11-28T17:49:06.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.795+0000 7f81dd170640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81d8157330 msgr2=0x7f81d81557a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:49:06.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.795+0000 7f81dd170640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81d8157330 0x7f81d81557a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:06.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.795+0000 7f81dd971640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81bc007400 0x7f81d815a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:06.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.796+0000 7f81dd971640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81bc007400 0x7f81d815a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:40934/0 (socket says 172.21.15.98:40934) 2023-11-28T17:49:06.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.796+0000 7f81dd971640 1 -- 172.21.15.98:0/2919483127 learned_addr learned my addr 172.21.15.98:0/2919483127 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:06.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.796+0000 7f81de172640 1 --2- 172.21.15.98:0/2919483127 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d8146c20 0x7f81d81470a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:06.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.796+0000 7f81dd971640 1 -- 172.21.15.98:0/2919483127 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d8146c20 msgr2=0x7f81d81470a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:06.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.796+0000 7f81dd971640 1 --2- 172.21.15.98:0/2919483127 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d8146c20 0x7f81d81470a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:06.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.796+0000 7f81dd971640 1 -- 172.21.15.98:0/2919483127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81d8157330 msgr2=0x7f81d81557a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:49:06.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.796+0000 7f81dd971640 1 --2- 172.21.15.98:0/2919483127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81d8157330 0x7f81d81557a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:06.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.796+0000 7f81dd971640 1 -- 172.21.15.98:0/2919483127 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c8009a20 con 0x7f81bc007400 2023-11-28T17:49:06.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.797+0000 7f81dd971640 1 --2- 172.21.15.98:0/2919483127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81bc007400 0x7f81d815a1f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f81d400c520 tx=0x7f81d400c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:06.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.797+0000 7f81ce7fc640 1 -- 172.21.15.98:0/2919483127 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d4005ce0 con 0x7f81bc007400 2023-11-28T17:49:06.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.797+0000 7f81ce7fc640 1 -- 172.21.15.98:0/2919483127 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d4005e40 con 0x7f81bc007400 2023-11-28T17:49:06.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.797+0000 7f81ce7fc640 1 -- 172.21.15.98:0/2919483127 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d4007270 con 0x7f81bc007400 2023-11-28T17:49:06.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.797+0000 7f81dfbd5640 1 -- 172.21.15.98:0/2919483127 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d8145320 con 0x7f81bc007400 2023-11-28T17:49:06.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.798+0000 7f81dfbd5640 1 -- 172.21.15.98:0/2919483127 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d8144d60 con 0x7f81bc007400 2023-11-28T17:49:06.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.799+0000 7f81dfbd5640 1 -- 172.21.15.98:0/2919483127 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d8067bf0 con 0x7f81bc007400 2023-11-28T17:49:06.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.802+0000 7f81ce7fc640 1 -- 172.21.15.98:0/2919483127 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f81d40073d0 con 0x7f81bc007400 2023-11-28T17:49:06.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.803+0000 7f81ce7fc640 1 --2- 172.21.15.98:0/2919483127 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f81bc06c1c0 0x7f81bc06e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:06.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.803+0000 7f81ce7fc640 1 -- 172.21.15.98:0/2919483127 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f81d40891f0 con 0x7f81bc007400 2023-11-28T17:49:06.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.804+0000 7f81dd170640 1 --2- 172.21.15.98:0/2919483127 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f81bc06c1c0 0x7f81bc06e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:06.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.805+0000 7f81dd170640 1 --2- 172.21.15.98:0/2919483127 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f81bc06c1c0 0x7f81bc06e680 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f81c8038000 tx=0x7f81c800f040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:06.818+0000 7f81ce7fc640 1 -- 172.21.15.98:0/2919483127 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f81d4053ff0 con 0x7f81bc007400 2023-11-28T17:49:07.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.093+0000 7f81dfbd5640 1 -- 172.21.15.98:0/2919483127 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f81d80c2470 con 0x7f81bc007400 2023-11-28T17:49:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.095+0000 7f81ce7fc640 1 -- 172.21.15.98:0/2919483127 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f81d4017620 con 0x7f81bc007400 2023-11-28T17:49:07.097 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":78,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:49:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.098+0000 7f81dfbd5640 1 -- 172.21.15.98:0/2919483127 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f81bc06c1c0 msgr2=0x7f81bc06e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.098+0000 7f81dfbd5640 1 --2- 172.21.15.98:0/2919483127 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f81bc06c1c0 0x7f81bc06e680 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f81c8038000 tx=0x7f81c800f040 comp rx=0 tx=0).stop 2023-11-28T17:49:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.098+0000 7f81dfbd5640 1 -- 172.21.15.98:0/2919483127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81bc007400 msgr2=0x7f81d815a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.098+0000 7f81dfbd5640 1 --2- 172.21.15.98:0/2919483127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81bc007400 0x7f81d815a1f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f81d400c520 tx=0x7f81d400c9e0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.098+0000 7f81dfbd5640 1 -- 172.21.15.98:0/2919483127 shutdown_connections 2023-11-28T17:49:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.098+0000 7f81dfbd5640 1 --2- 172.21.15.98:0/2919483127 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f81d8146c20 0x7f81d81470a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.098+0000 7f81dfbd5640 1 --2- 172.21.15.98:0/2919483127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81d8157330 0x7f81d81557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.098+0000 7f81dfbd5640 1 --2- 172.21.15.98:0/2919483127 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f81bc06c1c0 0x7f81bc06e680 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.098+0000 7f81dfbd5640 1 --2- 172.21.15.98:0/2919483127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f81bc007400 0x7f81d815a1f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.098+0000 7f81dfbd5640 1 -- 172.21.15.98:0/2919483127 >> 172.21.15.98:0/2919483127 conn(0x7f81d806a4d0 msgr2=0x7f81d81509e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:07.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.098+0000 7f81dfbd5640 1 -- 172.21.15.98:0/2919483127 shutdown_connections 2023-11-28T17:49:07.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.098+0000 7f81dfbd5640 1 -- 172.21.15.98:0/2919483127 wait complete. 2023-11-28T17:49:07.119 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2023-11-28T17:49:07.119 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2023-11-28T17:49:07.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:49:07.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.331+0000 7f53f6dcf640 1 Processor -- start 2023-11-28T17:49:07.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.331+0000 7f53f6dcf640 1 -- start start 2023-11-28T17:49:07.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.333+0000 7f53f6dcf640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f53f00c24e0 0x7f53f00c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.333+0000 7f53f6dcf640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f53f006b160 0x7f53f00c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.333+0000 7f53f6dcf640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53f0157ab0 0x7f53f015b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.333+0000 7f53f6dcf640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f00c2e00 con 0x7f53f006b160 2023-11-28T17:49:07.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.333+0000 7f53f6dcf640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f00c6300 con 0x7f53f00c24e0 2023-11-28T17:49:07.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.333+0000 7f53f6dcf640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f00c7430 con 0x7f53f0157ab0 2023-11-28T17:49:07.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.333+0000 7f53f55cc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f53f006b160 msgr2=0x7f53f00c28c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:49:07.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.333+0000 7f53f55cc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f53f006b160 0x7f53f00c28c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:07.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.334+0000 7f53f5dcd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f53f00c24e0 0x7f53f00c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:07.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.334+0000 7f53f65ce640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53f0157ab0 0x7f53f015b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:07.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.334+0000 7f53f5dcd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f53f00c24e0 0x7f53f00c6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:40938/0 (socket says 172.21.15.98:40938) 2023-11-28T17:49:07.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.334+0000 7f53f5dcd640 1 -- 172.21.15.98:0/3377598936 learned_addr learned my addr 172.21.15.98:0/3377598936 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:07.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.334+0000 7f53f5dcd640 1 -- 172.21.15.98:0/3377598936 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53f0157ab0 msgr2=0x7f53f015b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:07.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.334+0000 7f53f5dcd640 1 --2- 172.21.15.98:0/3377598936 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53f0157ab0 0x7f53f015b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.334+0000 7f53f5dcd640 1 -- 172.21.15.98:0/3377598936 >> v2:172.21.15.98:3300/0 conn(0x7f53f006b160 msgr2=0x7f53f00c28c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:49:07.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.334+0000 7f53f5dcd640 1 --2- 172.21.15.98:0/3377598936 >> v2:172.21.15.98:3300/0 conn(0x7f53f006b160 0x7f53f00c28c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.334+0000 7f53f5dcd640 1 -- 172.21.15.98:0/3377598936 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f00c7710 con 0x7f53f00c24e0 2023-11-28T17:49:07.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.336+0000 7f53f5dcd640 1 --2- 172.21.15.98:0/3377598936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f00c24e0 0x7f53f00c6dc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f53f0068150 tx=0x7f53ec02b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccd708debfb5ce6b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:07.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.337+0000 7f53f4dcb640 1 -- 172.21.15.98:0/3377598936 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec031800 con 0x7f53f00c24e0 2023-11-28T17:49:07.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.337+0000 7f53f4dcb640 1 -- 172.21.15.98:0/3377598936 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ec031960 con 0x7f53f00c24e0 2023-11-28T17:49:07.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.337+0000 7f53f4dcb640 1 -- 172.21.15.98:0/3377598936 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec031cf0 con 0x7f53f00c24e0 2023-11-28T17:49:07.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.337+0000 7f53f6dcf640 1 -- 172.21.15.98:0/3377598936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f00c24e0 msgr2=0x7f53f00c6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:07.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.337+0000 7f53f6dcf640 1 --2- 172.21.15.98:0/3377598936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f00c24e0 0x7f53f00c6dc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f53f0068150 tx=0x7f53ec02b8a0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.337+0000 7f53f6dcf640 1 -- 172.21.15.98:0/3377598936 shutdown_connections 2023-11-28T17:49:07.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.337+0000 7f53f6dcf640 1 --2- 172.21.15.98:0/3377598936 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53f0157ab0 0x7f53f015b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.337+0000 7f53f6dcf640 1 --2- 172.21.15.98:0/3377598936 >> v2:172.21.15.98:3300/0 conn(0x7f53f006b160 0x7f53f00c28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.337+0000 7f53f6dcf640 1 --2- 172.21.15.98:0/3377598936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f00c24e0 0x7f53f00c6dc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.337+0000 7f53f6dcf640 1 -- 172.21.15.98:0/3377598936 >> 172.21.15.98:0/3377598936 conn(0x7f53f00cafe0 msgr2=0x7f53f00cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:07.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.338+0000 7f53f6dcf640 1 -- 172.21.15.98:0/3377598936 shutdown_connections 2023-11-28T17:49:07.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.338+0000 7f53f6dcf640 1 -- 172.21.15.98:0/3377598936 wait complete. 2023-11-28T17:49:07.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.340+0000 7f53f6dcf640 1 Processor -- start 2023-11-28T17:49:07.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.340+0000 7f53f6dcf640 1 -- start start 2023-11-28T17:49:07.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.341+0000 7f53f6dcf640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f006b160 0x7f53f0071080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.341+0000 7f53f6dcf640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53f00c24e0 0x7f53f00715c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.341+0000 7f53f6dcf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53f0157ab0 0x7f53f014a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.341+0000 7f53f6dcf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f014d1d0 con 0x7f53f0157ab0 2023-11-28T17:49:07.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.341+0000 7f53f6dcf640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f0070bf0 con 0x7f53f006b160 2023-11-28T17:49:07.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.341+0000 7f53f6dcf640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f00705e0 con 0x7f53f00c24e0 2023-11-28T17:49:07.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.342+0000 7f53f65ce640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53f0157ab0 msgr2=0x7f53f014a870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:49:07.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.342+0000 7f53f65ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53f0157ab0 0x7f53f014a870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:07.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.342+0000 7f53f55cc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53f00c24e0 0x7f53f00715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:07.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.342+0000 7f53f5dcd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f006b160 0x7f53f0071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:07.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.342+0000 7f53f55cc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53f00c24e0 0x7f53f00715c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:37214/0 (socket says 172.21.15.98:37214) 2023-11-28T17:49:07.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.342+0000 7f53f55cc640 1 -- 172.21.15.98:0/770007196 learned_addr learned my addr 172.21.15.98:0/770007196 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:07.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.342+0000 7f53f5dcd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f006b160 0x7f53f0071080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:40948/0 (socket says 172.21.15.98:40948) 2023-11-28T17:49:07.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.343+0000 7f53f55cc640 1 -- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f006b160 msgr2=0x7f53f0071080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:07.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.343+0000 7f53f55cc640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f006b160 0x7f53f0071080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.343+0000 7f53f55cc640 1 -- 172.21.15.98:0/770007196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53f0157ab0 msgr2=0x7f53f014a870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:49:07.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.343+0000 7f53f55cc640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53f0157ab0 0x7f53f014a870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.343+0000 7f53f55cc640 1 -- 172.21.15.98:0/770007196 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec0094d0 con 0x7f53f00c24e0 2023-11-28T17:49:07.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.343+0000 7f53f55cc640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53f00c24e0 0x7f53f00715c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f53e000b520 tx=0x7f53e000b9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:07.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.344+0000 7f53e67fc640 1 -- 172.21.15.98:0/770007196 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e0003b90 con 0x7f53f00c24e0 2023-11-28T17:49:07.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.344+0000 7f53e67fc640 1 -- 172.21.15.98:0/770007196 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e0003cf0 con 0x7f53f00c24e0 2023-11-28T17:49:07.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.344+0000 7f53e67fc640 1 -- 172.21.15.98:0/770007196 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e0005290 con 0x7f53f00c24e0 2023-11-28T17:49:07.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.344+0000 7f53f6dcf640 1 -- 172.21.15.98:0/770007196 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f014d420 con 0x7f53f00c24e0 2023-11-28T17:49:07.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.344+0000 7f53f6dcf640 1 -- 172.21.15.98:0/770007196 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f0158c10 con 0x7f53f00c24e0 2023-11-28T17:49:07.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.345+0000 7f53f6dcf640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f014b520 0x7f53f014adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.345+0000 7f53f6dcf640 1 -- 172.21.15.98:0/770007196 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f0074310 con 0x7f53f014b520 2023-11-28T17:49:07.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.346+0000 7f53f5dcd640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f014b520 0x7f53f014adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:07.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.346+0000 7f53f5dcd640 1 -- 172.21.15.98:0/770007196 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53f00c68a0 con 0x7f53f014b520 2023-11-28T17:49:07.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.347+0000 7f53f5dcd640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f014b520 0x7f53f014adb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f53f006b0f0 tx=0x7f53ec004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:07.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.347+0000 7f53e67fc640 1 -- 172.21.15.98:0/770007196 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f53e00053f0 con 0x7f53f00c24e0 2023-11-28T17:49:07.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.348+0000 7f53e67fc640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f53d406b250 0x7f53d406d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.348+0000 7f53e67fc640 1 -- 172.21.15.98:0/770007196 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f53e0089660 con 0x7f53f00c24e0 2023-11-28T17:49:07.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.348+0000 7f53e67fc640 1 -- 172.21.15.98:0/770007196 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec049070 con 0x7f53f014b520 2023-11-28T17:49:07.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.348+0000 7f53f65ce640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f53d406b250 0x7f53d406d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:07.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.348+0000 7f53e67fc640 1 -- 172.21.15.98:0/770007196 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f53ec03a9f0 con 0x7f53f014b520 2023-11-28T17:49:07.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.348+0000 7f53e67fc640 1 -- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f014b520 msgr2=0x7f53f014adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:07.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.348+0000 7f53e67fc640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f014b520 0x7f53f014adb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f53f006b0f0 tx=0x7f53ec004730 comp rx=0 tx=0).stop 2023-11-28T17:49:07.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.348+0000 7f53e67fc640 1 -- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f014b520 msgr2=0x7f53f014adb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:49:07.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.348+0000 7f53e67fc640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f014b520 0x7f53f014adb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.348+0000 7f53f65ce640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f53d406b250 0x7f53d406d710 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f53f00c82c0 tx=0x7f53e800b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:07.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.355+0000 7f53f6dcf640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f01f2fd0 0x7f53f01f33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.355+0000 7f53f6dcf640 1 -- 172.21.15.98:0/770007196 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f00cae10 con 0x7f53f01f2fd0 2023-11-28T17:49:07.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.356+0000 7f53f5dcd640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f01f2fd0 0x7f53f01f33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:07.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.356+0000 7f53f5dcd640 1 -- 172.21.15.98:0/770007196 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f53f0070d60 con 0x7f53f01f2fd0 2023-11-28T17:49:07.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.357+0000 7f53f5dcd640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f01f2fd0 0x7f53f01f33e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f53ec0097c0 tx=0x7f53ec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:07.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.357+0000 7f53e67fc640 1 -- 172.21.15.98:0/770007196 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec003d20 con 0x7f53f01f2fd0 2023-11-28T17:49:07.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.358+0000 7f53e67fc640 1 -- 172.21.15.98:0/770007196 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4966 (secure 0 0 0) 0x7f53ec003e80 con 0x7f53f01f2fd0 2023-11-28T17:49:07.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.358+0000 7f53e67fc640 1 -- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f01f2fd0 msgr2=0x7f53f01f33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:07.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.358+0000 7f53e67fc640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f01f2fd0 0x7f53f01f33e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f53ec0097c0 tx=0x7f53ec0046c0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.358+0000 7f53e67fc640 1 -- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f01f2fd0 msgr2=0x7f53f01f33e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:49:07.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.358+0000 7f53e67fc640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53f01f2fd0 0x7f53f01f33e0 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f53ec0097c0 tx=0x7f53ec0046c0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.361 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:49:07.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:49:07.361 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:49:07.361 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:49:07.361 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 78, 2023-11-28T17:49:07.361 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:49:07.361 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:49:07.362 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:49:07.362 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:07.362 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2023-11-28T17:49:07.362 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:49:07.362 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:49:07.362 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:49:07.362 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:49:07.362 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:49:07.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:49:07.362 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:49:07.362 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:49:07.363 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:49:07.363 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:49:07.363 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:49:07.363 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:49:07.363 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:49:07.363 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:07.363 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:49:07.363 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:49:07.363 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:49:07.363 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:49:07.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:49:07.363 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:49:07.363 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:49:07.364 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:49:07.364 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:49:07.364 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:49:07.364 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:49:07.364 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:49:07.364 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:07.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.364 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:49:07.364 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:49:07.364 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:49:07.364 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:49:07.364 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:49:07.364 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:49:07.365 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:49:07.365 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:49:07.365 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:49:07.365 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:49:07.365 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:49:07.365 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:49:07.365 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:49:07.365 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:49:07.365 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:49:07.365 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:49:07.365 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:49:07.366 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:49:07.366 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:49:07.366 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:49:07.366 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:49:07.366 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:49:07.366 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:49:07.366 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:49:07.366 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:49:07.366 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:49:07.366 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:49:07.366 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:49:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:49:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:49:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:49:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:49:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:49:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:49:07.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:49:07.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:07.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:49:07.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:49:07.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:07.368 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.368 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:07.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:49:07.368 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:49:07.369 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:49:07.369 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:49:07.369 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:49:07.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.369 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:49:07.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:49:07.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:49:07.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:49:07.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:49:07.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:49:07.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:07.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:49:07.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:49:07.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:07.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.370 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:07.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:49:07.371 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:49:07.371 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:49:07.371 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:49:07.371 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:49:07.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.371 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:49:07.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:49:07.371 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:49:07.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:49:07.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:49:07.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:49:07.372 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:07.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:49:07.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:49:07.372 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:07.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:07.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:49:07.373 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:49:07.373 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:49:07.373 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:49:07.373 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:49:07.373 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.373 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:07.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.373 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:49:07.373 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:49:07.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:49:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:49:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:49:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:49:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:49:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:49:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2023-11-28T17:49:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:49:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:49:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:07.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.377 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:49:07.377 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:49:07.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.359+0000 7f53f6dcf640 1 -- 172.21.15.98:0/770007196 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f53d406b250 msgr2=0x7f53d406d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:07.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.359+0000 7f53f6dcf640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f53d406b250 0x7f53d406d710 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f53f00c82c0 tx=0x7f53e800b040 comp rx=0 tx=0).stop 2023-11-28T17:49:07.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.359+0000 7f53f6dcf640 1 -- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53f00c24e0 msgr2=0x7f53f00715c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:07.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.359+0000 7f53f6dcf640 1 --2- 172.21.15.98:0/770007196 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53f00c24e0 0x7f53f00715c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f53e000b520 tx=0x7f53e000b9e0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.359+0000 7f53f65ce640 1 -- 172.21.15.98:0/770007196 reap_dead start 2023-11-28T17:49:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.359+0000 7f53f65ce640 1 -- 172.21.15.98:0/770007196 reap_dead start 2023-11-28T17:49:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.359+0000 7f53f6dcf640 1 -- 172.21.15.98:0/770007196 shutdown_connections 2023-11-28T17:49:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.359+0000 7f53f6dcf640 1 -- 172.21.15.98:0/770007196 >> 172.21.15.98:0/770007196 conn(0x7f53f00cafe0 msgr2=0x7f53f00c8470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.359+0000 7f53f6dcf640 1 -- 172.21.15.98:0/770007196 shutdown_connections 2023-11-28T17:49:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.359+0000 7f53f6dcf640 1 -- 172.21.15.98:0/770007196 wait complete. 2023-11-28T17:49:07.379 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:49:07.379 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2023-11-28T17:49:07.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:49:07.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.585+0000 7f498c8d1640 1 Processor -- start 2023-11-28T17:49:07.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.586+0000 7f498c8d1640 1 -- start start 2023-11-28T17:49:07.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.587+0000 7f498c8d1640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4988153c80 0x7f498806b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.587+0000 7f498c8d1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498815d230 con 0x7f49880c7e30 2023-11-28T17:49:07.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.587+0000 7f498c8d1640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498815d4f0 con 0x7f49880c69b0 2023-11-28T17:49:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.587+0000 7f498c8d1640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498815e070 con 0x7f4988153c80 2023-11-28T17:49:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.587+0000 7f4987fff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f49880c7e30 legacy=0x7f4988164440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:49:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.588+0000 7f49877fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4988153c80 0x7f498806b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.588+0000 7f497ffff640 1 --1- >> v1:172.21.15.113:6789/0 conn(0x7f49880c69b0 0x7f49880c7720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.113:6789/0 says I am v1:172.21.15.98:49464/0 (socket says 172.21.15.98:49464) 2023-11-28T17:49:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.588+0000 7f497ffff640 1 -- 172.21.15.98:0/3007433937 learned_addr learned my addr 172.21.15.98:0/3007433937 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:49:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.588+0000 7f49877fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4988153c80 0x7f498806b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:37230/0 (socket says 172.21.15.98:37230) 2023-11-28T17:49:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.588+0000 7f49877fe640 1 -- 172.21.15.98:0/3007433937 >> v1:172.21.15.113:6789/0 conn(0x7f49880c69b0 legacy=0x7f49880c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.588+0000 7f49877fe640 1 -- 172.21.15.98:0/3007433937 >> v1:172.21.15.98:6789/0 conn(0x7f49880c7e30 legacy=0x7f4988164440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:49:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.588+0000 7f49877fe640 1 -- 172.21.15.98:0/3007433937 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498815d7d0 con 0x7f4988153c80 2023-11-28T17:49:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.590+0000 7f49877fe640 1 --2- 172.21.15.98:0/3007433937 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4988153c80 0x7f498806b940 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f49780097d0 tx=0x7f497802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a8a29a49f40380b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.590+0000 7f4986ffd640 1 -- 172.21.15.98:0/3007433937 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4978031890 con 0x7f4988153c80 2023-11-28T17:49:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.590+0000 7f4986ffd640 1 -- 172.21.15.98:0/3007433937 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49780319f0 con 0x7f4988153c80 2023-11-28T17:49:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.590+0000 7f4986ffd640 1 -- 172.21.15.98:0/3007433937 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4978031d80 con 0x7f4988153c80 2023-11-28T17:49:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.591+0000 7f498c8d1640 1 -- 172.21.15.98:0/3007433937 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4988153c80 msgr2=0x7f498806b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.591+0000 7f498c8d1640 1 --2- 172.21.15.98:0/3007433937 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4988153c80 0x7f498806b940 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f49780097d0 tx=0x7f497802b930 comp rx=0 tx=0).stop 2023-11-28T17:49:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.591+0000 7f498c8d1640 1 -- 172.21.15.98:0/3007433937 shutdown_connections 2023-11-28T17:49:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.591+0000 7f498c8d1640 1 --2- 172.21.15.98:0/3007433937 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4988153c80 0x7f498806b940 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.591+0000 7f498c8d1640 1 -- 172.21.15.98:0/3007433937 >> 172.21.15.98:0/3007433937 conn(0x7f49880cf300 msgr2=0x7f49880cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.591+0000 7f498c8d1640 1 -- 172.21.15.98:0/3007433937 shutdown_connections 2023-11-28T17:49:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.591+0000 7f498c8d1640 1 -- 172.21.15.98:0/3007433937 wait complete. 2023-11-28T17:49:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.593+0000 7f498c8d1640 1 Processor -- start 2023-11-28T17:49:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.593+0000 7f498c8d1640 1 -- start start 2023-11-28T17:49:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.594+0000 7f498c8d1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49880c69b0 0x7f49881e5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.594+0000 7f498c8d1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49880c7e30 0x7f49881e3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.594+0000 7f498c8d1640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4988153c80 0x7f49881e3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.594+0000 7f498c8d1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49881e2ec0 con 0x7f49880c69b0 2023-11-28T17:49:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.594+0000 7f498c8d1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498814d960 con 0x7f49880c7e30 2023-11-28T17:49:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.594+0000 7f498c8d1640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498814dbd0 con 0x7f4988153c80 2023-11-28T17:49:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.594+0000 7f49877fe640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49880c69b0 msgr2=0x7f49881e5460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:49:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.594+0000 7f49877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49880c69b0 0x7f49881e5460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.594+0000 7f4987fff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4988153c80 0x7f49881e3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.594+0000 7f497ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49880c7e30 0x7f49881e3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.594+0000 7f4987fff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4988153c80 0x7f49881e3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3301/0 says I am v2:172.21.15.98:37236/0 (socket says 172.21.15.98:37236) 2023-11-28T17:49:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.595+0000 7f4987fff640 1 -- 172.21.15.98:0/888049624 learned_addr learned my addr 172.21.15.98:0/888049624 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.595+0000 7f4987fff640 1 -- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49880c7e30 msgr2=0x7f49881e3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.595+0000 7f4987fff640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49880c7e30 0x7f49881e3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.595+0000 7f4987fff640 1 -- 172.21.15.98:0/888049624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49880c69b0 msgr2=0x7f49881e5460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:49:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.595+0000 7f4987fff640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49880c69b0 0x7f49881e5460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.595+0000 7f4987fff640 1 -- 172.21.15.98:0/888049624 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4978009400 con 0x7f4988153c80 2023-11-28T17:49:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.595+0000 7f497ffff640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49880c7e30 0x7f49881e3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:49:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.596+0000 7f4987fff640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4988153c80 0x7f49881e3b70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f498000be50 tx=0x7f498000be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.596+0000 7f4984ff9640 1 -- 172.21.15.98:0/888049624 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4980016430 con 0x7f4988153c80 2023-11-28T17:49:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.596+0000 7f4984ff9640 1 -- 172.21.15.98:0/888049624 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4980016590 con 0x7f4988153c80 2023-11-28T17:49:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.596+0000 7f4984ff9640 1 -- 172.21.15.98:0/888049624 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4980016980 con 0x7f4988153c80 2023-11-28T17:49:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.596+0000 7f498c8d1640 1 -- 172.21.15.98:0/888049624 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498814deb0 con 0x7f4988153c80 2023-11-28T17:49:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.596+0000 7f498c8d1640 1 -- 172.21.15.98:0/888049624 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49881e3110 con 0x7f4988153c80 2023-11-28T17:49:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.597+0000 7f498c8d1640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498814e540 0x7f49880c5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.597+0000 7f498c8d1640 1 -- 172.21.15.98:0/888049624 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4988068170 con 0x7f498814e540 2023-11-28T17:49:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.598+0000 7f49877fe640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498814e540 0x7f49880c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.598+0000 7f49877fe640 1 -- 172.21.15.98:0/888049624 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f49880cbb10 con 0x7f498814e540 2023-11-28T17:49:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.599+0000 7f49877fe640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498814e540 0x7f49880c5040 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f498806d6e0 tx=0x7f4978004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.599+0000 7f4984ff9640 1 -- 172.21.15.98:0/888049624 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4980012020 con 0x7f4988153c80 2023-11-28T17:49:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.600+0000 7f4984ff9640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f496806b280 0x7f496806d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.600+0000 7f4984ff9640 1 -- 172.21.15.98:0/888049624 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4980089cf0 con 0x7f4988153c80 2023-11-28T17:49:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.600+0000 7f497ffff640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f496806b280 0x7f496806d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.601+0000 7f4984ff9640 1 -- 172.21.15.98:0/888049624 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4978031b60 con 0x7f498814e540 2023-11-28T17:49:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.601+0000 7f4984ff9640 1 -- 172.21.15.98:0/888049624 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f4978031cc0 con 0x7f498814e540 2023-11-28T17:49:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.601+0000 7f4984ff9640 1 -- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498814e540 msgr2=0x7f49880c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.601+0000 7f4984ff9640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498814e540 0x7f49880c5040 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f498806d6e0 tx=0x7f4978004730 comp rx=0 tx=0).stop 2023-11-28T17:49:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.601+0000 7f4984ff9640 1 -- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498814e540 msgr2=0x7f49880c5040 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:49:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.601+0000 7f4984ff9640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f498814e540 0x7f49880c5040 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f498806d6e0 tx=0x7f4978004730 comp rx=0 tx=0).stop 2023-11-28T17:49:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.601+0000 7f497ffff640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f496806b280 0x7f496806d740 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f49880ce9d0 tx=0x7f4970009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.608+0000 7f498c8d1640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49880c5580 0x7f49881f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.608+0000 7f498c8d1640 1 -- 172.21.15.98:0/888049624 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498815dd90 con 0x7f49880c5580 2023-11-28T17:49:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.608+0000 7f49877fe640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49880c5580 0x7f49881f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.609+0000 7f49877fe640 1 -- 172.21.15.98:0/888049624 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f49880b50d0 con 0x7f49880c5580 2023-11-28T17:49:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.610+0000 7f49877fe640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49880c5580 0x7f49881f2e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4978009720 tx=0x7f49780046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.610+0000 7f4984ff9640 1 -- 172.21.15.98:0/888049624 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4978048040 con 0x7f49880c5580 2023-11-28T17:49:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.611+0000 7f4984ff9640 1 -- 172.21.15.98:0/888049624 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4964 (secure 0 0 0) 0x7f4978003d40 con 0x7f49880c5580 2023-11-28T17:49:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.611+0000 7f4984ff9640 1 -- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49880c5580 msgr2=0x7f49881f2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.611+0000 7f4984ff9640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49880c5580 0x7f49881f2e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4978009720 tx=0x7f49780046c0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.611+0000 7f4984ff9640 1 -- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49880c5580 msgr2=0x7f49881f2e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:49:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.611+0000 7f4984ff9640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f49880c5580 0x7f49881f2e00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:07.614 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:49:07.614 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:49:07.614 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:49:07.614 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:49:07.614 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 78, 2023-11-28T17:49:07.614 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:49:07.614 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:49:07.614 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:49:07.615 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:07.615 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2023-11-28T17:49:07.615 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:49:07.615 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:49:07.615 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:49:07.615 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:49:07.615 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:49:07.616 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:49:07.616 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:49:07.616 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:49:07.616 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:49:07.616 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:49:07.616 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:49:07.616 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:49:07.616 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:49:07.616 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:07.616 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:49:07.617 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:49:07.617 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:49:07.617 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:49:07.617 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:49:07.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:49:07.617 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:49:07.617 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:49:07.617 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:49:07.617 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:49:07.617 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:49:07.617 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:49:07.617 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:07.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.618 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:49:07.618 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:49:07.618 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:49:07.618 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:49:07.618 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:49:07.618 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:49:07.618 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:49:07.618 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:49:07.618 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:49:07.618 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:49:07.619 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:49:07.619 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:49:07.619 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:49:07.619 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:49:07.619 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:49:07.619 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:49:07.619 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:49:07.619 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:49:07.619 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:49:07.619 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:49:07.619 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:49:07.620 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:49:07.620 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:49:07.620 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:49:07.620 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:49:07.620 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:49:07.620 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:49:07.620 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:07.620 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:49:07.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.620 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:49:07.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.621 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:49:07.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:49:07.621 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:49:07.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:49:07.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:49:07.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:49:07.621 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:07.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:49:07.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:49:07.622 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:07.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.622 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:07.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:49:07.622 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:49:07.622 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:49:07.622 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:49:07.622 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:49:07.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.623 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:49:07.623 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:49:07.623 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:49:07.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:49:07.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:49:07.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:49:07.623 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:07.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:49:07.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:49:07.624 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:07.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.624 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:07.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:49:07.624 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:49:07.624 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:49:07.624 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:49:07.624 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:49:07.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.625 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:49:07.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:49:07.625 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:49:07.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:49:07.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:49:07.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:49:07.625 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:07.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:49:07.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:49:07.626 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:07.626 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.626 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:07.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:49:07.626 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:49:07.626 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:49:07.626 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:49:07.627 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:49:07.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.627 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:07.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.627 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:49:07.627 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:49:07.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.627 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:07.627 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:07.627 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:49:07.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.627 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:07.627 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:49:07.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.628 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:07.628 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:07.628 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:49:07.628 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.628 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:07.628 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:49:07.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.628 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:07.628 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:07.628 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:49:07.628 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.629 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:07.629 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:49:07.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.629 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:07.629 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:07.629 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:49:07.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.629 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:07.629 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:49:07.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:07.629 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:07.629 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:07.629 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:49:07.630 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:07.630 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:07.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:07.630 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:49:07.630 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:49:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.612+0000 7f498c8d1640 1 -- 172.21.15.98:0/888049624 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f496806b280 msgr2=0x7f496806d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.612+0000 7f498c8d1640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f496806b280 0x7f496806d740 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f49880ce9d0 tx=0x7f4970009040 comp rx=0 tx=0).stop 2023-11-28T17:49:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.612+0000 7f498c8d1640 1 -- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4988153c80 msgr2=0x7f49881e3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.612+0000 7f498c8d1640 1 --2- 172.21.15.98:0/888049624 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4988153c80 0x7f49881e3b70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f498000be50 tx=0x7f498000be80 comp rx=0 tx=0).stop 2023-11-28T17:49:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.612+0000 7f4987fff640 1 -- 172.21.15.98:0/888049624 reap_dead start 2023-11-28T17:49:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.612+0000 7f4987fff640 1 -- 172.21.15.98:0/888049624 reap_dead start 2023-11-28T17:49:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.612+0000 7f498c8d1640 1 -- 172.21.15.98:0/888049624 shutdown_connections 2023-11-28T17:49:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.612+0000 7f498c8d1640 1 -- 172.21.15.98:0/888049624 >> 172.21.15.98:0/888049624 conn(0x7f49880cf300 msgr2=0x7f498806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.612+0000 7f498c8d1640 1 -- 172.21.15.98:0/888049624 shutdown_connections 2023-11-28T17:49:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:07.612+0000 7f498c8d1640 1 -- 172.21.15.98:0/888049624 wait complete. 2023-11-28T17:49:07.632 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2023-11-28T17:49:07.632 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2023-11-28T17:49:08.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~257s 2023-11-28T17:49:10.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:49:10.165 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:49:14.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~263s 2023-11-28T17:49:21.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~270s 2023-11-28T17:49:27.633 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2023-11-28T17:49:27.633 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2023-11-28T17:49:27.633 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-28T17:49:27.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-28T17:49:27.643 INFO:tasks.ceph.mon.a:Started 2023-11-28T17:49:27.643 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:49:27.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:49:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.837+0000 7f4af7844640 1 Processor -- start 2023-11-28T17:49:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.837+0000 7f4af7844640 1 -- start start 2023-11-28T17:49:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.838+0000 7f4af7844640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4af00c8590 0x7f4af0156b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.838+0000 7f4af7844640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4af00c77b0 0x7f4af006ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.838+0000 7f4af7844640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af0157040 con 0x7f4af0150050 2023-11-28T17:49:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.838+0000 7f4af7844640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af006b3d0 con 0x7f4af00c77b0 2023-11-28T17:49:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.838+0000 7f4af7844640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af00c6450 con 0x7f4af00c8590 2023-11-28T17:49:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.839+0000 7f4af5de1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4af00c77b0 0x7f4af006ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.839+0000 7f4af5de1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4af00c77b0 0x7f4af006ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45564/0 (socket says 172.21.15.98:45564) 2023-11-28T17:49:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.839+0000 7f4af5de1640 1 -- 172.21.15.98:0/3594648857 learned_addr learned my addr 172.21.15.98:0/3594648857 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.839+0000 7f4af5de1640 1 -- 172.21.15.98:0/3594648857 >> v2:172.21.15.98:3300/0 conn(0x7f4af00c77b0 msgr2=0x7f4af006ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:49:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.839+0000 7f4af5de1640 1 -- 172.21.15.98:0/3594648857 >> v2:172.21.15.98:3300/0 conn(0x7f4af00c77b0 msgr2=0x7f4af006ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:49:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.839+0000 7f4af5de1640 1 --2- 172.21.15.98:0/3594648857 >> v2:172.21.15.98:3300/0 conn(0x7f4af00c77b0 0x7f4af006ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.839+0000 7f4af5de1640 1 --2- 172.21.15.98:0/3594648857 >> v2:172.21.15.98:3300/0 conn(0x7f4af00c77b0 0x7f4af006ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.839+0000 7f4af4ddf640 1 --2- 172.21.15.98:0/3594648857 >> v2:172.21.15.113:3300/0 conn(0x7f4af00c8590 0x7f4af0156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.840+0000 7f4af4ddf640 1 -- 172.21.15.98:0/3594648857 >> v2:172.21.15.98:3300/0 conn(0x7f4af00c77b0 msgr2=0x7f4af006ddb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:49:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.840+0000 7f4af4ddf640 1 --2- 172.21.15.98:0/3594648857 >> v2:172.21.15.98:3300/0 conn(0x7f4af00c77b0 0x7f4af006ddb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.840+0000 7f4af4ddf640 1 -- 172.21.15.98:0/3594648857 >> v1:172.21.15.113:6789/0 conn(0x7f4af0150050 legacy=0x7f4af00c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.840+0000 7f4af4ddf640 1 -- 172.21.15.98:0/3594648857 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af00c6730 con 0x7f4af00c8590 2023-11-28T17:49:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.841+0000 7f4af4ddf640 1 --2- 172.21.15.98:0/3594648857 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af00c8590 0x7f4af0156b00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4af00cabd0 tx=0x7f4ae002ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c0646336fbe5c70 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.841+0000 7f4ae7fff640 1 -- 172.21.15.98:0/3594648857 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae002dbe0 con 0x7f4af00c8590 2023-11-28T17:49:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.841+0000 7f4ae7fff640 1 -- 172.21.15.98:0/3594648857 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae002dd40 con 0x7f4af00c8590 2023-11-28T17:49:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.841+0000 7f4ae7fff640 1 -- 172.21.15.98:0/3594648857 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0035660 con 0x7f4af00c8590 2023-11-28T17:49:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.842+0000 7f4af7844640 1 -- 172.21.15.98:0/3594648857 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af00c8590 msgr2=0x7f4af0156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.842+0000 7f4af7844640 1 --2- 172.21.15.98:0/3594648857 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af00c8590 0x7f4af0156b00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4af00cabd0 tx=0x7f4ae002ce30 comp rx=0 tx=0).stop 2023-11-28T17:49:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.842+0000 7f4af7844640 1 -- 172.21.15.98:0/3594648857 shutdown_connections 2023-11-28T17:49:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.842+0000 7f4af7844640 1 --2- 172.21.15.98:0/3594648857 >> v2:172.21.15.98:3300/0 conn(0x7f4af00c77b0 0x7f4af006ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.842+0000 7f4af7844640 1 --2- 172.21.15.98:0/3594648857 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af00c8590 0x7f4af0156b00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.842+0000 7f4af7844640 1 -- 172.21.15.98:0/3594648857 >> 172.21.15.98:0/3594648857 conn(0x7f4af006a4d0 msgr2=0x7f4af006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.842+0000 7f4af7844640 1 -- 172.21.15.98:0/3594648857 shutdown_connections 2023-11-28T17:49:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.842+0000 7f4af7844640 1 -- 172.21.15.98:0/3594648857 wait complete. 2023-11-28T17:49:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.844+0000 7f4af7844640 1 Processor -- start 2023-11-28T17:49:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.844+0000 7f4af7844640 1 -- start start 2023-11-28T17:49:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.845+0000 7f4af7844640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4af00c77b0 0x7f4af00715d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.845+0000 7f4af7844640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af00c8590 0x7f4af01f01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.845+0000 7f4af7844640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af0150050 0x7f4af0155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.845+0000 7f4af7844640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af01efd40 con 0x7f4af0150050 2023-11-28T17:49:27.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.845+0000 7f4af7844640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af01ef710 con 0x7f4af00c8590 2023-11-28T17:49:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.845+0000 7f4af7844640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af0070ca0 con 0x7f4af00c77b0 2023-11-28T17:49:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.845+0000 7f4af5de1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af0150050 0x7f4af0155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.845+0000 7f4af5de1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af0150050 0x7f4af0155320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45572/0 (socket says 172.21.15.98:45572) 2023-11-28T17:49:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.845+0000 7f4af5de1640 1 -- 172.21.15.98:0/4036480658 learned_addr learned my addr 172.21.15.98:0/4036480658 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.845+0000 7f4af5de1640 1 -- 172.21.15.98:0/4036480658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af0150050 msgr2=0x7f4af0155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-11-28T17:49:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.845+0000 7f4af5de1640 1 -- 172.21.15.98:0/4036480658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af0150050 msgr2=0x7f4af0155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:49:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.845+0000 7f4af5de1640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af0150050 0x7f4af0155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.845+0000 7f4af5de1640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af0150050 0x7f4af0155320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.845+0000 7f4af55e0640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4af00c77b0 0x7f4af00715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:27.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.845+0000 7f4af4ddf640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af00c8590 0x7f4af01f01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:27.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.846+0000 7f4af55e0640 1 -- 172.21.15.98:0/4036480658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af00c8590 msgr2=0x7f4af01f01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.846+0000 7f4af55e0640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af00c8590 0x7f4af01f01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.846+0000 7f4af55e0640 1 -- 172.21.15.98:0/4036480658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af0150050 msgr2=0x7f4af0155320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:49:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.846+0000 7f4af55e0640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af0150050 0x7f4af0155320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.846+0000 7f4af55e0640 1 -- 172.21.15.98:0/4036480658 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae000a5a0 con 0x7f4af00c77b0 2023-11-28T17:49:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.848+0000 7f4af55e0640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4af00c77b0 0x7f4af00715d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4aec00c390 tx=0x7f4aec00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.848+0000 7f4ae5ffb640 1 -- 172.21.15.98:0/4036480658 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aec0052d0 con 0x7f4af00c77b0 2023-11-28T17:49:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.848+0000 7f4ae5ffb640 1 -- 172.21.15.98:0/4036480658 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aec005430 con 0x7f4af00c77b0 2023-11-28T17:49:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.848+0000 7f4af7844640 1 -- 172.21.15.98:0/4036480658 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af0070f80 con 0x7f4af00c77b0 2023-11-28T17:49:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.848+0000 7f4af7844640 1 -- 172.21.15.98:0/4036480658 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af01ef9d0 con 0x7f4af00c77b0 2023-11-28T17:49:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.848+0000 7f4ae5ffb640 1 -- 172.21.15.98:0/4036480658 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aec003830 con 0x7f4af00c77b0 2023-11-28T17:49:27.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.848+0000 7f4af7844640 1 -- 172.21.15.98:0/4036480658 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af0067bf0 con 0x7f4af00c77b0 2023-11-28T17:49:27.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.850+0000 7f4ae5ffb640 1 -- 172.21.15.98:0/4036480658 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4aec003990 con 0x7f4af00c77b0 2023-11-28T17:49:27.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.851+0000 7f4ae5ffb640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4ad406b2a0 0x7f4ad406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:27.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.851+0000 7f4ae5ffb640 1 -- 172.21.15.98:0/4036480658 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4aec089620 con 0x7f4af00c77b0 2023-11-28T17:49:27.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.851+0000 7f4af4ddf640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4ad406b2a0 0x7f4ad406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:27.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.851+0000 7f4af4ddf640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4ad406b2a0 0x7f4ad406d760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0037000 tx=0x7f4ae0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:27.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:27.864+0000 7f4ae5ffb640 1 -- 172.21.15.98:0/4036480658 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4aec054430 con 0x7f4af00c77b0 2023-11-28T17:49:28.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:28.112+0000 7f4af7844640 1 -- 172.21.15.98:0/4036480658 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4af0074f60 con 0x7f4af00c77b0 2023-11-28T17:49:28.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:28.114+0000 7f4ae5ffb640 1 -- 172.21.15.98:0/4036480658 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7f4aec017080 con 0x7f4af00c77b0 2023-11-28T17:49:28.115 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":78,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:49:28.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:28.116+0000 7f4af7844640 1 -- 172.21.15.98:0/4036480658 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4ad406b2a0 msgr2=0x7f4ad406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:28.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:28.116+0000 7f4af7844640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4ad406b2a0 0x7f4ad406d760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0037000 tx=0x7f4ae0008040 comp rx=0 tx=0).stop 2023-11-28T17:49:28.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:28.116+0000 7f4af7844640 1 -- 172.21.15.98:0/4036480658 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4af00c77b0 msgr2=0x7f4af00715d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:28.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:28.116+0000 7f4af7844640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4af00c77b0 0x7f4af00715d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4aec00c390 tx=0x7f4aec00c850 comp rx=0 tx=0).stop 2023-11-28T17:49:28.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:28.116+0000 7f4af7844640 1 -- 172.21.15.98:0/4036480658 shutdown_connections 2023-11-28T17:49:28.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:28.116+0000 7f4af7844640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af0150050 0x7f4af0155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:28.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:28.116+0000 7f4af7844640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f4ad406b2a0 0x7f4ad406d760 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:28.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:28.116+0000 7f4af7844640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4af00c8590 0x7f4af01f01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:28.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:28.116+0000 7f4af7844640 1 --2- 172.21.15.98:0/4036480658 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4af00c77b0 0x7f4af00715d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:28.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:28.116+0000 7f4af7844640 1 -- 172.21.15.98:0/4036480658 >> 172.21.15.98:0/4036480658 conn(0x7f4af006a4d0 msgr2=0x7f4af014eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:28.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:28.118+0000 7f4af7844640 1 -- 172.21.15.98:0/4036480658 shutdown_connections 2023-11-28T17:49:28.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:28.118+0000 7f4af7844640 1 -- 172.21.15.98:0/4036480658 wait complete. 2023-11-28T17:49:28.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~277s 2023-11-28T17:49:31.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:49:31.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.360+0000 7f6fe9db0640 1 Processor -- start 2023-11-28T17:49:31.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.360+0000 7f6fe9db0640 1 -- start start 2023-11-28T17:49:31.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.361+0000 7f6fe9db0640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f6fe4150c00 0x7f6fe40c2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:31.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.361+0000 7f6fe9db0640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40c6600 0x7f6fe40c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:31.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.361+0000 7f6fe9db0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe4157850 con 0x7f6fe40c5c00 2023-11-28T17:49:31.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.361+0000 7f6fe9db0640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe40c6fd0 con 0x7f6fe4150c00 2023-11-28T17:49:31.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.361+0000 7f6fe9db0640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4156980 con 0x7f6fe40c6600 2023-11-28T17:49:31.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.362+0000 7f6fe2ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6fe40c5c00 0x7f6fe40c44e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57472/0 (socket says 172.21.15.98:57472) 2023-11-28T17:49:31.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.362+0000 7f6fe2ffd640 1 -- 172.21.15.98:0/239401032 learned_addr learned my addr 172.21.15.98:0/239401032 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:49:31.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.362+0000 7f6fe3fff640 1 --2- 172.21.15.98:0/239401032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40c6600 0x7f6fe40c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:31.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.362+0000 7f6fe37fe640 1 --2- 172.21.15.98:0/239401032 >> v2:172.21.15.113:3300/0 conn(0x7f6fe4150c00 0x7f6fe40c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:31.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.363+0000 7f6fe3fff640 1 -- 172.21.15.98:0/239401032 >> v2:172.21.15.113:3300/0 conn(0x7f6fe4150c00 msgr2=0x7f6fe40c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:31.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.363+0000 7f6fe3fff640 1 --2- 172.21.15.98:0/239401032 >> v2:172.21.15.113:3300/0 conn(0x7f6fe4150c00 0x7f6fe40c2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:31.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.363+0000 7f6fe3fff640 1 -- 172.21.15.98:0/239401032 >> v1:172.21.15.98:6789/0 conn(0x7f6fe40c5c00 legacy=0x7f6fe40c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:31.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.363+0000 7f6fe3fff640 1 -- 172.21.15.98:0/239401032 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe4156c40 con 0x7f6fe40c6600 2023-11-28T17:49:31.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.364+0000 7f6fe3fff640 1 --2- 172.21.15.98:0/239401032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40c6600 0x7f6fe40c6a90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009d90 tx=0x7f6fdc02e0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e665484bed49886f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:31.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.365+0000 7f6fe27fc640 1 -- 172.21.15.98:0/239401032 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc02ee70 con 0x7f6fe40c6600 2023-11-28T17:49:31.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.365+0000 7f6fe27fc640 1 -- 172.21.15.98:0/239401032 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fdc036440 con 0x7f6fe40c6600 2023-11-28T17:49:31.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.365+0000 7f6fe27fc640 1 -- 172.21.15.98:0/239401032 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0367d0 con 0x7f6fe40c6600 2023-11-28T17:49:31.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.366+0000 7f6fe9db0640 1 -- 172.21.15.98:0/239401032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40c6600 msgr2=0x7f6fe40c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:31.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.366+0000 7f6fe9db0640 1 --2- 172.21.15.98:0/239401032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40c6600 0x7f6fe40c6a90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009d90 tx=0x7f6fdc02e0c0 comp rx=0 tx=0).stop 2023-11-28T17:49:31.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.366+0000 7f6fe9db0640 1 -- 172.21.15.98:0/239401032 shutdown_connections 2023-11-28T17:49:31.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.366+0000 7f6fe9db0640 1 --2- 172.21.15.98:0/239401032 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40c6600 0x7f6fe40c6a90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:31.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.366+0000 7f6fe9db0640 1 --2- 172.21.15.98:0/239401032 >> v2:172.21.15.113:3300/0 conn(0x7f6fe4150c00 0x7f6fe40c2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:31.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.366+0000 7f6fe9db0640 1 -- 172.21.15.98:0/239401032 >> 172.21.15.98:0/239401032 conn(0x7f6fe406a4d0 msgr2=0x7f6fe406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:31.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.366+0000 7f6fe9db0640 1 -- 172.21.15.98:0/239401032 shutdown_connections 2023-11-28T17:49:31.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.366+0000 7f6fe9db0640 1 -- 172.21.15.98:0/239401032 wait complete. 2023-11-28T17:49:31.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.368+0000 7f6fe9db0640 1 Processor -- start 2023-11-28T17:49:31.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.368+0000 7f6fe9db0640 1 -- start start 2023-11-28T17:49:31.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.369+0000 7f6fe9db0640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40c5c00 0x7f6fe4151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:31.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.369+0000 7f6fe9db0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe40c6600 0x7f6fe41ee4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:31.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.369+0000 7f6fe9db0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe4150c00 0x7f6fe4152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:31.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.369+0000 7f6fe9db0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4151fa0 con 0x7f6fe40c6600 2023-11-28T17:49:31.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.369+0000 7f6fe9db0640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4147980 con 0x7f6fe4150c00 2023-11-28T17:49:31.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.369+0000 7f6fe9db0640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4147c60 con 0x7f6fe40c5c00 2023-11-28T17:49:31.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.369+0000 7f6fe2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe40c6600 0x7f6fe41ee4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:31.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.369+0000 7f6fe2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe40c6600 0x7f6fe41ee4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39110/0 (socket says 172.21.15.98:39110) 2023-11-28T17:49:31.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.369+0000 7f6fe2ffd640 1 -- 172.21.15.98:0/2925152418 learned_addr learned my addr 172.21.15.98:0/2925152418 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:31.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.369+0000 7f6fe2ffd640 1 -- 172.21.15.98:0/2925152418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe40c6600 msgr2=0x7f6fe41ee4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-11-28T17:49:31.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.369+0000 7f6fe2ffd640 1 -- 172.21.15.98:0/2925152418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe40c6600 msgr2=0x7f6fe41ee4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:49:31.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.369+0000 7f6fe2ffd640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe40c6600 0x7f6fe41ee4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:31.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.370+0000 7f6fe37fe640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40c5c00 0x7f6fe4151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:31.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.370+0000 7f6fe2ffd640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe40c6600 0x7f6fe41ee4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:31.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.370+0000 7f6fe3fff640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe4150c00 0x7f6fe4152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:31.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.370+0000 7f6fe37fe640 1 -- 172.21.15.98:0/2925152418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe4150c00 msgr2=0x7f6fe4152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:31.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.370+0000 7f6fe37fe640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe4150c00 0x7f6fe4152ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:31.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.370+0000 7f6fe37fe640 1 -- 172.21.15.98:0/2925152418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe40c6600 msgr2=0x7f6fe41ee4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:49:31.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.370+0000 7f6fe37fe640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe40c6600 0x7f6fe41ee4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:31.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.370+0000 7f6fe37fe640 1 -- 172.21.15.98:0/2925152418 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc009a70 con 0x7f6fe40c5c00 2023-11-28T17:49:31.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.371+0000 7f6fe37fe640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40c5c00 0x7f6fe4151a60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400e870 tx=0x7f6fd400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:31.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.371+0000 7f6fcbfff640 1 -- 172.21.15.98:0/2925152418 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd400ee60 con 0x7f6fe40c5c00 2023-11-28T17:49:31.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.371+0000 7f6fcbfff640 1 -- 172.21.15.98:0/2925152418 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd400c940 con 0x7f6fe40c5c00 2023-11-28T17:49:31.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.371+0000 7f6fe9db0640 1 -- 172.21.15.98:0/2925152418 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe41eed80 con 0x7f6fe40c5c00 2023-11-28T17:49:31.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.372+0000 7f6fcbfff640 1 -- 172.21.15.98:0/2925152418 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd400ccd0 con 0x7f6fe40c5c00 2023-11-28T17:49:31.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.372+0000 7f6fe9db0640 1 -- 172.21.15.98:0/2925152418 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe41529b0 con 0x7f6fe40c5c00 2023-11-28T17:49:31.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.373+0000 7f6fe9db0640 1 -- 172.21.15.98:0/2925152418 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe4067bf0 con 0x7f6fe40c5c00 2023-11-28T17:49:31.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.374+0000 7f6fcbfff640 1 -- 172.21.15.98:0/2925152418 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6fd4013470 con 0x7f6fe40c5c00 2023-11-28T17:49:31.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.376+0000 7f6fcbfff640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6fd006b2d0 0x7f6fd006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:31.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.376+0000 7f6fcbfff640 1 -- 172.21.15.98:0/2925152418 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6fd408a3b0 con 0x7f6fe40c5c00 2023-11-28T17:49:31.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.376+0000 7f6fe2ffd640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6fd006b2d0 0x7f6fd006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:31.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.376+0000 7f6fe2ffd640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6fd006b2d0 0x7f6fd006d790 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40676f0 tx=0x7f6fd8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:31.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.388+0000 7f6fcbfff640 1 -- 172.21.15.98:0/2925152418 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6fd4055230 con 0x7f6fe40c5c00 2023-11-28T17:49:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.649+0000 7f6fe9db0640 1 -- 172.21.15.98:0/2925152418 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6fe40c8840 con 0x7f6fe40c5c00 2023-11-28T17:49:31.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.650+0000 7f6fcbfff640 1 -- 172.21.15.98:0/2925152418 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7f6fd4017620 con 0x7f6fe40c5c00 2023-11-28T17:49:31.651 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":78,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":25,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:49:31.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.653+0000 7f6fe9db0640 1 -- 172.21.15.98:0/2925152418 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6fd006b2d0 msgr2=0x7f6fd006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:31.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.653+0000 7f6fe9db0640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6fd006b2d0 0x7f6fd006d790 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40676f0 tx=0x7f6fd8009040 comp rx=0 tx=0).stop 2023-11-28T17:49:31.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.653+0000 7f6fe9db0640 1 -- 172.21.15.98:0/2925152418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40c5c00 msgr2=0x7f6fe4151a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:31.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.653+0000 7f6fe9db0640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40c5c00 0x7f6fe4151a60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400e870 tx=0x7f6fd400ed30 comp rx=0 tx=0).stop 2023-11-28T17:49:31.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.653+0000 7f6fe9db0640 1 -- 172.21.15.98:0/2925152418 shutdown_connections 2023-11-28T17:49:31.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.653+0000 7f6fe9db0640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f6fd006b2d0 0x7f6fd006d790 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:31.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.653+0000 7f6fe9db0640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6fe4150c00 0x7f6fe4152ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:31.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.653+0000 7f6fe9db0640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fe40c6600 0x7f6fe41ee4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:31.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.653+0000 7f6fe9db0640 1 --2- 172.21.15.98:0/2925152418 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6fe40c5c00 0x7f6fe4151a60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:31.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.653+0000 7f6fe9db0640 1 -- 172.21.15.98:0/2925152418 >> 172.21.15.98:0/2925152418 conn(0x7f6fe406a4d0 msgr2=0x7f6fe406dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:31.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.653+0000 7f6fe9db0640 1 -- 172.21.15.98:0/2925152418 shutdown_connections 2023-11-28T17:49:31.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:31.654+0000 7f6fe9db0640 1 -- 172.21.15.98:0/2925152418 wait complete. 2023-11-28T17:49:34.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:49:34.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.900+0000 7f2014e7c640 1 Processor -- start 2023-11-28T17:49:34.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.900+0000 7f2014e7c640 1 -- start start 2023-11-28T17:49:34.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.902+0000 7f2014e7c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2010151d80 0x7f2010157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:34.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.902+0000 7f2014e7c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20100c4910 0x7f20100c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:34.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.902+0000 7f2014e7c640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f201015b150 con 0x7f201015bc10 2023-11-28T17:49:34.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.902+0000 7f2014e7c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f201006e1f0 con 0x7f20100c4910 2023-11-28T17:49:34.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.902+0000 7f2014e7c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f201006e4d0 con 0x7f2010151d80 2023-11-28T17:49:34.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.902+0000 7f200ed9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20100c4910 0x7f20100c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:34.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.902+0000 7f200ed9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20100c4910 0x7f20100c4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39118/0 (socket says 172.21.15.98:39118) 2023-11-28T17:49:34.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.902+0000 7f200ed9d640 1 -- 172.21.15.98:0/117042752 learned_addr learned my addr 172.21.15.98:0/117042752 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:34.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.903+0000 7f200e59c640 1 --2- 172.21.15.98:0/117042752 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2010151d80 0x7f2010157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:34.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.903+0000 7f200ed9d640 1 -- 172.21.15.98:0/117042752 >> v2:172.21.15.98:3300/0 conn(0x7f20100c4910 msgr2=0x7f20100c4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:49:34.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.903+0000 7f200ed9d640 1 -- 172.21.15.98:0/117042752 >> v2:172.21.15.98:3300/0 conn(0x7f20100c4910 msgr2=0x7f20100c4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:49:34.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.903+0000 7f200ed9d640 1 --2- 172.21.15.98:0/117042752 >> v2:172.21.15.98:3300/0 conn(0x7f20100c4910 0x7f20100c4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:34.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.903+0000 7f200ed9d640 1 --2- 172.21.15.98:0/117042752 >> v2:172.21.15.98:3300/0 conn(0x7f20100c4910 0x7f20100c4da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.903+0000 7f200e59c640 1 -- 172.21.15.98:0/117042752 >> v2:172.21.15.98:3300/0 conn(0x7f20100c4910 msgr2=0x7f20100c4da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:49:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.903+0000 7f200e59c640 1 --2- 172.21.15.98:0/117042752 >> v2:172.21.15.98:3300/0 conn(0x7f20100c4910 0x7f20100c4da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.903+0000 7f200e59c640 1 -- 172.21.15.98:0/117042752 >> v1:172.21.15.113:6789/0 conn(0x7f201015bc10 legacy=0x7f20100c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.903+0000 7f200e59c640 1 -- 172.21.15.98:0/117042752 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201006e7b0 con 0x7f2010151d80 2023-11-28T17:49:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.903+0000 7f200dd9b640 1 -- 172.21.15.98:0/117042752 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217667638 0 0) 0x7f201015b150 con 0x7f201015bc10 2023-11-28T17:49:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.905+0000 7f200e59c640 1 --2- 172.21.15.98:0/117042752 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2010151d80 0x7f2010157b60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f20000097a0 tx=0x7f200002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82c5af5bf5e31647 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.905+0000 7f200dd9b640 1 -- 172.21.15.98:0/117042752 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2000030610 con 0x7f2010151d80 2023-11-28T17:49:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.905+0000 7f200dd9b640 1 -- 172.21.15.98:0/117042752 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2000030770 con 0x7f2010151d80 2023-11-28T17:49:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.905+0000 7f200dd9b640 1 -- 172.21.15.98:0/117042752 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2000030b00 con 0x7f2010151d80 2023-11-28T17:49:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.906+0000 7f2014e7c640 1 -- 172.21.15.98:0/117042752 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2010151d80 msgr2=0x7f2010157b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.906+0000 7f2014e7c640 1 --2- 172.21.15.98:0/117042752 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2010151d80 0x7f2010157b60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f20000097a0 tx=0x7f200002b960 comp rx=0 tx=0).stop 2023-11-28T17:49:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.906+0000 7f2014e7c640 1 -- 172.21.15.98:0/117042752 shutdown_connections 2023-11-28T17:49:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.906+0000 7f2014e7c640 1 --2- 172.21.15.98:0/117042752 >> v2:172.21.15.98:3300/0 conn(0x7f20100c4910 0x7f20100c4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.906+0000 7f2014e7c640 1 --2- 172.21.15.98:0/117042752 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2010151d80 0x7f2010157b60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.906+0000 7f2014e7c640 1 -- 172.21.15.98:0/117042752 >> 172.21.15.98:0/117042752 conn(0x7f201006a4d0 msgr2=0x7f201006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.906+0000 7f2014e7c640 1 -- 172.21.15.98:0/117042752 shutdown_connections 2023-11-28T17:49:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.906+0000 7f2014e7c640 1 -- 172.21.15.98:0/117042752 wait complete. 2023-11-28T17:49:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.908+0000 7f2014e7c640 1 Processor -- start 2023-11-28T17:49:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.908+0000 7f2014e7c640 1 -- start start 2023-11-28T17:49:34.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.909+0000 7f2014e7c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20100c4910 0x7f2010070da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:34.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.909+0000 7f2014e7c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2010151d80 0x7f20101eecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:34.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.909+0000 7f2014e7c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f201015bc10 0x7f20101ef230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:34.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.909+0000 7f2014e7c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20101ee550 con 0x7f20100c4910 2023-11-28T17:49:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.909+0000 7f2014e7c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2010156260 con 0x7f201015bc10 2023-11-28T17:49:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.909+0000 7f2014e7c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20101ed6d0 con 0x7f2010151d80 2023-11-28T17:49:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.909+0000 7f200e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20100c4910 0x7f2010070da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.909+0000 7f200e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20100c4910 0x7f2010070da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39120/0 (socket says 172.21.15.98:39120) 2023-11-28T17:49:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.909+0000 7f200e59c640 1 -- 172.21.15.98:0/1197279611 learned_addr learned my addr 172.21.15.98:0/1197279611 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.909+0000 7f200e59c640 1 -- 172.21.15.98:0/1197279611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20100c4910 msgr2=0x7f2010070da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-11-28T17:49:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.909+0000 7f200e59c640 1 -- 172.21.15.98:0/1197279611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20100c4910 msgr2=0x7f2010070da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:49:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.909+0000 7f200e59c640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20100c4910 0x7f2010070da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.910+0000 7f2007fff640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2010151d80 0x7f20101eecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.910+0000 7f200e59c640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20100c4910 0x7f2010070da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.910+0000 7f200ed9d640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f201015bc10 0x7f20101ef230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.910+0000 7f2007fff640 1 -- 172.21.15.98:0/1197279611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f201015bc10 msgr2=0x7f20101ef230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.910+0000 7f2007fff640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f201015bc10 0x7f20101ef230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.910+0000 7f2007fff640 1 -- 172.21.15.98:0/1197279611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20100c4910 msgr2=0x7f2010070da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:49:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.910+0000 7f2007fff640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20100c4910 0x7f2010070da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.910+0000 7f2007fff640 1 -- 172.21.15.98:0/1197279611 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2000009400 con 0x7f2010151d80 2023-11-28T17:49:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.911+0000 7f2007fff640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2010151d80 0x7f20101eecf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800c450 tx=0x7f1ff800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.911+0000 7f2006ffd640 1 -- 172.21.15.98:0/1197279611 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff80035f0 con 0x7f2010151d80 2023-11-28T17:49:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.911+0000 7f2006ffd640 1 -- 172.21.15.98:0/1197279611 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff8003750 con 0x7f2010151d80 2023-11-28T17:49:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.911+0000 7f2006ffd640 1 -- 172.21.15.98:0/1197279611 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8003b00 con 0x7f2010151d80 2023-11-28T17:49:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.911+0000 7f2014e7c640 1 -- 172.21.15.98:0/1197279611 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20101ed940 con 0x7f2010151d80 2023-11-28T17:49:34.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.911+0000 7f2014e7c640 1 -- 172.21.15.98:0/1197279611 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20101edf00 con 0x7f2010151d80 2023-11-28T17:49:34.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.912+0000 7f2014e7c640 1 -- 172.21.15.98:0/1197279611 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2010067bf0 con 0x7f2010151d80 2023-11-28T17:49:34.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.914+0000 7f2006ffd640 1 -- 172.21.15.98:0/1197279611 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f1ff8003c60 con 0x7f2010151d80 2023-11-28T17:49:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.915+0000 7f2006ffd640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1ff006b2a0 0x7f1ff006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.915+0000 7f2006ffd640 1 -- 172.21.15.98:0/1197279611 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1ff8089560 con 0x7f2010151d80 2023-11-28T17:49:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.915+0000 7f200e59c640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1ff006b2a0 0x7f1ff006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:34.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.916+0000 7f200e59c640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1ff006b2a0 0x7f1ff006d760 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f200002b9a0 tx=0x7f2000008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:34.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:34.927+0000 7f2006ffd640 1 -- 172.21.15.98:0/1197279611 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ff8054450 con 0x7f2010151d80 2023-11-28T17:49:35.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~283s 2023-11-28T17:49:35.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:35.186+0000 7f2014e7c640 1 -- 172.21.15.98:0/1197279611 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f20101eea30 con 0x7f2010151d80 2023-11-28T17:49:35.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:35.187+0000 7f2006ffd640 1 -- 172.21.15.98:0/1197279611 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7f20101eea30 con 0x7f2010151d80 2023-11-28T17:49:35.188 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":78,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":28,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:49:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:35.190+0000 7f2014e7c640 1 -- 172.21.15.98:0/1197279611 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1ff006b2a0 msgr2=0x7f1ff006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:35.190+0000 7f2014e7c640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1ff006b2a0 0x7f1ff006d760 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f200002b9a0 tx=0x7f2000008030 comp rx=0 tx=0).stop 2023-11-28T17:49:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:35.190+0000 7f2014e7c640 1 -- 172.21.15.98:0/1197279611 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2010151d80 msgr2=0x7f20101eecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:35.190+0000 7f2014e7c640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2010151d80 0x7f20101eecf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800c450 tx=0x7f1ff800c910 comp rx=0 tx=0).stop 2023-11-28T17:49:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:35.190+0000 7f2014e7c640 1 -- 172.21.15.98:0/1197279611 shutdown_connections 2023-11-28T17:49:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:35.190+0000 7f2014e7c640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1ff006b2a0 0x7f1ff006d760 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:35.190+0000 7f2014e7c640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f201015bc10 0x7f20101ef230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:35.190+0000 7f2014e7c640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2010151d80 0x7f20101eecf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:35.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:35.190+0000 7f2014e7c640 1 --2- 172.21.15.98:0/1197279611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20100c4910 0x7f2010070da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:35.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:35.190+0000 7f2014e7c640 1 -- 172.21.15.98:0/1197279611 >> 172.21.15.98:0/1197279611 conn(0x7f201006a4d0 msgr2=0x7f201014f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:35.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:35.190+0000 7f2014e7c640 1 -- 172.21.15.98:0/1197279611 shutdown_connections 2023-11-28T17:49:35.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:35.190+0000 7f2014e7c640 1 -- 172.21.15.98:0/1197279611 wait complete. 2023-11-28T17:49:38.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:49:38.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.436+0000 7ffaf3989640 1 Processor -- start 2023-11-28T17:49:38.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.436+0000 7ffaf3989640 1 -- start start 2023-11-28T17:49:38.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.437+0000 7ffaf3989640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffaec150c00 0x7ffaec0c2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:38.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.437+0000 7ffaf3989640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7ffaec0c6600 0x7ffaec0c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:38.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.437+0000 7ffaf3989640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec157850 con 0x7ffaec0c5c00 2023-11-28T17:49:38.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.437+0000 7ffaf3989640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaec0c6fa0 con 0x7ffaec0c6600 2023-11-28T17:49:38.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.437+0000 7ffaf3989640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec156980 con 0x7ffaec150c00 2023-11-28T17:49:38.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.438+0000 7ffaf0f24640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ffaec0c5c00 0x7ffaec0c44e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57488/0 (socket says 172.21.15.98:57488) 2023-11-28T17:49:38.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.438+0000 7ffaf0f24640 1 -- 172.21.15.98:0/2971150457 learned_addr learned my addr 172.21.15.98:0/2971150457 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:49:38.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.438+0000 7ffaf1725640 1 --2- 172.21.15.98:0/2971150457 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffaec150c00 0x7ffaec0c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:38.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.438+0000 7ffaf1f26640 1 --2- 172.21.15.98:0/2971150457 >> v2:172.21.15.113:3300/0 conn(0x7ffaec0c6600 0x7ffaec0c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:38.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.439+0000 7ffaf1f26640 1 -- 172.21.15.98:0/2971150457 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffaec150c00 msgr2=0x7ffaec0c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:38.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.439+0000 7ffaf1f26640 1 --2- 172.21.15.98:0/2971150457 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffaec150c00 0x7ffaec0c2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:38.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.439+0000 7ffaf1f26640 1 -- 172.21.15.98:0/2971150457 >> v1:172.21.15.98:6789/0 conn(0x7ffaec0c5c00 legacy=0x7ffaec0c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:38.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.439+0000 7ffaf1f26640 1 -- 172.21.15.98:0/2971150457 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec156c40 con 0x7ffaec0c6600 2023-11-28T17:49:38.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.440+0000 7ffaf1f26640 1 --2- 172.21.15.98:0/2971150457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaec0c6600 0x7ffaec0c6a60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ffae4009d90 tx=0x7ffae402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bdace1164b4e08f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:38.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.440+0000 7ffae3fff640 1 -- 172.21.15.98:0/2971150457 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae402cc50 con 0x7ffaec0c6600 2023-11-28T17:49:38.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.440+0000 7ffae3fff640 1 -- 172.21.15.98:0/2971150457 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae402cdb0 con 0x7ffaec0c6600 2023-11-28T17:49:38.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.440+0000 7ffae3fff640 1 -- 172.21.15.98:0/2971150457 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae4034660 con 0x7ffaec0c6600 2023-11-28T17:49:38.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.441+0000 7ffaf3989640 1 -- 172.21.15.98:0/2971150457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaec0c6600 msgr2=0x7ffaec0c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:38.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.441+0000 7ffaf3989640 1 --2- 172.21.15.98:0/2971150457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaec0c6600 0x7ffaec0c6a60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ffae4009d90 tx=0x7ffae402bea0 comp rx=0 tx=0).stop 2023-11-28T17:49:38.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.441+0000 7ffaf3989640 1 -- 172.21.15.98:0/2971150457 shutdown_connections 2023-11-28T17:49:38.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.441+0000 7ffaf3989640 1 --2- 172.21.15.98:0/2971150457 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaec0c6600 0x7ffaec0c6a60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:38.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.441+0000 7ffaf3989640 1 --2- 172.21.15.98:0/2971150457 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffaec150c00 0x7ffaec0c2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:38.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.441+0000 7ffaf3989640 1 -- 172.21.15.98:0/2971150457 >> 172.21.15.98:0/2971150457 conn(0x7ffaec06a4d0 msgr2=0x7ffaec06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:38.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.441+0000 7ffaf3989640 1 -- 172.21.15.98:0/2971150457 shutdown_connections 2023-11-28T17:49:38.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.441+0000 7ffaf3989640 1 -- 172.21.15.98:0/2971150457 wait complete. 2023-11-28T17:49:38.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.443+0000 7ffaf3989640 1 Processor -- start 2023-11-28T17:49:38.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.443+0000 7ffaf3989640 1 -- start start 2023-11-28T17:49:38.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.444+0000 7ffaf3989640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaec0c5c00 0x7ffaec1f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:38.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.444+0000 7ffaf3989640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffaec0c6600 0x7ffaec147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:38.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.444+0000 7ffaf3989640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec150c00 0x7ffaec1483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:38.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.444+0000 7ffaf3989640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec145d60 con 0x7ffaec150c00 2023-11-28T17:49:38.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.444+0000 7ffaf3989640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec1f3720 con 0x7ffaec0c5c00 2023-11-28T17:49:38.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.444+0000 7ffaf3989640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec1f39e0 con 0x7ffaec0c6600 2023-11-28T17:49:38.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.444+0000 7ffaf1f26640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec150c00 0x7ffaec1483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:38.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.444+0000 7ffaf1f26640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec150c00 0x7ffaec1483e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39130/0 (socket says 172.21.15.98:39130) 2023-11-28T17:49:38.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.444+0000 7ffaf1f26640 1 -- 172.21.15.98:0/181654031 learned_addr learned my addr 172.21.15.98:0/181654031 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:38.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.444+0000 7ffaf1f26640 1 -- 172.21.15.98:0/181654031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec150c00 msgr2=0x7ffaec1483e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-11-28T17:49:38.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.444+0000 7ffaf1f26640 1 -- 172.21.15.98:0/181654031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec150c00 msgr2=0x7ffaec1483e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:49:38.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.444+0000 7ffaf1f26640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec150c00 0x7ffaec1483e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:38.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.444+0000 7ffaf1f26640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec150c00 0x7ffaec1483e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:38.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.444+0000 7ffaf1725640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaec0c5c00 0x7ffaec1f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:38.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.444+0000 7ffaf0f24640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffaec0c6600 0x7ffaec147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:38.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.445+0000 7ffaf1725640 1 -- 172.21.15.98:0/181654031 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffaec0c6600 msgr2=0x7ffaec147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:38.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.445+0000 7ffaf1725640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffaec0c6600 0x7ffaec147ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:38.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.445+0000 7ffaf1725640 1 -- 172.21.15.98:0/181654031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec150c00 msgr2=0x7ffaec1483e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:49:38.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.445+0000 7ffaf1725640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec150c00 0x7ffaec1483e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:38.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.445+0000 7ffaf1725640 1 -- 172.21.15.98:0/181654031 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae4009a70 con 0x7ffaec0c5c00 2023-11-28T17:49:38.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.445+0000 7ffaf1725640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaec0c5c00 0x7ffaec1f2df0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffae800e8a0 tx=0x7ffae800ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:38.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.446+0000 7ffae1ffb640 1 -- 172.21.15.98:0/181654031 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8009e60 con 0x7ffaec0c5c00 2023-11-28T17:49:38.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.446+0000 7ffae1ffb640 1 -- 172.21.15.98:0/181654031 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae800c940 con 0x7ffaec0c5c00 2023-11-28T17:49:38.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.446+0000 7ffae1ffb640 1 -- 172.21.15.98:0/181654031 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae800ccd0 con 0x7ffaec0c5c00 2023-11-28T17:49:38.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.446+0000 7ffaf3989640 1 -- 172.21.15.98:0/181654031 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec1454c0 con 0x7ffaec0c5c00 2023-11-28T17:49:38.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.446+0000 7ffaf3989640 1 -- 172.21.15.98:0/181654031 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec145a80 con 0x7ffaec0c5c00 2023-11-28T17:49:38.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.447+0000 7ffaf3989640 1 -- 172.21.15.98:0/181654031 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaec067bf0 con 0x7ffaec0c5c00 2023-11-28T17:49:38.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.448+0000 7ffae1ffb640 1 -- 172.21.15.98:0/181654031 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ffae8013470 con 0x7ffaec0c5c00 2023-11-28T17:49:38.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.449+0000 7ffae1ffb640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ffad006b2a0 0x7ffad006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:38.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.449+0000 7ffae1ffb640 1 -- 172.21.15.98:0/181654031 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ffae808a4c0 con 0x7ffaec0c5c00 2023-11-28T17:49:38.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.450+0000 7ffaf0f24640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ffad006b2a0 0x7ffad006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:38.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.450+0000 7ffaf0f24640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ffad006b2a0 0x7ffad006d760 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0c2aa0 tx=0x7ffadc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:38.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.461+0000 7ffae1ffb640 1 -- 172.21.15.98:0/181654031 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffae80552b0 con 0x7ffaec0c5c00 2023-11-28T17:49:38.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.706+0000 7ffaf3989640 1 -- 172.21.15.98:0/181654031 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ffaec074f60 con 0x7ffaec0c5c00 2023-11-28T17:49:38.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.708+0000 7ffae1ffb640 1 -- 172.21.15.98:0/181654031 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7ffae8021620 con 0x7ffaec0c5c00 2023-11-28T17:49:38.708 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":78,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":32,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:49:38.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.710+0000 7ffaf3989640 1 -- 172.21.15.98:0/181654031 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ffad006b2a0 msgr2=0x7ffad006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:38.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.710+0000 7ffaf3989640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ffad006b2a0 0x7ffad006d760 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0c2aa0 tx=0x7ffadc009040 comp rx=0 tx=0).stop 2023-11-28T17:49:38.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.710+0000 7ffaf3989640 1 -- 172.21.15.98:0/181654031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaec0c5c00 msgr2=0x7ffaec1f2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:38.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.710+0000 7ffaf3989640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaec0c5c00 0x7ffaec1f2df0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffae800e8a0 tx=0x7ffae800ed60 comp rx=0 tx=0).stop 2023-11-28T17:49:38.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.710+0000 7ffaf3989640 1 -- 172.21.15.98:0/181654031 shutdown_connections 2023-11-28T17:49:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.710+0000 7ffaf3989640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaec150c00 0x7ffaec1483e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.710+0000 7ffaf3989640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7ffaec0c6600 0x7ffaec147ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.710+0000 7ffaf3989640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7ffad006b2a0 0x7ffad006d760 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.710+0000 7ffaf3989640 1 --2- 172.21.15.98:0/181654031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffaec0c5c00 0x7ffaec1f2df0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.710+0000 7ffaf3989640 1 -- 172.21.15.98:0/181654031 >> 172.21.15.98:0/181654031 conn(0x7ffaec06a4d0 msgr2=0x7ffaec06dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.710+0000 7ffaf3989640 1 -- 172.21.15.98:0/181654031 shutdown_connections 2023-11-28T17:49:38.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:38.710+0000 7ffaf3989640 1 -- 172.21.15.98:0/181654031 wait complete. 2023-11-28T17:49:40.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:49:40.317 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:49:41.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:49:41.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~290s 2023-11-28T17:49:41.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.952+0000 7f99de163640 1 Processor -- start 2023-11-28T17:49:41.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.952+0000 7f99de163640 1 -- start start 2023-11-28T17:49:41.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.954+0000 7f99de163640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f99d8159bf0 0x7f99d8161880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:41.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.954+0000 7f99de163640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d80c08c0 con 0x7f99d8150250 2023-11-28T17:49:41.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.954+0000 7f99de163640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d80c7530 con 0x7f99d80c6d40 2023-11-28T17:49:41.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.954+0000 7f99de163640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d806e910 con 0x7f99d8159bf0 2023-11-28T17:49:41.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.954+0000 7f99dd161640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f99d8150250 0x7f99d80c7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44500/0 (socket says 172.21.15.98:44500) 2023-11-28T17:49:41.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.954+0000 7f99dd161640 1 -- 172.21.15.98:0/3930969934 learned_addr learned my addr 172.21.15.98:0/3930969934 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:49:41.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.954+0000 7f99dd962640 1 --2- 172.21.15.98:0/3930969934 >> v2:172.21.15.98:3300/0 conn(0x7f99d8159bf0 0x7f99d8161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.955+0000 7f99dd962640 1 -- 172.21.15.98:0/3930969934 >> v2:172.21.15.98:3300/0 conn(0x7f99d8159bf0 msgr2=0x7f99d8161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:49:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.955+0000 7f99dd962640 1 -- 172.21.15.98:0/3930969934 >> v2:172.21.15.98:3300/0 conn(0x7f99d8159bf0 msgr2=0x7f99d8161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:49:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.955+0000 7f99dd962640 1 --2- 172.21.15.98:0/3930969934 >> v2:172.21.15.98:3300/0 conn(0x7f99d8159bf0 0x7f99d8161880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.955+0000 7f99dd962640 1 --2- 172.21.15.98:0/3930969934 >> v2:172.21.15.98:3300/0 conn(0x7f99d8159bf0 0x7f99d8161880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.955+0000 7f99cffff640 1 -- 172.21.15.98:0/3930969934 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854659158 0 0) 0x7f99d80c7530 con 0x7f99d80c6d40 2023-11-28T17:49:41.956 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.955+0000 7f99cffff640 1 -- 172.21.15.98:0/3930969934 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99c4003520 con 0x7f99d80c6d40 2023-11-28T17:49:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.956+0000 7f99cffff640 1 -- 172.21.15.98:0/3930969934 <== mon.1 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3903851559 0 0) 0x7f99c4003520 con 0x7f99d80c6d40 2023-11-28T17:49:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.956+0000 7f99cffff640 1 -- 172.21.15.98:0/3930969934 >> v2:172.21.15.98:3300/0 conn(0x7f99d8159bf0 msgr2=0x7f99d8161880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:49:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.956+0000 7f99cffff640 1 --2- 172.21.15.98:0/3930969934 >> v2:172.21.15.98:3300/0 conn(0x7f99d8159bf0 0x7f99d8161880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.956+0000 7f99cffff640 1 -- 172.21.15.98:0/3930969934 >> v1:172.21.15.98:6789/0 conn(0x7f99d8150250 legacy=0x7f99d80c7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.956+0000 7f99cffff640 1 -- 172.21.15.98:0/3930969934 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d8150630 con 0x7f99d80c6d40 2023-11-28T17:49:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.956+0000 7f99cffff640 1 -- 172.21.15.98:0/3930969934 <== mon.1 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f99c8001670 con 0x7f99d80c6d40 2023-11-28T17:49:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.956+0000 7f99cffff640 1 -- 172.21.15.98:0/3930969934 >> v1:172.21.15.113:6789/0 conn(0x7f99d80c6d40 legacy=0x7f99d81594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.956+0000 7f99cffff640 1 --2- 172.21.15.98:0/3930969934 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99c40039c0 0x7f99c4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.956+0000 7f99cffff640 1 --2- 172.21.15.98:0/3930969934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99c40047c0 0x7f99c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:41.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.956+0000 7f99cffff640 1 --2- 172.21.15.98:0/3930969934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 0x7f99c400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.956+0000 7f99cffff640 1 -- 172.21.15.98:0/3930969934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d806e910 con 0x7f99c40051d0 2023-11-28T17:49:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.956+0000 7f99cffff640 1 -- 172.21.15.98:0/3930969934 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c400c120 con 0x7f99c40047c0 2023-11-28T17:49:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.956+0000 7f99cffff640 1 -- 172.21.15.98:0/3930969934 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c400c2e0 con 0x7f99c40039c0 2023-11-28T17:49:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.956+0000 7f99dd962640 1 --2- 172.21.15.98:0/3930969934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 0x7f99c400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.957+0000 7f99dd161640 1 --2- 172.21.15.98:0/3930969934 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99c40039c0 0x7f99c4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.957+0000 7f99dc960640 1 --2- 172.21.15.98:0/3930969934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99c40047c0 0x7f99c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.957+0000 7f99dd962640 1 -- 172.21.15.98:0/3930969934 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99c40039c0 msgr2=0x7f99c4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.957+0000 7f99dd962640 1 --2- 172.21.15.98:0/3930969934 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99c40039c0 0x7f99c4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.957+0000 7f99dd962640 1 -- 172.21.15.98:0/3930969934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99c40047c0 msgr2=0x7f99c4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.957+0000 7f99dd962640 1 --2- 172.21.15.98:0/3930969934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99c40047c0 0x7f99c4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.957+0000 7f99dd962640 1 -- 172.21.15.98:0/3930969934 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c4003520 con 0x7f99c40051d0 2023-11-28T17:49:41.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.957+0000 7f99dc960640 1 --2- 172.21.15.98:0/3930969934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99c40047c0 0x7f99c4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:41.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.958+0000 7f99dd962640 1 -- 172.21.15.98:0/3930969934 reap_dead start 2023-11-28T17:49:41.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.958+0000 7f99dd962640 1 --2- 172.21.15.98:0/3930969934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 0x7f99c400b980 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009fd0 tx=0x7f99c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8850d484441f5186 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:41.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.958+0000 7f99cffff640 1 -- 172.21.15.98:0/3930969934 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d002ccb0 con 0x7f99c40051d0 2023-11-28T17:49:41.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.958+0000 7f99cffff640 1 -- 172.21.15.98:0/3930969934 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d002ce30 con 0x7f99c40051d0 2023-11-28T17:49:41.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.958+0000 7f99cffff640 1 -- 172.21.15.98:0/3930969934 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d0034680 con 0x7f99c40051d0 2023-11-28T17:49:41.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.959+0000 7f99de163640 1 -- 172.21.15.98:0/3930969934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 msgr2=0x7f99c400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:41.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.959+0000 7f99de163640 1 --2- 172.21.15.98:0/3930969934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 0x7f99c400b980 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009fd0 tx=0x7f99c40057f0 comp rx=0 tx=0).stop 2023-11-28T17:49:41.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.959+0000 7f99de163640 1 -- 172.21.15.98:0/3930969934 shutdown_connections 2023-11-28T17:49:41.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.959+0000 7f99de163640 1 --2- 172.21.15.98:0/3930969934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 0x7f99c400b980 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:41.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.959+0000 7f99de163640 1 -- 172.21.15.98:0/3930969934 >> 172.21.15.98:0/3930969934 conn(0x7f99d8066230 msgr2=0x7f99d806a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:41.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.959+0000 7f99de163640 1 -- 172.21.15.98:0/3930969934 shutdown_connections 2023-11-28T17:49:41.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.959+0000 7f99de163640 1 -- 172.21.15.98:0/3930969934 wait complete. 2023-11-28T17:49:41.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.961+0000 7f99de163640 1 Processor -- start 2023-11-28T17:49:41.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.961+0000 7f99de163640 1 -- start start 2023-11-28T17:49:41.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.962+0000 7f99de163640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 0x7f99d80c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:41.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.962+0000 7f99de163640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99d8139d30 0x7f99d80ca420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:41.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.962+0000 7f99de163640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99d80c4b20 0x7f99d80c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:41.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.962+0000 7f99de163640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d8159870 con 0x7f99c40051d0 2023-11-28T17:49:41.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.962+0000 7f99de163640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d81571e0 con 0x7f99d80c4b20 2023-11-28T17:49:41.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.962+0000 7f99de163640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d80c2de0 con 0x7f99d8139d30 2023-11-28T17:49:41.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.962+0000 7f99dd161640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 0x7f99d80c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:41.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.962+0000 7f99dd161640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 0x7f99d80c9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45940/0 (socket says 172.21.15.98:45940) 2023-11-28T17:49:41.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.962+0000 7f99dd161640 1 -- 172.21.15.98:0/3464849252 learned_addr learned my addr 172.21.15.98:0/3464849252 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:41.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.962+0000 7f99dd161640 1 -- 172.21.15.98:0/3464849252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 msgr2=0x7f99d80c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-11-28T17:49:41.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.962+0000 7f99dd161640 1 -- 172.21.15.98:0/3464849252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 msgr2=0x7f99d80c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:49:41.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.962+0000 7f99dd161640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 0x7f99d80c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:41.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.962+0000 7f99dd161640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 0x7f99d80c9ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:41.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.963+0000 7f99dc960640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99d8139d30 0x7f99d80ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:41.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.963+0000 7f99dd962640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99d80c4b20 0x7f99d80c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:41.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.963+0000 7f99dc960640 1 -- 172.21.15.98:0/3464849252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99d80c4b20 msgr2=0x7f99d80c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:41.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.963+0000 7f99dc960640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99d80c4b20 0x7f99d80c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:41.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.963+0000 7f99dc960640 1 -- 172.21.15.98:0/3464849252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 msgr2=0x7f99d80c9ee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:49:41.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.963+0000 7f99dc960640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 0x7f99d80c9ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:41.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.963+0000 7f99dc960640 1 -- 172.21.15.98:0/3464849252 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d8159590 con 0x7f99d8139d30 2023-11-28T17:49:41.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.964+0000 7f99dc960640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99d8139d30 0x7f99d80ca420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f99c800d720 tx=0x7f99c800dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:41.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.964+0000 7f99cdffb640 1 -- 172.21.15.98:0/3464849252 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c8005480 con 0x7f99d8139d30 2023-11-28T17:49:41.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.964+0000 7f99de163640 1 -- 172.21.15.98:0/3464849252 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d80c2820 con 0x7f99d8139d30 2023-11-28T17:49:41.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.964+0000 7f99cdffb640 1 -- 172.21.15.98:0/3464849252 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c800de30 con 0x7f99d8139d30 2023-11-28T17:49:41.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.964+0000 7f99cdffb640 1 -- 172.21.15.98:0/3464849252 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c8001c30 con 0x7f99d8139d30 2023-11-28T17:49:41.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.964+0000 7f99de163640 1 -- 172.21.15.98:0/3464849252 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d81591c0 con 0x7f99d8139d30 2023-11-28T17:49:41.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.965+0000 7f99de163640 1 -- 172.21.15.98:0/3464849252 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d8074db0 con 0x7f99d8139d30 2023-11-28T17:49:41.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.967+0000 7f99cdffb640 1 -- 172.21.15.98:0/3464849252 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f99c800b850 con 0x7f99d8139d30 2023-11-28T17:49:41.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.968+0000 7f99cdffb640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f99bc06b2d0 0x7f99bc06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:41.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.968+0000 7f99cdffb640 1 -- 172.21.15.98:0/3464849252 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f99c808a460 con 0x7f99d8139d30 2023-11-28T17:49:41.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.968+0000 7f99dd161640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f99bc06b2d0 0x7f99bc06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:41.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.969+0000 7f99dd161640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f99bc06b2d0 0x7f99bc06d790 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f99d4000f60 tx=0x7f99d4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:41.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:41.979+0000 7f99cdffb640 1 -- 172.21.15.98:0/3464849252 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99c8055240 con 0x7f99d8139d30 2023-11-28T17:49:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:42.239+0000 7f99de163640 1 -- 172.21.15.98:0/3464849252 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f99d8159350 con 0x7f99d8139d30 2023-11-28T17:49:42.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:42.240+0000 7f99cdffb640 1 -- 172.21.15.98:0/3464849252 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7f99d8159350 con 0x7f99d8139d30 2023-11-28T17:49:42.241 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":78,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":35,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:49:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:42.243+0000 7f99de163640 1 -- 172.21.15.98:0/3464849252 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f99bc06b2d0 msgr2=0x7f99bc06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:42.243+0000 7f99de163640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f99bc06b2d0 0x7f99bc06d790 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f99d4000f60 tx=0x7f99d4009040 comp rx=0 tx=0).stop 2023-11-28T17:49:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:42.243+0000 7f99de163640 1 -- 172.21.15.98:0/3464849252 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99d8139d30 msgr2=0x7f99d80ca420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:42.243+0000 7f99de163640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99d8139d30 0x7f99d80ca420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f99c800d720 tx=0x7f99c800dbe0 comp rx=0 tx=0).stop 2023-11-28T17:49:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:42.243+0000 7f99de163640 1 -- 172.21.15.98:0/3464849252 shutdown_connections 2023-11-28T17:49:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:42.243+0000 7f99de163640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f99bc06b2d0 0x7f99bc06d790 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:42.243+0000 7f99de163640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99d80c4b20 0x7f99d80c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:42.243+0000 7f99de163640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f99d8139d30 0x7f99d80ca420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:42.243+0000 7f99de163640 1 --2- 172.21.15.98:0/3464849252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99c40051d0 0x7f99d80c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:42.243+0000 7f99de163640 1 -- 172.21.15.98:0/3464849252 >> 172.21.15.98:0/3464849252 conn(0x7f99d8066230 msgr2=0x7f99d8071350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:42.244+0000 7f99de163640 1 -- 172.21.15.98:0/3464849252 shutdown_connections 2023-11-28T17:49:42.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:42.244+0000 7f99de163640 1 -- 172.21.15.98:0/3464849252 wait complete. 2023-11-28T17:49:45.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:49:45.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.484+0000 7f270b694640 1 Processor -- start 2023-11-28T17:49:45.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.484+0000 7f270b694640 1 -- start start 2023-11-28T17:49:45.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.486+0000 7f270b694640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2704157b60 0x7f270415bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:45.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.486+0000 7f270b694640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270415b150 con 0x7f27040c4910 2023-11-28T17:49:45.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.486+0000 7f270b694640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270406e1f0 con 0x7f2704151d80 2023-11-28T17:49:45.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.486+0000 7f270b694640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2704157f90 con 0x7f2704157b60 2023-11-28T17:49:45.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.486+0000 7f2708c2f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2704157b60 0x7f270415bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:45.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.486+0000 7f2709c31640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f27040c4910 0x7f27041624e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44508/0 (socket says 172.21.15.98:44508) 2023-11-28T17:49:45.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.486+0000 7f2709c31640 1 -- 172.21.15.98:0/1124825831 learned_addr learned my addr 172.21.15.98:0/1124825831 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:49:45.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.487+0000 7f2708c2f640 1 -- 172.21.15.98:0/1124825831 >> v2:172.21.15.98:3300/0 conn(0x7f2704157b60 msgr2=0x7f270415bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:49:45.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.487+0000 7f2708c2f640 1 -- 172.21.15.98:0/1124825831 >> v2:172.21.15.98:3300/0 conn(0x7f2704157b60 msgr2=0x7f270415bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:49:45.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.487+0000 7f2708c2f640 1 --2- 172.21.15.98:0/1124825831 >> v2:172.21.15.98:3300/0 conn(0x7f2704157b60 0x7f270415bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:45.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.487+0000 7f2708c2f640 1 --2- 172.21.15.98:0/1124825831 >> v2:172.21.15.98:3300/0 conn(0x7f2704157b60 0x7f270415bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:45.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.487+0000 7f2703fff640 1 -- 172.21.15.98:0/1124825831 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594629513 0 0) 0x7f270406e1f0 con 0x7f2704151d80 2023-11-28T17:49:45.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.487+0000 7f2703fff640 1 -- 172.21.15.98:0/1124825831 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f4003520 con 0x7f2704151d80 2023-11-28T17:49:45.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.488+0000 7f2703fff640 1 -- 172.21.15.98:0/1124825831 <== mon.1 v1:172.21.15.113:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3622763838 0 0) 0x7f26f4003520 con 0x7f2704151d80 2023-11-28T17:49:45.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.488+0000 7f2703fff640 1 -- 172.21.15.98:0/1124825831 >> v2:172.21.15.98:3300/0 conn(0x7f2704157b60 msgr2=0x7f270415bc10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:49:45.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.488+0000 7f2703fff640 1 --2- 172.21.15.98:0/1124825831 >> v2:172.21.15.98:3300/0 conn(0x7f2704157b60 0x7f270415bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:45.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.488+0000 7f2703fff640 1 -- 172.21.15.98:0/1124825831 >> v1:172.21.15.98:6789/0 conn(0x7f27040c4910 legacy=0x7f27041624e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:45.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.488+0000 7f2703fff640 1 -- 172.21.15.98:0/1124825831 --> v1:172.21.15.113:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270406e4d0 con 0x7f2704151d80 2023-11-28T17:49:45.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.488+0000 7f2703fff640 1 -- 172.21.15.98:0/1124825831 <== mon.1 v1:172.21.15.113:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f26f80019f0 con 0x7f2704151d80 2023-11-28T17:49:45.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.488+0000 7f2703fff640 1 -- 172.21.15.98:0/1124825831 >> v1:172.21.15.113:6789/0 conn(0x7f2704151d80 legacy=0x7f27040c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:45.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.488+0000 7f2703fff640 1 --2- 172.21.15.98:0/1124825831 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f40039c0 0x7f26f4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:45.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.488+0000 7f2703fff640 1 --2- 172.21.15.98:0/1124825831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26f40047c0 0x7f26f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:45.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.488+0000 7f2703fff640 1 --2- 172.21.15.98:0/1124825831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f40051d0 0x7f26f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:45.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.488+0000 7f2703fff640 1 -- 172.21.15.98:0/1124825831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f400c0e0 con 0x7f26f40051d0 2023-11-28T17:49:45.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.488+0000 7f2703fff640 1 -- 172.21.15.98:0/1124825831 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f400c250 con 0x7f26f40047c0 2023-11-28T17:49:45.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.488+0000 7f2703fff640 1 -- 172.21.15.98:0/1124825831 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f400c410 con 0x7f26f40039c0 2023-11-28T17:49:45.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.489+0000 7f2709c31640 1 --2- 172.21.15.98:0/1124825831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f40051d0 0x7f26f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:45.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.489+0000 7f2709430640 1 --2- 172.21.15.98:0/1124825831 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f40039c0 0x7f26f4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:45.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.489+0000 7f2708c2f640 1 --2- 172.21.15.98:0/1124825831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26f40047c0 0x7f26f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:45.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.489+0000 7f2709c31640 1 -- 172.21.15.98:0/1124825831 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f40039c0 msgr2=0x7f26f4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:45.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.489+0000 7f2709c31640 1 --2- 172.21.15.98:0/1124825831 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f40039c0 0x7f26f4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:45.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.489+0000 7f2709c31640 1 -- 172.21.15.98:0/1124825831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26f40047c0 msgr2=0x7f26f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:45.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.489+0000 7f2709c31640 1 --2- 172.21.15.98:0/1124825831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26f40047c0 0x7f26f4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:45.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.489+0000 7f2709c31640 1 -- 172.21.15.98:0/1124825831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f4003520 con 0x7f26f40051d0 2023-11-28T17:49:45.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.489+0000 7f2708c2f640 1 --2- 172.21.15.98:0/1124825831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f26f40047c0 0x7f26f4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:45.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.490+0000 7f2709c31640 1 -- 172.21.15.98:0/1124825831 reap_dead start 2023-11-28T17:49:45.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.491+0000 7f2709c31640 1 --2- 172.21.15.98:0/1124825831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f40051d0 0x7f26f400b980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f26fc001810 tx=0x7f26f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee29efa04dea2d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:45.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.491+0000 7f2703fff640 1 -- 172.21.15.98:0/1124825831 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc02f070 con 0x7f26f40051d0 2023-11-28T17:49:45.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.491+0000 7f2703fff640 1 -- 172.21.15.98:0/1124825831 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26fc007070 con 0x7f26f40051d0 2023-11-28T17:49:45.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.491+0000 7f2703fff640 1 -- 172.21.15.98:0/1124825831 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26fc007400 con 0x7f26f40051d0 2023-11-28T17:49:45.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.492+0000 7f270b694640 1 -- 172.21.15.98:0/1124825831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f40051d0 msgr2=0x7f26f400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:45.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.492+0000 7f270b694640 1 --2- 172.21.15.98:0/1124825831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f40051d0 0x7f26f400b980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f26fc001810 tx=0x7f26f40057f0 comp rx=0 tx=0).stop 2023-11-28T17:49:45.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.492+0000 7f270b694640 1 -- 172.21.15.98:0/1124825831 shutdown_connections 2023-11-28T17:49:45.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.492+0000 7f270b694640 1 --2- 172.21.15.98:0/1124825831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26f40051d0 0x7f26f400b980 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:45.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.492+0000 7f270b694640 1 -- 172.21.15.98:0/1124825831 >> 172.21.15.98:0/1124825831 conn(0x7f270406a4d0 msgr2=0x7f270406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:45.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.492+0000 7f270b694640 1 -- 172.21.15.98:0/1124825831 shutdown_connections 2023-11-28T17:49:45.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.492+0000 7f270b694640 1 -- 172.21.15.98:0/1124825831 wait complete. 2023-11-28T17:49:45.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.494+0000 7f270b694640 1 Processor -- start 2023-11-28T17:49:45.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.494+0000 7f270b694640 1 -- start start 2023-11-28T17:49:45.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.495+0000 7f270b694640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f40051d0 0x7f27040c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:45.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.495+0000 7f270b694640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27041e0890 0x7f27041efa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:45.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.495+0000 7f270b694640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27041e0cd0 0x7f27041499b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:45.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.495+0000 7f270b694640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27040c44c0 con 0x7f27041e0cd0 2023-11-28T17:49:45.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.495+0000 7f270b694640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27040c17b0 con 0x7f27041e0890 2023-11-28T17:49:45.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.495+0000 7f270b694640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270414b2c0 con 0x7f26f40051d0 2023-11-28T17:49:45.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.495+0000 7f2709c31640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27041e0cd0 0x7f27041499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:45.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.495+0000 7f2709c31640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27041e0cd0 0x7f27041499b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45964/0 (socket says 172.21.15.98:45964) 2023-11-28T17:49:45.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.495+0000 7f2709c31640 1 -- 172.21.15.98:0/1863106321 learned_addr learned my addr 172.21.15.98:0/1863106321 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:45.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.495+0000 7f2709c31640 1 -- 172.21.15.98:0/1863106321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27041e0cd0 msgr2=0x7f27041499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-11-28T17:49:45.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.495+0000 7f2709c31640 1 -- 172.21.15.98:0/1863106321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27041e0cd0 msgr2=0x7f27041499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:49:45.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.495+0000 7f2709c31640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27041e0cd0 0x7f27041499b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:45.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.495+0000 7f2709c31640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27041e0cd0 0x7f27041499b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:45.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.495+0000 7f2709430640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f40051d0 0x7f27040c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:45.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.495+0000 7f2708c2f640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27041e0890 0x7f27041efa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:45.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.496+0000 7f2709430640 1 -- 172.21.15.98:0/1863106321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27041e0890 msgr2=0x7f27041efa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:45.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.496+0000 7f2709430640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27041e0890 0x7f27041efa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:45.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.496+0000 7f2709430640 1 -- 172.21.15.98:0/1863106321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27041e0cd0 msgr2=0x7f27041499b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:49:45.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.496+0000 7f2709430640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27041e0cd0 0x7f27041499b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.496+0000 7f2709430640 1 -- 172.21.15.98:0/1863106321 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270414b5a0 con 0x7f26f40051d0 2023-11-28T17:49:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.496+0000 7f2709430640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f40051d0 0x7f27040c5b60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f26f80013d0 tx=0x7f26f800ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.497+0000 7f2701ffb640 1 -- 172.21.15.98:0/1863106321 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f8017070 con 0x7f26f40051d0 2023-11-28T17:49:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.497+0000 7f2701ffb640 1 -- 172.21.15.98:0/1863106321 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f8003b80 con 0x7f26f40051d0 2023-11-28T17:49:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.497+0000 7f2701ffb640 1 -- 172.21.15.98:0/1863106321 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f8003d10 con 0x7f26f40051d0 2023-11-28T17:49:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.497+0000 7f270b694640 1 -- 172.21.15.98:0/1863106321 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270414b880 con 0x7f26f40051d0 2023-11-28T17:49:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.497+0000 7f270b694640 1 -- 172.21.15.98:0/1863106321 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27040c3cc0 con 0x7f26f40051d0 2023-11-28T17:49:45.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.498+0000 7f270b694640 1 -- 172.21.15.98:0/1863106321 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2704067bf0 con 0x7f26f40051d0 2023-11-28T17:49:45.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.500+0000 7f2701ffb640 1 -- 172.21.15.98:0/1863106321 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f26f8006b50 con 0x7f26f40051d0 2023-11-28T17:49:45.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.501+0000 7f2701ffb640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f26e806b2a0 0x7f26e806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:45.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.501+0000 7f2701ffb640 1 -- 172.21.15.98:0/1863106321 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f26f808a0b0 con 0x7f26f40051d0 2023-11-28T17:49:45.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.501+0000 7f2708c2f640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f26e806b2a0 0x7f26e806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:45.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.502+0000 7f2708c2f640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f26e806b2a0 0x7f26e806d760 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2704072a40 tx=0x7f26f0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:45.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.512+0000 7f2701ffb640 1 -- 172.21.15.98:0/1863106321 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f26f8054e00 con 0x7f26f40051d0 2023-11-28T17:49:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.775+0000 7f270b694640 1 -- 172.21.15.98:0/1863106321 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f27040c6820 con 0x7f26f40051d0 2023-11-28T17:49:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.777+0000 7f2701ffb640 1 -- 172.21.15.98:0/1863106321 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7f26f8058420 con 0x7f26f40051d0 2023-11-28T17:49:45.778 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":78,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":39,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:49:45.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.779+0000 7f270b694640 1 -- 172.21.15.98:0/1863106321 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f26e806b2a0 msgr2=0x7f26e806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:45.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.779+0000 7f270b694640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f26e806b2a0 0x7f26e806d760 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2704072a40 tx=0x7f26f0008040 comp rx=0 tx=0).stop 2023-11-28T17:49:45.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.779+0000 7f270b694640 1 -- 172.21.15.98:0/1863106321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f40051d0 msgr2=0x7f27040c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:45.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.779+0000 7f270b694640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f40051d0 0x7f27040c5b60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f26f80013d0 tx=0x7f26f800ef80 comp rx=0 tx=0).stop 2023-11-28T17:49:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.780+0000 7f270b694640 1 -- 172.21.15.98:0/1863106321 shutdown_connections 2023-11-28T17:49:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.780+0000 7f270b694640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27041e0cd0 0x7f27041499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.780+0000 7f270b694640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f26e806b2a0 0x7f26e806d760 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.780+0000 7f270b694640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27041e0890 0x7f27041efa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.780+0000 7f270b694640 1 --2- 172.21.15.98:0/1863106321 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f26f40051d0 0x7f27040c5b60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.780+0000 7f270b694640 1 -- 172.21.15.98:0/1863106321 >> 172.21.15.98:0/1863106321 conn(0x7f270406a4d0 msgr2=0x7f27041e3250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.780+0000 7f270b694640 1 -- 172.21.15.98:0/1863106321 shutdown_connections 2023-11-28T17:49:45.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:45.780+0000 7f270b694640 1 -- 172.21.15.98:0/1863106321 wait complete. 2023-11-28T17:49:48.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~297s 2023-11-28T17:49:48.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:49:49.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.016+0000 7fb6d711d640 1 Processor -- start 2023-11-28T17:49:49.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.016+0000 7fb6d711d640 1 -- start start 2023-11-28T17:49:49.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.018+0000 7fb6d711d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb6d0157b60 0x7fb6d015bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:49.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.018+0000 7fb6d711d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fb6d00c4910 0x7fb6d00c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:49.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.018+0000 7fb6d711d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d015b150 con 0x7fb6d0151d80 2023-11-28T17:49:49.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.018+0000 7fb6d711d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6d00c5260 con 0x7fb6d0157b60 2023-11-28T17:49:49.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.018+0000 7fb6d711d640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6d0157f90 con 0x7fb6d00c4910 2023-11-28T17:49:49.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.018+0000 7fb6c7fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb6d0157b60 0x7fb6d015bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:49.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.018+0000 7fb6c7fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb6d0157b60 0x7fb6d015bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45980/0 (socket says 172.21.15.98:45980) 2023-11-28T17:49:49.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.018+0000 7fb6c7fff640 1 -- 172.21.15.98:0/2110520197 learned_addr learned my addr 172.21.15.98:0/2110520197 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:49.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.018+0000 7fb6d4eb9640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb6d0151d80 0x7fb6d00c5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44518/0 (socket says 172.21.15.98:44518) 2023-11-28T17:49:49.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.018+0000 7fb6c7fff640 1 -- 172.21.15.98:0/2110520197 >> v2:172.21.15.98:3300/0 conn(0x7fb6d0157b60 msgr2=0x7fb6d015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:49:49.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.018+0000 7fb6c7fff640 1 -- 172.21.15.98:0/2110520197 >> v2:172.21.15.98:3300/0 conn(0x7fb6d0157b60 msgr2=0x7fb6d015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:49:49.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.018+0000 7fb6c7fff640 1 --2- 172.21.15.98:0/2110520197 >> v2:172.21.15.98:3300/0 conn(0x7fb6d0157b60 0x7fb6d015bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:49.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.018+0000 7fb6c7fff640 1 --2- 172.21.15.98:0/2110520197 >> v2:172.21.15.98:3300/0 conn(0x7fb6d0157b60 0x7fb6d015bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:49.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.018+0000 7fb6d56ba640 1 --2- 172.21.15.98:0/2110520197 >> v2:172.21.15.113:3300/0 conn(0x7fb6d00c4910 0x7fb6d00c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:49.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.019+0000 7fb6d56ba640 1 -- 172.21.15.98:0/2110520197 >> v2:172.21.15.98:3300/0 conn(0x7fb6d0157b60 msgr2=0x7fb6d015bc10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:49:49.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.019+0000 7fb6d56ba640 1 --2- 172.21.15.98:0/2110520197 >> v2:172.21.15.98:3300/0 conn(0x7fb6d0157b60 0x7fb6d015bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:49.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.019+0000 7fb6d56ba640 1 -- 172.21.15.98:0/2110520197 >> v1:172.21.15.98:6789/0 conn(0x7fb6d0151d80 legacy=0x7fb6d00c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:49.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.019+0000 7fb6d56ba640 1 -- 172.21.15.98:0/2110520197 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d006e1f0 con 0x7fb6d00c4910 2023-11-28T17:49:49.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.021+0000 7fb6d56ba640 1 --2- 172.21.15.98:0/2110520197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d00c4910 0x7fb6d00c4d20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb6d00ca2d0 tx=0x7fb6c802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64e727d0f4332529 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:49.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.021+0000 7fb6c77fe640 1 -- 172.21.15.98:0/2110520197 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8030610 con 0x7fb6d00c4910 2023-11-28T17:49:49.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.021+0000 7fb6c77fe640 1 -- 172.21.15.98:0/2110520197 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c8030770 con 0x7fb6d00c4910 2023-11-28T17:49:49.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.021+0000 7fb6c77fe640 1 -- 172.21.15.98:0/2110520197 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8030b00 con 0x7fb6d00c4910 2023-11-28T17:49:49.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.022+0000 7fb6d711d640 1 -- 172.21.15.98:0/2110520197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d00c4910 msgr2=0x7fb6d00c4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:49.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.022+0000 7fb6d711d640 1 --2- 172.21.15.98:0/2110520197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d00c4910 0x7fb6d00c4d20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb6d00ca2d0 tx=0x7fb6c802b960 comp rx=0 tx=0).stop 2023-11-28T17:49:49.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.022+0000 7fb6d711d640 1 -- 172.21.15.98:0/2110520197 shutdown_connections 2023-11-28T17:49:49.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.022+0000 7fb6d711d640 1 --2- 172.21.15.98:0/2110520197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d00c4910 0x7fb6d00c4d20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:49.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.022+0000 7fb6d711d640 1 --2- 172.21.15.98:0/2110520197 >> v2:172.21.15.98:3300/0 conn(0x7fb6d0157b60 0x7fb6d015bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:49.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.022+0000 7fb6d711d640 1 -- 172.21.15.98:0/2110520197 >> 172.21.15.98:0/2110520197 conn(0x7fb6d006a4d0 msgr2=0x7fb6d006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:49.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.022+0000 7fb6d711d640 1 -- 172.21.15.98:0/2110520197 shutdown_connections 2023-11-28T17:49:49.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.022+0000 7fb6d711d640 1 -- 172.21.15.98:0/2110520197 wait complete. 2023-11-28T17:49:49.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.024+0000 7fb6d711d640 1 Processor -- start 2023-11-28T17:49:49.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.024+0000 7fb6d711d640 1 -- start start 2023-11-28T17:49:49.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.025+0000 7fb6d711d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d00c4910 0x7fb6d01f0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:49.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.025+0000 7fb6d711d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6d0151d80 0x7fb6d014fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:49.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.025+0000 7fb6d711d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6d0157b60 0x7fb6d014b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:49.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.025+0000 7fb6d711d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d0145720 con 0x7fb6d0151d80 2023-11-28T17:49:49.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.025+0000 7fb6d711d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d01ee0d0 con 0x7fb6d00c4910 2023-11-28T17:49:49.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.025+0000 7fb6d711d640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d01ed7c0 con 0x7fb6d0157b60 2023-11-28T17:49:49.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.025+0000 7fb6c7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6d0151d80 0x7fb6d014fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:49.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.025+0000 7fb6c7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6d0151d80 0x7fb6d014fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45996/0 (socket says 172.21.15.98:45996) 2023-11-28T17:49:49.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.025+0000 7fb6c7fff640 1 -- 172.21.15.98:0/2547976534 learned_addr learned my addr 172.21.15.98:0/2547976534 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:49.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.025+0000 7fb6c7fff640 1 -- 172.21.15.98:0/2547976534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6d0151d80 msgr2=0x7fb6d014fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-11-28T17:49:49.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.025+0000 7fb6c7fff640 1 -- 172.21.15.98:0/2547976534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6d0151d80 msgr2=0x7fb6d014fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:49:49.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.025+0000 7fb6c7fff640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6d0151d80 0x7fb6d014fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:49.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.025+0000 7fb6c7fff640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6d0151d80 0x7fb6d014fda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:49.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.026+0000 7fb6d56ba640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6d0157b60 0x7fb6d014b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.026+0000 7fb6d4eb9640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d00c4910 0x7fb6d01f0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.026+0000 7fb6d56ba640 1 -- 172.21.15.98:0/2547976534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d00c4910 msgr2=0x7fb6d01f0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.026+0000 7fb6d56ba640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d00c4910 0x7fb6d01f0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.026+0000 7fb6d56ba640 1 -- 172.21.15.98:0/2547976534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6d0151d80 msgr2=0x7fb6d014fda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:49:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.026+0000 7fb6d56ba640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6d0151d80 0x7fb6d014fda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:49.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.026+0000 7fb6d56ba640 1 -- 172.21.15.98:0/2547976534 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c8009400 con 0x7fb6d0157b60 2023-11-28T17:49:49.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.027+0000 7fb6d56ba640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6d0157b60 0x7fb6d014b7e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb6d00c6f70 tx=0x7fb6c80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.029+0000 7fb6c57fa640 1 -- 172.21.15.98:0/2547976534 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c803e070 con 0x7fb6d0157b60 2023-11-28T17:49:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.030+0000 7fb6d711d640 1 -- 172.21.15.98:0/2547976534 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d0156300 con 0x7fb6d0157b60 2023-11-28T17:49:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.030+0000 7fb6c57fa640 1 -- 172.21.15.98:0/2547976534 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c8039910 con 0x7fb6d0157b60 2023-11-28T17:49:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.030+0000 7fb6c57fa640 1 -- 172.21.15.98:0/2547976534 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8039ca0 con 0x7fb6d0157b60 2023-11-28T17:49:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.030+0000 7fb6d711d640 1 -- 172.21.15.98:0/2547976534 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d01eda80 con 0x7fb6d0157b60 2023-11-28T17:49:49.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.031+0000 7fb6d711d640 1 -- 172.21.15.98:0/2547976534 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d0067bf0 con 0x7fb6d0157b60 2023-11-28T17:49:49.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.035+0000 7fb6c57fa640 1 -- 172.21.15.98:0/2547976534 <== mon.2 v2:172.21.15.113:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb6c804c450 con 0x7fb6d0157b60 2023-11-28T17:49:49.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.036+0000 7fb6c57fa640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb6bc06b2a0 0x7fb6bc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:49.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.036+0000 7fb6c57fa640 1 -- 172.21.15.98:0/2547976534 <== mon.2 v2:172.21.15.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb6c80abca0 con 0x7fb6d0157b60 2023-11-28T17:49:49.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.036+0000 7fb6d4eb9640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb6bc06b2a0 0x7fb6bc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:49.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.037+0000 7fb6d4eb9640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb6bc06b2a0 0x7fb6bc06d760 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc007fd0 tx=0x7fb6cc0016d0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:49.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.046+0000 7fb6c57fa640 1 -- 172.21.15.98:0/2547976534 <== mon.2 v2:172.21.15.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6c8076960 con 0x7fb6d0157b60 2023-11-28T17:49:49.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.319+0000 7fb6d711d640 1 -- 172.21.15.98:0/2547976534 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb6d01ee240 con 0x7fb6d0157b60 2023-11-28T17:49:49.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.320+0000 7fb6c57fa640 1 -- 172.21.15.98:0/2547976534 <== mon.2 v2:172.21.15.113:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7fb6c8034080 con 0x7fb6d0157b60 2023-11-28T17:49:49.321 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":78,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":42,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:49:49.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.323+0000 7fb6d711d640 1 -- 172.21.15.98:0/2547976534 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb6bc06b2a0 msgr2=0x7fb6bc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.323+0000 7fb6d711d640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb6bc06b2a0 0x7fb6bc06d760 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc007fd0 tx=0x7fb6cc0016d0 comp rx=0 tx=0).stop 2023-11-28T17:49:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.323+0000 7fb6d711d640 1 -- 172.21.15.98:0/2547976534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6d0157b60 msgr2=0x7fb6d014b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.323+0000 7fb6d711d640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6d0157b60 0x7fb6d014b7e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb6d00c6f70 tx=0x7fb6c80046c0 comp rx=0 tx=0).stop 2023-11-28T17:49:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.323+0000 7fb6d711d640 1 -- 172.21.15.98:0/2547976534 shutdown_connections 2023-11-28T17:49:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.323+0000 7fb6d711d640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fb6d0157b60 0x7fb6d014b7e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.323+0000 7fb6d711d640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6d0151d80 0x7fb6d014fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.323+0000 7fb6d711d640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fb6bc06b2a0 0x7fb6bc06d760 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.323+0000 7fb6d711d640 1 --2- 172.21.15.98:0/2547976534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb6d00c4910 0x7fb6d01f0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.323+0000 7fb6d711d640 1 -- 172.21.15.98:0/2547976534 >> 172.21.15.98:0/2547976534 conn(0x7fb6d006a4d0 msgr2=0x7fb6d00c1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.323+0000 7fb6d711d640 1 -- 172.21.15.98:0/2547976534 shutdown_connections 2023-11-28T17:49:49.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:49.323+0000 7fb6d711d640 1 -- 172.21.15.98:0/2547976534 wait complete. 2023-11-28T17:49:52.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:49:52.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.565+0000 7f34dbfff640 1 Processor -- start 2023-11-28T17:49:52.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.566+0000 7f34dbfff640 1 -- start start 2023-11-28T17:49:52.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.567+0000 7f34dbfff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34dc153bb0 0x7f34dc15a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:52.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.567+0000 7f34dbfff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f34dc0ca390 0x7f34dc0ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:52.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.567+0000 7f34dbfff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34dc15d330 con 0x7f34dc0cf280 2023-11-28T17:49:52.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.567+0000 7f34dbfff640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34dc15d120 con 0x7f34dc0ca390 2023-11-28T17:49:52.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.567+0000 7f34dbfff640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc15d610 con 0x7f34dc153bb0 2023-11-28T17:49:52.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.568+0000 7f34db7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f34dc0cf280 0x7f34dc0b3760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33944/0 (socket says 172.21.15.98:33944) 2023-11-28T17:49:52.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.568+0000 7f34db7fe640 1 -- 172.21.15.98:0/3953214738 learned_addr learned my addr 172.21.15.98:0/3953214738 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:49:52.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.568+0000 7f34daffd640 1 --2- 172.21.15.98:0/3953214738 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34dc153bb0 0x7f34dc15a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:52.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.568+0000 7f34da7fc640 1 --2- 172.21.15.98:0/3953214738 >> v2:172.21.15.113:3300/0 conn(0x7f34dc0ca390 0x7f34dc0ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:52.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.569+0000 7f34daffd640 1 -- 172.21.15.98:0/3953214738 >> v2:172.21.15.113:3300/0 conn(0x7f34dc0ca390 msgr2=0x7f34dc0ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:52.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.569+0000 7f34daffd640 1 --2- 172.21.15.98:0/3953214738 >> v2:172.21.15.113:3300/0 conn(0x7f34dc0ca390 0x7f34dc0ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:52.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.569+0000 7f34daffd640 1 -- 172.21.15.98:0/3953214738 >> v1:172.21.15.98:6789/0 conn(0x7f34dc0cf280 legacy=0x7f34dc0b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:52.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.569+0000 7f34daffd640 1 -- 172.21.15.98:0/3953214738 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34dc15d8f0 con 0x7f34dc153bb0 2023-11-28T17:49:52.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.569+0000 7f34da7fc640 1 --2- 172.21.15.98:0/3953214738 >> v2:172.21.15.113:3300/0 conn(0x7f34dc0ca390 0x7f34dc0ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:49:52.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.570+0000 7f34daffd640 1 --2- 172.21.15.98:0/3953214738 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34dc153bb0 0x7f34dc15a6d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f34cc009d90 tx=0x7f34cc02bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b65b0d424e1da4f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:52.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.571+0000 7f34d9ffb640 1 -- 172.21.15.98:0/3953214738 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc02cce0 con 0x7f34dc153bb0 2023-11-28T17:49:52.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.571+0000 7f34d9ffb640 1 -- 172.21.15.98:0/3953214738 <== mon.2 v2:172.21.15.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34cc02ce40 con 0x7f34dc153bb0 2023-11-28T17:49:52.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.571+0000 7f34d9ffb640 1 -- 172.21.15.98:0/3953214738 <== mon.2 v2:172.21.15.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc034660 con 0x7f34dc153bb0 2023-11-28T17:49:52.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.571+0000 7f34dbfff640 1 -- 172.21.15.98:0/3953214738 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34dc153bb0 msgr2=0x7f34dc15a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:52.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.571+0000 7f34dbfff640 1 --2- 172.21.15.98:0/3953214738 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34dc153bb0 0x7f34dc15a6d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f34cc009d90 tx=0x7f34cc02bf30 comp rx=0 tx=0).stop 2023-11-28T17:49:52.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.571+0000 7f34dbfff640 1 -- 172.21.15.98:0/3953214738 shutdown_connections 2023-11-28T17:49:52.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.571+0000 7f34dbfff640 1 --2- 172.21.15.98:0/3953214738 >> v2:172.21.15.113:3300/0 conn(0x7f34dc0ca390 0x7f34dc0ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:52.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.571+0000 7f34dbfff640 1 --2- 172.21.15.98:0/3953214738 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34dc153bb0 0x7f34dc15a6d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:52.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.571+0000 7f34dbfff640 1 -- 172.21.15.98:0/3953214738 >> 172.21.15.98:0/3953214738 conn(0x7f34dc071c00 msgr2=0x7f34dc072010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:52.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.572+0000 7f34dbfff640 1 -- 172.21.15.98:0/3953214738 shutdown_connections 2023-11-28T17:49:52.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.572+0000 7f34dbfff640 1 -- 172.21.15.98:0/3953214738 wait complete. 2023-11-28T17:49:52.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.573+0000 7f34dbfff640 1 Processor -- start 2023-11-28T17:49:52.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.573+0000 7f34dbfff640 1 -- start start 2023-11-28T17:49:52.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.574+0000 7f34dbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34dc0ca390 0x7f34dc1521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:52.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.574+0000 7f34dbfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34dc0cf280 0x7f34dc1526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:52.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.574+0000 7f34dbfff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34dc153bb0 0x7f34dc155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:52.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.574+0000 7f34dbfff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc157700 con 0x7f34dc0ca390 2023-11-28T17:49:52.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.574+0000 7f34dbfff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc1f0d60 con 0x7f34dc0cf280 2023-11-28T17:49:52.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.574+0000 7f34dbfff640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc1efe90 con 0x7f34dc153bb0 2023-11-28T17:49:52.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.575+0000 7f34daffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34dc0ca390 0x7f34dc1521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:52.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.575+0000 7f34daffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34dc0ca390 0x7f34dc1521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59568/0 (socket says 172.21.15.98:59568) 2023-11-28T17:49:52.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.575+0000 7f34daffd640 1 -- 172.21.15.98:0/1027708444 learned_addr learned my addr 172.21.15.98:0/1027708444 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:52.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.575+0000 7f34daffd640 1 -- 172.21.15.98:0/1027708444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34dc0ca390 msgr2=0x7f34dc1521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-11-28T17:49:52.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.575+0000 7f34daffd640 1 -- 172.21.15.98:0/1027708444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34dc0ca390 msgr2=0x7f34dc1521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:49:52.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.575+0000 7f34daffd640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34dc0ca390 0x7f34dc1521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:49:52.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.575+0000 7f34daffd640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34dc0ca390 0x7f34dc1521a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:49:52.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.575+0000 7f34db7fe640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34dc153bb0 0x7f34dc155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:52.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.575+0000 7f34da7fc640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34dc0cf280 0x7f34dc1526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:52.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.576+0000 7f34da7fc640 1 -- 172.21.15.98:0/1027708444 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34dc153bb0 msgr2=0x7f34dc155580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:52.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.576+0000 7f34da7fc640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34dc153bb0 0x7f34dc155580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:52.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.576+0000 7f34da7fc640 1 -- 172.21.15.98:0/1027708444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34dc0ca390 msgr2=0x7f34dc1521a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:49:52.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.576+0000 7f34da7fc640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34dc0ca390 0x7f34dc1521a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:52.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.576+0000 7f34da7fc640 1 -- 172.21.15.98:0/1027708444 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c4009400 con 0x7f34dc0cf280 2023-11-28T17:49:52.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.576+0000 7f34da7fc640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34dc0cf280 0x7f34dc1526e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f34c400e930 tx=0x7f34c400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:52.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.576+0000 7f34d37fe640 1 -- 172.21.15.98:0/1027708444 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c400ca00 con 0x7f34dc0cf280 2023-11-28T17:49:52.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.576+0000 7f34d37fe640 1 -- 172.21.15.98:0/1027708444 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c400cb60 con 0x7f34dc0cf280 2023-11-28T17:49:52.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.576+0000 7f34d37fe640 1 -- 172.21.15.98:0/1027708444 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c4013470 con 0x7f34dc0cf280 2023-11-28T17:49:52.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.577+0000 7f34dbfff640 1 -- 172.21.15.98:0/1027708444 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34cc009a70 con 0x7f34dc0cf280 2023-11-28T17:49:52.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.577+0000 7f34dbfff640 1 -- 172.21.15.98:0/1027708444 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34dc1f00e0 con 0x7f34dc0cf280 2023-11-28T17:49:52.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.578+0000 7f34dbfff640 1 -- 172.21.15.98:0/1027708444 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34dc0ca100 con 0x7f34dc0cf280 2023-11-28T17:49:52.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.579+0000 7f34d37fe640 1 -- 172.21.15.98:0/1027708444 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f34c40135d0 con 0x7f34dc0cf280 2023-11-28T17:49:52.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.580+0000 7f34d37fe640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f34bc06b2d0 0x7f34bc06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:52.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.580+0000 7f34d37fe640 1 -- 172.21.15.98:0/1027708444 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f34c408a4f0 con 0x7f34dc0cf280 2023-11-28T17:49:52.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.580+0000 7f34daffd640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f34bc06b2d0 0x7f34bc06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:52.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.581+0000 7f34daffd640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f34bc06b2d0 0x7f34bc06d790 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f34cc02bf70 tx=0x7f34cc030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:52.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.592+0000 7f34d37fe640 1 -- 172.21.15.98:0/1027708444 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34c4055470 con 0x7f34dc0cf280 2023-11-28T17:49:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.856+0000 7f34dbfff640 1 -- 172.21.15.98:0/1027708444 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f34dc074db0 con 0x7f34dc0cf280 2023-11-28T17:49:52.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.858+0000 7f34d37fe640 1 -- 172.21.15.98:0/1027708444 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7f34c4017620 con 0x7f34dc0cf280 2023-11-28T17:49:52.858 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":78,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":46,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:49:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.861+0000 7f34dbfff640 1 -- 172.21.15.98:0/1027708444 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f34bc06b2d0 msgr2=0x7f34bc06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.861+0000 7f34dbfff640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f34bc06b2d0 0x7f34bc06d790 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f34cc02bf70 tx=0x7f34cc030000 comp rx=0 tx=0).stop 2023-11-28T17:49:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.861+0000 7f34dbfff640 1 -- 172.21.15.98:0/1027708444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34dc0cf280 msgr2=0x7f34dc1526e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.861+0000 7f34dbfff640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34dc0cf280 0x7f34dc1526e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f34c400e930 tx=0x7f34c400edf0 comp rx=0 tx=0).stop 2023-11-28T17:49:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.861+0000 7f34dbfff640 1 -- 172.21.15.98:0/1027708444 shutdown_connections 2023-11-28T17:49:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.861+0000 7f34dbfff640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f34dc153bb0 0x7f34dc155580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.861+0000 7f34dbfff640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f34bc06b2d0 0x7f34bc06d790 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.861+0000 7f34dbfff640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34dc0cf280 0x7f34dc1526e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.861+0000 7f34dbfff640 1 --2- 172.21.15.98:0/1027708444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34dc0ca390 0x7f34dc1521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:52.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.861+0000 7f34dbfff640 1 -- 172.21.15.98:0/1027708444 >> 172.21.15.98:0/1027708444 conn(0x7f34dc071c00 msgr2=0x7f34dc0b4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:52.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.861+0000 7f34dbfff640 1 -- 172.21.15.98:0/1027708444 shutdown_connections 2023-11-28T17:49:52.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:52.861+0000 7f34dbfff640 1 -- 172.21.15.98:0/1027708444 wait complete. 2023-11-28T17:49:55.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~304s 2023-11-28T17:49:55.529 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-11-28T17:49:55.529 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:49:55.529 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:49:55.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:49:55.599 INFO:tasks.workunit.client.0.smithi098.stderr:fileops.c:546:1: fatal error: opening dependency file .deps/fileops.Tpo: Software caused connection abort 2023-11-28T17:49:55.599 INFO:tasks.workunit.client.0.smithi098.stderr: 546 | } 2023-11-28T17:49:55.599 INFO:tasks.workunit.client.0.smithi098.stderr: | ^ 2023-11-28T17:49:55.599 INFO:tasks.workunit.client.0.smithi098.stderr:compilation terminated. 2023-11-28T17:49:55.605 INFO:tasks.workunit.client.0.smithi098.stderr:make[1]: *** [Makefile:324: fileops.o] Error 1 2023-11-28T17:49:55.606 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2023-11-28T17:49:55.608 INFO:tasks.workunit.client.0.smithi098.stderr:make: *** [Makefile:217: all] Error 2 2023-11-28T17:49:55.608 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-28T17:49:55.608 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2023-11-28T17:49:55.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-11-28T17:49:55.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.785+0000 7f1feca00640 1 Processor -- start 2023-11-28T17:49:55.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.785+0000 7f1feca00640 1 -- start start 2023-11-28T17:49:55.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.787+0000 7f1feca00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1fe8157b60 0x7f1fe815bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:55.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.787+0000 7f1feca00640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1fe80c4910 0x7f1fe80c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:55.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.787+0000 7f1feca00640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe815b150 con 0x7f1fe8151d80 2023-11-28T17:49:55.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.787+0000 7f1feca00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe8157f90 con 0x7f1fe8157b60 2023-11-28T17:49:55.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.787+0000 7f1feca00640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe815c150 con 0x7f1fe80c4910 2023-11-28T17:49:55.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.787+0000 7f1fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1fe8157b60 0x7f1fe815bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:55.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.787+0000 7f1fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1fe8157b60 0x7f1fe815bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59584/0 (socket says 172.21.15.98:59584) 2023-11-28T17:49:55.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.787+0000 7f1fdffff640 1 -- 172.21.15.98:0/2232531371 learned_addr learned my addr 172.21.15.98:0/2232531371 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:55.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.787+0000 7f1fe6d9d640 1 --2- 172.21.15.98:0/2232531371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1fe80c4910 0x7f1fe80c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:55.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.788+0000 7f1fdffff640 1 -- 172.21.15.98:0/2232531371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1fe80c4910 msgr2=0x7f1fe80c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:55.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.788+0000 7f1fdffff640 1 --2- 172.21.15.98:0/2232531371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1fe80c4910 0x7f1fe80c4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:55.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.788+0000 7f1fdffff640 1 -- 172.21.15.98:0/2232531371 >> v1:172.21.15.113:6789/0 conn(0x7f1fe8151d80 legacy=0x7f1fe80c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:55.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.788+0000 7f1fdffff640 1 -- 172.21.15.98:0/2232531371 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe806e1f0 con 0x7f1fe8157b60 2023-11-28T17:49:55.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.788+0000 7f1fe6d9d640 1 --2- 172.21.15.98:0/2232531371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1fe80c4910 0x7f1fe80c4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:55.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.789+0000 7f1fdffff640 1 --2- 172.21.15.98:0/2232531371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe8157b60 0x7f1fe815bc10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009870 tx=0x7f1fd002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a4399750f139430 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:55.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.789+0000 7f1fe5d9b640 1 -- 172.21.15.98:0/2232531371 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd002f840 con 0x7f1fe8157b60 2023-11-28T17:49:55.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.789+0000 7f1fe5d9b640 1 -- 172.21.15.98:0/2232531371 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd002f9a0 con 0x7f1fe8157b60 2023-11-28T17:49:55.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.789+0000 7f1fe5d9b640 1 -- 172.21.15.98:0/2232531371 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd002fd30 con 0x7f1fe8157b60 2023-11-28T17:49:55.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.790+0000 7f1feca00640 1 -- 172.21.15.98:0/2232531371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe8157b60 msgr2=0x7f1fe815bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:55.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.790+0000 7f1feca00640 1 --2- 172.21.15.98:0/2232531371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe8157b60 0x7f1fe815bc10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009870 tx=0x7f1fd002ba20 comp rx=0 tx=0).stop 2023-11-28T17:49:55.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.790+0000 7f1feca00640 1 -- 172.21.15.98:0/2232531371 shutdown_connections 2023-11-28T17:49:55.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.790+0000 7f1feca00640 1 --2- 172.21.15.98:0/2232531371 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1fe80c4910 0x7f1fe80c4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:55.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.790+0000 7f1feca00640 1 --2- 172.21.15.98:0/2232531371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe8157b60 0x7f1fe815bc10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:55.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.790+0000 7f1feca00640 1 -- 172.21.15.98:0/2232531371 >> 172.21.15.98:0/2232531371 conn(0x7f1fe806a4d0 msgr2=0x7f1fe806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:55.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.790+0000 7f1feca00640 1 -- 172.21.15.98:0/2232531371 shutdown_connections 2023-11-28T17:49:55.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.790+0000 7f1feca00640 1 -- 172.21.15.98:0/2232531371 wait complete. 2023-11-28T17:49:55.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.792+0000 7f1feca00640 1 Processor -- start 2023-11-28T17:49:55.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.792+0000 7f1feca00640 1 -- start start 2023-11-28T17:49:55.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.793+0000 7f1feca00640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1fe80c4910 0x7f1fe814e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:55.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.793+0000 7f1feca00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe8151d80 0x7f1fe8155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:55.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.793+0000 7f1feca00640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fe8157b60 0x7f1fe81561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:55.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.793+0000 7f1feca00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe81f30d0 con 0x7f1fe8151d80 2023-11-28T17:49:55.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.793+0000 7f1feca00640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe80c18b0 con 0x7f1fe8157b60 2023-11-28T17:49:55.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.793+0000 7f1feca00640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe81f3300 con 0x7f1fe80c4910 2023-11-28T17:49:55.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.793+0000 7f1fdffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe8151d80 0x7f1fe8155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:55.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.793+0000 7f1fdffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe8151d80 0x7f1fe8155c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59592/0 (socket says 172.21.15.98:59592) 2023-11-28T17:49:55.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.793+0000 7f1fdffff640 1 -- 172.21.15.98:0/824126099 learned_addr learned my addr 172.21.15.98:0/824126099 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:55.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.794+0000 7f1fe6d9d640 1 --2- 172.21.15.98:0/824126099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fe8157b60 0x7f1fe81561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:55.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.794+0000 7f1fdffff640 1 -- 172.21.15.98:0/824126099 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1fe80c4910 msgr2=0x7f1fe814e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:55.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.794+0000 7f1fe659c640 1 --2- 172.21.15.98:0/824126099 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1fe80c4910 0x7f1fe814e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:55.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.794+0000 7f1fdffff640 1 --2- 172.21.15.98:0/824126099 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1fe80c4910 0x7f1fe814e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:55.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.794+0000 7f1fdffff640 1 -- 172.21.15.98:0/824126099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fe8157b60 msgr2=0x7f1fe81561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:55.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.794+0000 7f1fdffff640 1 --2- 172.21.15.98:0/824126099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fe8157b60 0x7f1fe81561c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:55.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.794+0000 7f1fdffff640 1 -- 172.21.15.98:0/824126099 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd00094d0 con 0x7f1fe8151d80 2023-11-28T17:49:55.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.794+0000 7f1fe6d9d640 1 --2- 172.21.15.98:0/824126099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fe8157b60 0x7f1fe81561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:55.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.794+0000 7f1fdffff640 1 --2- 172.21.15.98:0/824126099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe8151d80 0x7f1fe8155c80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1fd002ba60 tx=0x7f1fd0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:55.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.794+0000 7f1fdeffd640 1 -- 172.21.15.98:0/824126099 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd003c040 con 0x7f1fe8151d80 2023-11-28T17:49:55.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.794+0000 7f1fdeffd640 1 -- 172.21.15.98:0/824126099 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd002f810 con 0x7f1fe8151d80 2023-11-28T17:49:55.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.795+0000 7f1fdeffd640 1 -- 172.21.15.98:0/824126099 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0042610 con 0x7f1fe8151d80 2023-11-28T17:49:55.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.795+0000 7f1feca00640 1 -- 172.21.15.98:0/824126099 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe81efeb0 con 0x7f1fe8151d80 2023-11-28T17:49:55.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.795+0000 7f1feca00640 1 -- 172.21.15.98:0/824126099 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe81f0470 con 0x7f1fe8151d80 2023-11-28T17:49:55.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.796+0000 7f1feca00640 1 -- 172.21.15.98:0/824126099 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe8067bf0 con 0x7f1fe8151d80 2023-11-28T17:49:55.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.797+0000 7f1fdeffd640 1 -- 172.21.15.98:0/824126099 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f1fd002d3d0 con 0x7f1fe8151d80 2023-11-28T17:49:55.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.797+0000 7f1fdeffd640 1 --2- 172.21.15.98:0/824126099 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1fc806b2d0 0x7f1fc806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:55.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.798+0000 7f1fdeffd640 1 -- 172.21.15.98:0/824126099 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1fd00ac060 con 0x7f1fe8151d80 2023-11-28T17:49:55.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.798+0000 7f1fe659c640 1 --2- 172.21.15.98:0/824126099 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1fc806b2d0 0x7f1fc806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:55.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.798+0000 7f1fe659c640 1 --2- 172.21.15.98:0/824126099 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1fc806b2d0 0x7f1fc806d790 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8000f60 tx=0x7f1fd8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.803+0000 7f1fdeffd640 1 -- 172.21.15.98:0/824126099 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1fd0076d20 con 0x7f1fe8151d80 2023-11-28T17:49:55.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:49:55.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.997+0000 7f1feca00640 1 -- 172.21.15.98:0/824126099 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fe806b2d0 con 0x7f1fe8151d80 2023-11-28T17:49:55.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:55.999+0000 7f1fdeffd640 1 -- 172.21.15.98:0/824126099 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16766 (secure 0 0 0) 0x7f1fd0047070 con 0x7f1fe8151d80 2023-11-28T17:49:56.000 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:49:56.000 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:42:18.305147+0000","last_up_change":"2023-11-28T17:41:09.556993+0000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:41:10.303280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:41:14.890871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:41:15.704103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6810","nonce":2188704385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6813","nonce":2188704385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6819","nonce":2188704385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6816","nonce":2188704385}]},"public_addr":"172.21.15.98:6810/2188704385","cluster_addr":"172.21.15.98:6813/2188704385","heartbeat_back_addr":"172.21.15.98:6819/2188704385","heartbeat_front_addr":"172.21.15.98:6816/2188704385","state":["exists","up"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6801","nonce":3918226860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6803","nonce":3918226860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6807","nonce":3918226860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6805","nonce":3918226860}]},"public_addr":"172.21.15.98:6801/3918226860","cluster_addr":"172.21.15.98:6803/3918226860","heartbeat_back_addr":"172.21.15.98:6807/3918226860","heartbeat_front_addr":"172.21.15.98:6805/3918226860","state":["exists","up"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6812","nonce":2691532400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6815","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6818","nonce":2691532400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6823","nonce":2691532400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6821","nonce":2691532400}]},"public_addr":"172.21.15.98:6812/2691532400","cluster_addr":"172.21.15.98:6818/2691532400","heartbeat_back_addr":"172.21.15.98:6823/2691532400","heartbeat_front_addr":"172.21.15.98:6821/2691532400","state":["exists","up"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6825","nonce":1905487008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6827","nonce":1905487008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6831","nonce":1905487008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6829","nonce":1905487008}]},"public_addr":"172.21.15.98:6825/1905487008","cluster_addr":"172.21.15.98:6827/1905487008","heartbeat_back_addr":"172.21.15.98:6831/1905487008","heartbeat_front_addr":"172.21.15.98:6829/1905487008","state":["exists","up"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6801","nonce":2974677215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6803","nonce":2974677215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6807","nonce":2974677215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6805","nonce":2974677215}]},"public_addr":"172.21.15.113:6801/2974677215","cluster_addr":"172.21.15.113:6803/2974677215","heartbeat_back_addr":"172.21.15.113:6807/2974677215","heartbeat_front_addr":"172.21.15.113:6805/2974677215","state":["exists","up"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6817","nonce":2685000355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6821","nonce":2685000355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6829","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6831","nonce":2685000355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6823","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6826","nonce":2685000355}]},"public_addr":"172.21.15.113:6817/2685000355","cluster_addr":"172.21.15.113:6821/2685000355","heartbeat_back_addr":"172.21.15.113:6831/2685000355","heartbeat_front_addr":"172.21.15.113:6826/2685000355","state":["exists","up"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6809","nonce":2764300831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6811","nonce":2764300831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6815","nonce":2764300831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6813","nonce":2764300831}]},"public_addr":"172.21.15.113:6809/2764300831","cluster_addr":"172.21.15.113:6811/2764300831","heartbeat_back_addr":"172.21.15.113:6815/2764300831","heartbeat_front_addr":"172.21.15.113:6813/2764300831","state":["exists","up"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6819","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6820","nonce":1656248400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6824","nonce":1656248400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6830","nonce":1656248400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6825","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6827","nonce":1656248400}]},"public_addr":"172.21.15.113:6820/1656248400","cluster_addr":"172.21.15.113:6824/1656248400","heartbeat_back_addr":"172.21.15.113:6830/1656248400","heartbeat_front_addr":"172.21.15.113:6827/1656248400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.379549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.312196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.117494+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.388128+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.360608+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.411101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.355995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.390362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:49:56.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.001+0000 7f1feca00640 1 -- 172.21.15.98:0/824126099 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1fc806b2d0 msgr2=0x7f1fc806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.001+0000 7f1feca00640 1 --2- 172.21.15.98:0/824126099 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1fc806b2d0 0x7f1fc806d790 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8000f60 tx=0x7f1fd8009040 comp rx=0 tx=0).stop 2023-11-28T17:49:56.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.001+0000 7f1feca00640 1 -- 172.21.15.98:0/824126099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe8151d80 msgr2=0x7f1fe8155c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.001+0000 7f1feca00640 1 --2- 172.21.15.98:0/824126099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe8151d80 0x7f1fe8155c80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1fd002ba60 tx=0x7f1fd0004730 comp rx=0 tx=0).stop 2023-11-28T17:49:56.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.002+0000 7f1feca00640 1 -- 172.21.15.98:0/824126099 shutdown_connections 2023-11-28T17:49:56.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.002+0000 7f1feca00640 1 --2- 172.21.15.98:0/824126099 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f1fc806b2d0 0x7f1fc806d790 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.002+0000 7f1feca00640 1 --2- 172.21.15.98:0/824126099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fe8157b60 0x7f1fe81561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.002+0000 7f1feca00640 1 --2- 172.21.15.98:0/824126099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fe8151d80 0x7f1fe8155c80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.002+0000 7f1feca00640 1 --2- 172.21.15.98:0/824126099 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1fe80c4910 0x7f1fe814e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.002+0000 7f1feca00640 1 -- 172.21.15.98:0/824126099 >> 172.21.15.98:0/824126099 conn(0x7f1fe806a4d0 msgr2=0x7f1fe80c7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:56.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.002+0000 7f1feca00640 1 -- 172.21.15.98:0/824126099 shutdown_connections 2023-11-28T17:49:56.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.002+0000 7f1feca00640 1 -- 172.21.15.98:0/824126099 wait complete. 2023-11-28T17:49:56.022 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:41:10.303280+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:41:14.890871+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:41:15.704103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:49:56.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:49:56.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.072+0000 7fc309e1e640 1 Processor -- start 2023-11-28T17:49:56.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.072+0000 7fc309e1e640 1 -- start start 2023-11-28T17:49:56.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.073+0000 7fc309e1e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3040c7870 0x7fc30415de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.073+0000 7fc309e1e640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fc3040c7140 0x7fc304156f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.074+0000 7fc309e1e640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30415a230 con 0x7fc3040c7c50 2023-11-28T17:49:56.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.074+0000 7fc309e1e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc30415a510 con 0x7fc3040c7870 2023-11-28T17:49:56.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.074+0000 7fc309e1e640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc30415a7f0 con 0x7fc3040c7140 2023-11-28T17:49:56.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.074+0000 7fc3037fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3040c7870 0x7fc30415de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.074+0000 7fc3037fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3040c7870 0x7fc30415de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59596/0 (socket says 172.21.15.98:59596) 2023-11-28T17:49:56.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.074+0000 7fc3037fe640 1 -- 172.21.15.98:0/326908120 learned_addr learned my addr 172.21.15.98:0/326908120 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:56.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.074+0000 7fc303fff640 1 --2- 172.21.15.98:0/326908120 >> v2:172.21.15.113:3300/0 conn(0x7fc3040c7140 0x7fc304156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.074+0000 7fc3037fe640 1 -- 172.21.15.98:0/326908120 >> v2:172.21.15.113:3300/0 conn(0x7fc3040c7140 msgr2=0x7fc304156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:56.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.074+0000 7fc3037fe640 1 --2- 172.21.15.98:0/326908120 >> v2:172.21.15.113:3300/0 conn(0x7fc3040c7140 0x7fc304156f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.074+0000 7fc3037fe640 1 -- 172.21.15.98:0/326908120 >> v1:172.21.15.113:6789/0 conn(0x7fc3040c7c50 legacy=0x7fc304071700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:56.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.074+0000 7fc3037fe640 1 -- 172.21.15.98:0/326908120 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30415aad0 con 0x7fc3040c7870 2023-11-28T17:49:56.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.075+0000 7fc3037fe640 1 --2- 172.21.15.98:0/326908120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3040c7870 0x7fc30415de60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc3040c9c80 tx=0x7fc2f402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9300ce26e7bbee9b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.076+0000 7fc3027fc640 1 -- 172.21.15.98:0/326908120 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f40317e0 con 0x7fc3040c7870 2023-11-28T17:49:56.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.076+0000 7fc3027fc640 1 -- 172.21.15.98:0/326908120 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f4031940 con 0x7fc3040c7870 2023-11-28T17:49:56.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.076+0000 7fc3027fc640 1 -- 172.21.15.98:0/326908120 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4031cd0 con 0x7fc3040c7870 2023-11-28T17:49:56.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.076+0000 7fc309e1e640 1 -- 172.21.15.98:0/326908120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3040c7870 msgr2=0x7fc30415de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.076+0000 7fc309e1e640 1 --2- 172.21.15.98:0/326908120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3040c7870 0x7fc30415de60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc3040c9c80 tx=0x7fc2f402b880 comp rx=0 tx=0).stop 2023-11-28T17:49:56.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.076+0000 7fc309e1e640 1 -- 172.21.15.98:0/326908120 shutdown_connections 2023-11-28T17:49:56.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.076+0000 7fc309e1e640 1 --2- 172.21.15.98:0/326908120 >> v2:172.21.15.113:3300/0 conn(0x7fc3040c7140 0x7fc304156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.077+0000 7fc309e1e640 1 --2- 172.21.15.98:0/326908120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3040c7870 0x7fc30415de60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.077+0000 7fc309e1e640 1 -- 172.21.15.98:0/326908120 >> 172.21.15.98:0/326908120 conn(0x7fc3040c8d10 msgr2=0x7fc3040c90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:56.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.077+0000 7fc309e1e640 1 -- 172.21.15.98:0/326908120 shutdown_connections 2023-11-28T17:49:56.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.077+0000 7fc309e1e640 1 -- 172.21.15.98:0/326908120 wait complete. 2023-11-28T17:49:56.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.079+0000 7fc309e1e640 1 Processor -- start 2023-11-28T17:49:56.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.079+0000 7fc309e1e640 1 -- start start 2023-11-28T17:49:56.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.081+0000 7fc309e1e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3040c7140 0x7fc3041ef240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.081+0000 7fc309e1e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3040c7870 0x7fc3040cb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.081+0000 7fc309e1e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3040c7c50 0x7fc3040c2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.081+0000 7fc309e1e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30415c2a0 con 0x7fc3040c7c50 2023-11-28T17:49:56.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.081+0000 7fc309e1e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30415c540 con 0x7fc3040c7140 2023-11-28T17:49:56.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.081+0000 7fc309e1e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc30415c820 con 0x7fc3040c7870 2023-11-28T17:49:56.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.081+0000 7fc303fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3040c7c50 0x7fc3040c2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.081+0000 7fc303fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3040c7c50 0x7fc3040c2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59602/0 (socket says 172.21.15.98:59602) 2023-11-28T17:49:56.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.081+0000 7fc303fff640 1 -- 172.21.15.98:0/1685172007 learned_addr learned my addr 172.21.15.98:0/1685172007 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:56.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.082+0000 7fc3037fe640 1 --2- 172.21.15.98:0/1685172007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3040c7140 0x7fc3041ef240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.082+0000 7fc302ffd640 1 --2- 172.21.15.98:0/1685172007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3040c7870 0x7fc3040cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.082+0000 7fc303fff640 1 -- 172.21.15.98:0/1685172007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3040c7870 msgr2=0x7fc3040cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.082+0000 7fc303fff640 1 --2- 172.21.15.98:0/1685172007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3040c7870 0x7fc3040cb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.082+0000 7fc303fff640 1 -- 172.21.15.98:0/1685172007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3040c7140 msgr2=0x7fc3041ef240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.082+0000 7fc303fff640 1 --2- 172.21.15.98:0/1685172007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3040c7140 0x7fc3041ef240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.082+0000 7fc303fff640 1 -- 172.21.15.98:0/1685172007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f40094d0 con 0x7fc3040c7c50 2023-11-28T17:49:56.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.082+0000 7fc303fff640 1 --2- 172.21.15.98:0/1685172007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3040c7c50 0x7fc3040c2a80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc007e60 tx=0x7fc2fc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.082+0000 7fc2ebfff640 1 -- 172.21.15.98:0/1685172007 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc014420 con 0x7fc3040c7c50 2023-11-28T17:49:56.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.082+0000 7fc2ebfff640 1 -- 172.21.15.98:0/1685172007 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2fc0145a0 con 0x7fc3040c7c50 2023-11-28T17:49:56.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.082+0000 7fc2ebfff640 1 -- 172.21.15.98:0/1685172007 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc014950 con 0x7fc3040c7c50 2023-11-28T17:49:56.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.083+0000 7fc309e1e640 1 -- 172.21.15.98:0/1685172007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc30415cb00 con 0x7fc3040c7c50 2023-11-28T17:49:56.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.083+0000 7fc309e1e640 1 -- 172.21.15.98:0/1685172007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc30415d0c0 con 0x7fc3040c7c50 2023-11-28T17:49:56.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.084+0000 7fc309e1e640 1 -- 172.21.15.98:0/1685172007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc30415d280 con 0x7fc3040c7c50 2023-11-28T17:49:56.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.085+0000 7fc2ebfff640 1 -- 172.21.15.98:0/1685172007 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fc2fc014ab0 con 0x7fc3040c7c50 2023-11-28T17:49:56.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.086+0000 7fc2ebfff640 1 --2- 172.21.15.98:0/1685172007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc2f006b2d0 0x7fc2f006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.086+0000 7fc2ebfff640 1 -- 172.21.15.98:0/1685172007 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc2fc08a5f0 con 0x7fc3040c7c50 2023-11-28T17:49:56.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.087+0000 7fc3037fe640 1 --2- 172.21.15.98:0/1685172007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc2f006b2d0 0x7fc2f006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.088+0000 7fc3037fe640 1 --2- 172.21.15.98:0/1685172007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc2f006b2d0 0x7fc2f006d790 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4036000 tx=0x7fc2f403d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.091+0000 7fc2ebfff640 1 -- 172.21.15.98:0/1685172007 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2fc055350 con 0x7fc3040c7c50 2023-11-28T17:49:56.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.209+0000 7f281a009640 1 Processor -- start 2023-11-28T17:49:56.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.209+0000 7f281a009640 1 -- start start 2023-11-28T17:49:56.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.211+0000 7f281a009640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28141540d0 0x7f281415dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.211+0000 7f281a009640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f28140cb070 0x7f281406b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.211+0000 7f281a009640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281406a660 con 0x7f28140ca280 2023-11-28T17:49:56.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.211+0000 7f281a009640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28140cbdd0 con 0x7f28140cb070 2023-11-28T17:49:56.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.211+0000 7f281a009640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28141544b0 con 0x7f28141540d0 2023-11-28T17:49:56.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.211+0000 7f2812ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f28140cb070 0x7f281406b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.212+0000 7f2812ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f28140cb070 0x7f281406b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:36542/0 (socket says 172.21.15.98:36542) 2023-11-28T17:49:56.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.212+0000 7f2812ffd640 1 -- 172.21.15.98:0/928558765 learned_addr learned my addr 172.21.15.98:0/928558765 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:56.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.212+0000 7f28137fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28141540d0 0x7f281415dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.212+0000 7f2812ffd640 1 -- 172.21.15.98:0/928558765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28141540d0 msgr2=0x7f281415dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:56.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.212+0000 7f2812ffd640 1 --2- 172.21.15.98:0/928558765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28141540d0 0x7f281415dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.212+0000 7f2812ffd640 1 -- 172.21.15.98:0/928558765 >> v1:172.21.15.113:6789/0 conn(0x7f28140ca280 legacy=0x7f2814159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.212+0000 7f2812ffd640 1 -- 172.21.15.98:0/928558765 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281415e1b0 con 0x7f28140cb070 2023-11-28T17:49:56.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.213+0000 7f2812ffd640 1 --2- 172.21.15.98:0/928558765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28140cb070 0x7f281406b950 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2808009870 tx=0x7f280802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d07df07eb67efbe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.214+0000 7f28127fc640 1 -- 172.21.15.98:0/928558765 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808030840 con 0x7f28140cb070 2023-11-28T17:49:56.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.214+0000 7f28127fc640 1 -- 172.21.15.98:0/928558765 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28080309a0 con 0x7f28140cb070 2023-11-28T17:49:56.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.214+0000 7f28127fc640 1 -- 172.21.15.98:0/928558765 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808030d30 con 0x7f28140cb070 2023-11-28T17:49:56.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.214+0000 7f281a009640 1 -- 172.21.15.98:0/928558765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28140cb070 msgr2=0x7f281406b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.214+0000 7f281a009640 1 --2- 172.21.15.98:0/928558765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28140cb070 0x7f281406b950 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2808009870 tx=0x7f280802b980 comp rx=0 tx=0).stop 2023-11-28T17:49:56.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.214+0000 7f281a009640 1 -- 172.21.15.98:0/928558765 shutdown_connections 2023-11-28T17:49:56.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.214+0000 7f281a009640 1 --2- 172.21.15.98:0/928558765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28140cb070 0x7f281406b950 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.214+0000 7f281a009640 1 --2- 172.21.15.98:0/928558765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28141540d0 0x7f281415dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.214+0000 7f281a009640 1 -- 172.21.15.98:0/928558765 >> 172.21.15.98:0/928558765 conn(0x7f28140cf490 msgr2=0x7f28140cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.215+0000 7f281a009640 1 -- 172.21.15.98:0/928558765 shutdown_connections 2023-11-28T17:49:56.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.215+0000 7f281a009640 1 -- 172.21.15.98:0/928558765 wait complete. 2023-11-28T17:49:56.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.216+0000 7f281a009640 1 Processor -- start 2023-11-28T17:49:56.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.216+0000 7f281a009640 1 -- start start 2023-11-28T17:49:56.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.217+0000 7f281a009640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28140ca280 0x7f281415a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.217+0000 7f281a009640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28140cb070 0x7f28140c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.217+0000 7f281a009640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28141540d0 0x7f281415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.217+0000 7f281a009640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28140c3200 con 0x7f28140cb070 2023-11-28T17:49:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.217+0000 7f281a009640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28140c4c00 con 0x7f28140ca280 2023-11-28T17:49:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.217+0000 7f281a009640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2814156360 con 0x7f28141540d0 2023-11-28T17:49:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.218+0000 7f2812ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28140cb070 0x7f28140c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.218+0000 7f2812ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28140cb070 0x7f28140c3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59606/0 (socket says 172.21.15.98:59606) 2023-11-28T17:49:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.218+0000 7f2812ffd640 1 -- 172.21.15.98:0/57596667 learned_addr learned my addr 172.21.15.98:0/57596667 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.218+0000 7f28137fe640 1 --2- 172.21.15.98:0/57596667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28140ca280 0x7f281415a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.218+0000 7f2813fff640 1 --2- 172.21.15.98:0/57596667 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28141540d0 0x7f281415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.218+0000 7f2812ffd640 1 -- 172.21.15.98:0/57596667 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28141540d0 msgr2=0x7f281415c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.218+0000 7f2812ffd640 1 --2- 172.21.15.98:0/57596667 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28141540d0 0x7f281415c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.218+0000 7f2812ffd640 1 -- 172.21.15.98:0/57596667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28140ca280 msgr2=0x7f281415a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.218+0000 7f2812ffd640 1 --2- 172.21.15.98:0/57596667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28140ca280 0x7f281415a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.218+0000 7f2812ffd640 1 -- 172.21.15.98:0/57596667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28080094d0 con 0x7f28140cb070 2023-11-28T17:49:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.218+0000 7f2813fff640 1 --2- 172.21.15.98:0/57596667 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28141540d0 0x7f281415c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.219+0000 7f2812ffd640 1 --2- 172.21.15.98:0/57596667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28140cb070 0x7f28140c3e50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2808000c00 tx=0x7f2808004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.219+0000 7f27fbfff640 1 -- 172.21.15.98:0/57596667 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808039c90 con 0x7f28140cb070 2023-11-28T17:49:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.219+0000 7f281a009640 1 -- 172.21.15.98:0/57596667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28140c5d20 con 0x7f28140cb070 2023-11-28T17:49:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.219+0000 7f27fbfff640 1 -- 172.21.15.98:0/57596667 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2808039df0 con 0x7f28140cb070 2023-11-28T17:49:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.219+0000 7f27fbfff640 1 -- 172.21.15.98:0/57596667 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808043640 con 0x7f28140cb070 2023-11-28T17:49:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.219+0000 7f281a009640 1 -- 172.21.15.98:0/57596667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28140c5110 con 0x7f28140cb070 2023-11-28T17:49:56.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.220+0000 7f281a009640 1 -- 172.21.15.98:0/57596667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2814074f60 con 0x7f28140cb070 2023-11-28T17:49:56.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.221+0000 7f27fbfff640 1 -- 172.21.15.98:0/57596667 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f28080437a0 con 0x7f28140cb070 2023-11-28T17:49:56.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.222+0000 7f27fbfff640 1 --2- 172.21.15.98:0/57596667 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f280006b2a0 0x7f280006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.223+0000 7f27fbfff640 1 -- 172.21.15.98:0/57596667 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f280803e070 con 0x7f28140cb070 2023-11-28T17:49:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.223+0000 7f28137fe640 1 --2- 172.21.15.98:0/57596667 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f280006b2a0 0x7f280006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.223+0000 7f28137fe640 1 --2- 172.21.15.98:0/57596667 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f280006b2a0 0x7f280006d760 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2804009a30 tx=0x7f2804008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.227+0000 7f27fbfff640 1 -- 172.21.15.98:0/57596667 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2808077ae0 con 0x7f28140cb070 2023-11-28T17:49:56.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.351+0000 7fc309e1e640 1 -- 172.21.15.98:0/1685172007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc304074f60 con 0x7fc3040c7c50 2023-11-28T17:49:56.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.352+0000 7fc2ebfff640 1 -- 172.21.15.98:0/1685172007 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7fc2fc058970 con 0x7fc3040c7c50 2023-11-28T17:49:56.354 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","modified":"2023-11-28T17:40:09.105954Z","created":"2023-11-28T17:40:09.105954Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3301","nonce":0},{"type":"v1","addr":"172.21.15.113:6790","nonce":0}]},"addr":"172.21.15.113:6790/0","public_addr":"172.21.15.113:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2023-11-28T17:49:56.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.354+0000 7fc309e1e640 1 -- 172.21.15.98:0/1685172007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc2f006b2d0 msgr2=0x7fc2f006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.354+0000 7fc309e1e640 1 --2- 172.21.15.98:0/1685172007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc2f006b2d0 0x7fc2f006d790 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4036000 tx=0x7fc2f403d040 comp rx=0 tx=0).stop 2023-11-28T17:49:56.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.354+0000 7fc309e1e640 1 -- 172.21.15.98:0/1685172007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3040c7c50 msgr2=0x7fc3040c2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.354+0000 7fc309e1e640 1 --2- 172.21.15.98:0/1685172007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3040c7c50 0x7fc3040c2a80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc007e60 tx=0x7fc2fc00e4b0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.354+0000 7fc309e1e640 1 -- 172.21.15.98:0/1685172007 shutdown_connections 2023-11-28T17:49:56.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.354+0000 7fc309e1e640 1 --2- 172.21.15.98:0/1685172007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3040c7c50 0x7fc3040c2a80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.354+0000 7fc309e1e640 1 --2- 172.21.15.98:0/1685172007 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fc3040c7870 0x7fc3040cb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.354+0000 7fc309e1e640 1 --2- 172.21.15.98:0/1685172007 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fc2f006b2d0 0x7fc2f006d790 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.354+0000 7fc309e1e640 1 --2- 172.21.15.98:0/1685172007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc3040c7140 0x7fc3041ef240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.354+0000 7fc309e1e640 1 -- 172.21.15.98:0/1685172007 >> 172.21.15.98:0/1685172007 conn(0x7fc3040c8d10 msgr2=0x7fc304157d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:56.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.355+0000 7fc309e1e640 1 -- 172.21.15.98:0/1685172007 shutdown_connections 2023-11-28T17:49:56.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.355+0000 7fc309e1e640 1 -- 172.21.15.98:0/1685172007 wait complete. 2023-11-28T17:49:56.373 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2023-11-28T17:49:56.373 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2023-11-28T17:49:56.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2023-11-28T17:49:56.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.421+0000 7f281a009640 1 -- 172.21.15.98:0/57596667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f28140cae50 con 0x7f28140cb070 2023-11-28T17:49:56.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.421+0000 7f27fbfff640 1 -- 172.21.15.98:0/57596667 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2808034020 con 0x7f28140cb070 2023-11-28T17:49:56.422 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-11-28T17:49:56.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.424+0000 7f281a009640 1 -- 172.21.15.98:0/57596667 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f280006b2a0 msgr2=0x7f280006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.424+0000 7f281a009640 1 --2- 172.21.15.98:0/57596667 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f280006b2a0 0x7f280006d760 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2804009a30 tx=0x7f2804008040 comp rx=0 tx=0).stop 2023-11-28T17:49:56.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.424+0000 7f281a009640 1 -- 172.21.15.98:0/57596667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28140cb070 msgr2=0x7f28140c3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.424+0000 7f281a009640 1 --2- 172.21.15.98:0/57596667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28140cb070 0x7f28140c3e50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2808000c00 tx=0x7f2808004730 comp rx=0 tx=0).stop 2023-11-28T17:49:56.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.424+0000 7f281a009640 1 -- 172.21.15.98:0/57596667 shutdown_connections 2023-11-28T17:49:56.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.424+0000 7f281a009640 1 --2- 172.21.15.98:0/57596667 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f28141540d0 0x7f281415c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.424+0000 7f281a009640 1 --2- 172.21.15.98:0/57596667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28140cb070 0x7f28140c3e50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.424+0000 7f281a009640 1 --2- 172.21.15.98:0/57596667 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f280006b2a0 0x7f280006d760 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.424+0000 7f281a009640 1 --2- 172.21.15.98:0/57596667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28140ca280 0x7f281415a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.424+0000 7f281a009640 1 -- 172.21.15.98:0/57596667 >> 172.21.15.98:0/57596667 conn(0x7f28140cf490 msgr2=0x7f28140cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:56.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.424+0000 7f281a009640 1 -- 172.21.15.98:0/57596667 shutdown_connections 2023-11-28T17:49:56.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.424+0000 7f281a009640 1 -- 172.21.15.98:0/57596667 wait complete. 2023-11-28T17:49:56.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:49:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.560+0000 7fdd20eec640 1 Processor -- start 2023-11-28T17:49:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.560+0000 7fdd20eec640 1 -- start start 2023-11-28T17:49:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.562+0000 7fdd20eec640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd1c0c4760 0x7fdd1c0c2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.562+0000 7fdd20eec640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c156980 con 0x7fdd1c15a0f0 2023-11-28T17:49:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.562+0000 7fdd20eec640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c156c40 con 0x7fdd1c0c4380 2023-11-28T17:49:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.562+0000 7fdd20eec640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c0c2a60 con 0x7fdd1c0c4760 2023-11-28T17:49:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.562+0000 7fdd1ad9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdd1c15a0f0 0x7fdd1c15a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33950/0 (socket says 172.21.15.98:33950) 2023-11-28T17:49:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.562+0000 7fdd1ad9d640 1 -- 172.21.15.98:0/2290746847 learned_addr learned my addr 172.21.15.98:0/2290746847 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:49:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.562+0000 7fdd13fff640 1 --2- 172.21.15.98:0/2290746847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd1c0c4760 0x7fdd1c0c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd19d9b640 1 -- 172.21.15.98:0/2290746847 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192284549 0 0) 0x7fdd1c156980 con 0x7fdd1c15a0f0 2023-11-28T17:49:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd19d9b640 1 -- 172.21.15.98:0/2290746847 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd08005730 con 0x7fdd1c15a0f0 2023-11-28T17:49:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd19d9b640 1 -- 172.21.15.98:0/2290746847 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1786757096 0 0) 0x7fdd08005730 con 0x7fdd1c15a0f0 2023-11-28T17:49:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd19d9b640 1 -- 172.21.15.98:0/2290746847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd1c0c4760 msgr2=0x7fdd1c0c2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd19d9b640 1 --2- 172.21.15.98:0/2290746847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd1c0c4760 0x7fdd1c0c2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd19d9b640 1 -- 172.21.15.98:0/2290746847 >> v1:172.21.15.113:6789/0 conn(0x7fdd1c0c4380 legacy=0x7fdd1c0c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd19d9b640 1 -- 172.21.15.98:0/2290746847 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c156f20 con 0x7fdd1c15a0f0 2023-11-28T17:49:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd19d9b640 1 -- 172.21.15.98:0/2290746847 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7fdd140019f0 con 0x7fdd1c15a0f0 2023-11-28T17:49:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd19d9b640 1 -- 172.21.15.98:0/2290746847 >> v1:172.21.15.98:6789/0 conn(0x7fdd1c15a0f0 legacy=0x7fdd1c15a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd19d9b640 1 --2- 172.21.15.98:0/2290746847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd08005bb0 0x7fdd08005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd19d9b640 1 --2- 172.21.15.98:0/2290746847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd080069d0 0x7fdd08006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd19d9b640 1 --2- 172.21.15.98:0/2290746847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd1c0c4760 0x7fdd08007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd19d9b640 1 -- 172.21.15.98:0/2290746847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c0c2a60 con 0x7fdd08005bb0 2023-11-28T17:49:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd19d9b640 1 -- 172.21.15.98:0/2290746847 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd08003520 con 0x7fdd080069d0 2023-11-28T17:49:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd19d9b640 1 -- 172.21.15.98:0/2290746847 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd080036e0 con 0x7fdd1c0c4760 2023-11-28T17:49:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.563+0000 7fdd1a59c640 1 --2- 172.21.15.98:0/2290746847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd08005bb0 0x7fdd08005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.564+0000 7fdd13fff640 1 --2- 172.21.15.98:0/2290746847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd080069d0 0x7fdd08006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.564+0000 7fdd1ad9d640 1 --2- 172.21.15.98:0/2290746847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd1c0c4760 0x7fdd08007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.564+0000 7fdd1a59c640 1 -- 172.21.15.98:0/2290746847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd1c0c4760 msgr2=0x7fdd08007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.564+0000 7fdd1a59c640 1 --2- 172.21.15.98:0/2290746847 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd1c0c4760 0x7fdd08007390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.564+0000 7fdd1a59c640 1 -- 172.21.15.98:0/2290746847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd080069d0 msgr2=0x7fdd08006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.564+0000 7fdd1a59c640 1 --2- 172.21.15.98:0/2290746847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd080069d0 0x7fdd08006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.564+0000 7fdd1a59c640 1 -- 172.21.15.98:0/2290746847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd08005730 con 0x7fdd08005bb0 2023-11-28T17:49:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.564+0000 7fdd13fff640 1 --2- 172.21.15.98:0/2290746847 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd080069d0 0x7fdd08006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.565+0000 7fdd1a59c640 1 --2- 172.21.15.98:0/2290746847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd08005bb0 0x7fdd08005fd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c001720 tx=0x7fdd08007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b360495f1f6397a5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.565+0000 7fdd19d9b640 1 -- 172.21.15.98:0/2290746847 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c0052d0 con 0x7fdd08005bb0 2023-11-28T17:49:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.565+0000 7fdd19d9b640 1 -- 172.21.15.98:0/2290746847 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0c02e050 con 0x7fdd08005bb0 2023-11-28T17:49:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.565+0000 7fdd19d9b640 1 -- 172.21.15.98:0/2290746847 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c0035f0 con 0x7fdd08005bb0 2023-11-28T17:49:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.566+0000 7fdd20eec640 1 -- 172.21.15.98:0/2290746847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd08005bb0 msgr2=0x7fdd08005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.566+0000 7fdd20eec640 1 --2- 172.21.15.98:0/2290746847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd08005bb0 0x7fdd08005fd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c001720 tx=0x7fdd08007a00 comp rx=0 tx=0).stop 2023-11-28T17:49:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.566+0000 7fdd1ad9d640 1 -- 172.21.15.98:0/2290746847 reap_dead start 2023-11-28T17:49:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.566+0000 7fdd20eec640 1 -- 172.21.15.98:0/2290746847 shutdown_connections 2023-11-28T17:49:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.566+0000 7fdd20eec640 1 -- 172.21.15.98:0/2290746847 >> 172.21.15.98:0/2290746847 conn(0x7fdd1c06c0f0 msgr2=0x7fdd1c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.566+0000 7fdd20eec640 1 -- 172.21.15.98:0/2290746847 shutdown_connections 2023-11-28T17:49:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.566+0000 7fdd20eec640 1 -- 172.21.15.98:0/2290746847 wait complete. 2023-11-28T17:49:56.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.568+0000 7fdd20eec640 1 Processor -- start 2023-11-28T17:49:56.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.568+0000 7fdd20eec640 1 -- start start 2023-11-28T17:49:56.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.569+0000 7fdd20eec640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd08005bb0 0x7fdd1c156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.569+0000 7fdd20eec640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd1c1eaf60 0x7fdd1c158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.569+0000 7fdd20eec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c157ae0 0x7fdd1c143a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.569+0000 7fdd20eec640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c14d140 con 0x7fdd1c157ae0 2023-11-28T17:49:56.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.569+0000 7fdd20eec640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c14fb90 con 0x7fdd1c1eaf60 2023-11-28T17:49:56.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.569+0000 7fdd20eec640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c155ef0 con 0x7fdd08005bb0 2023-11-28T17:49:56.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.569+0000 7fdd1ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c157ae0 0x7fdd1c143a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.569+0000 7fdd1ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c157ae0 0x7fdd1c143a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59626/0 (socket says 172.21.15.98:59626) 2023-11-28T17:49:56.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.569+0000 7fdd1ad9d640 1 -- 172.21.15.98:0/1897260658 learned_addr learned my addr 172.21.15.98:0/1897260658 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:56.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.569+0000 7fdd13fff640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd1c1eaf60 0x7fdd1c158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.569+0000 7fdd1a59c640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd08005bb0 0x7fdd1c156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.570+0000 7fdd1ad9d640 1 -- 172.21.15.98:0/1897260658 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd08005bb0 msgr2=0x7fdd1c156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.570+0000 7fdd1ad9d640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fdd08005bb0 0x7fdd1c156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.570+0000 7fdd1ad9d640 1 -- 172.21.15.98:0/1897260658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd1c1eaf60 msgr2=0x7fdd1c158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.570+0000 7fdd1ad9d640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd1c1eaf60 0x7fdd1c158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.570+0000 7fdd1ad9d640 1 -- 172.21.15.98:0/1897260658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c06b070 con 0x7fdd1c157ae0 2023-11-28T17:49:56.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.570+0000 7fdd13fff640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd1c1eaf60 0x7fdd1c158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:49:56.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.570+0000 7fdd1ad9d640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c157ae0 0x7fdd1c143a00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c15a4d0 tx=0x7fdd1400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.570+0000 7fdd12ffd640 1 -- 172.21.15.98:0/1897260658 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd14004d60 con 0x7fdd1c157ae0 2023-11-28T17:49:56.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.570+0000 7fdd12ffd640 1 -- 172.21.15.98:0/1897260658 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1400bb80 con 0x7fdd1c157ae0 2023-11-28T17:49:56.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.570+0000 7fdd20eec640 1 -- 172.21.15.98:0/1897260658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1c14f2f0 con 0x7fdd1c157ae0 2023-11-28T17:49:56.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.570+0000 7fdd12ffd640 1 -- 172.21.15.98:0/1897260658 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd14007110 con 0x7fdd1c157ae0 2023-11-28T17:49:56.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.570+0000 7fdd20eec640 1 -- 172.21.15.98:0/1897260658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1c14f8b0 con 0x7fdd1c157ae0 2023-11-28T17:49:56.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.572+0000 7fdd20eec640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c143f40 0x7fdd1c1ee4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.572+0000 7fdd20eec640 1 -- 172.21.15.98:0/1897260658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c0744d0 con 0x7fdd1c143f40 2023-11-28T17:49:56.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.572+0000 7fdd1a59c640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c143f40 0x7fdd1c1ee4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.572+0000 7fdd1a59c640 1 -- 172.21.15.98:0/1897260658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd1c150220 con 0x7fdd1c143f40 2023-11-28T17:49:56.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.573+0000 7fdd1a59c640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c143f40 0x7fdd1c1ee4d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdd080079b0 tx=0x7fdd0c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.573+0000 7fdd12ffd640 1 -- 172.21.15.98:0/1897260658 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c007a30 con 0x7fdd1c143f40 2023-11-28T17:49:56.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.573+0000 7fdd12ffd640 1 -- 172.21.15.98:0/1897260658 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fdd14007270 con 0x7fdd1c157ae0 2023-11-28T17:49:56.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.575+0000 7fdd12ffd640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdcfc06b340 0x7fdcfc06d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.575+0000 7fdd12ffd640 1 -- 172.21.15.98:0/1897260658 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7fdd0c007660 con 0x7fdd1c143f40 2023-11-28T17:49:56.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.575+0000 7fdd12ffd640 1 -- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c143f40 msgr2=0x7fdd1c1ee4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.575+0000 7fdd12ffd640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c143f40 0x7fdd1c1ee4d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdd080079b0 tx=0x7fdd0c001120 comp rx=0 tx=0).stop 2023-11-28T17:49:56.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.575+0000 7fdd12ffd640 1 -- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c143f40 msgr2=0x7fdd1c1ee4d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:49:56.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.575+0000 7fdd12ffd640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c143f40 0x7fdd1c1ee4d0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdd080079b0 tx=0x7fdd0c001120 comp rx=0 tx=0).stop 2023-11-28T17:49:56.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.575+0000 7fdd12ffd640 1 -- 172.21.15.98:0/1897260658 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdd14089da0 con 0x7fdd1c157ae0 2023-11-28T17:49:56.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.575+0000 7fdd13fff640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdcfc06b340 0x7fdcfc06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.576+0000 7fdd13fff640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdcfc06b340 0x7fdcfc06d800 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c1ec090 tx=0x7fdd04008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.582+0000 7fdd20eec640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c1eea10 0x7fdd1c1eedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.582+0000 7fdd20eec640 1 -- 172.21.15.98:0/1897260658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c1577c0 con 0x7fdd1c1eea10 2023-11-28T17:49:56.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.583+0000 7fdd1a59c640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c1eea10 0x7fdd1c1eedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.583+0000 7fdd1a59c640 1 -- 172.21.15.98:0/1897260658 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdd1c1574e0 con 0x7fdd1c1eea10 2023-11-28T17:49:56.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.583+0000 7fdd1a59c640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c1eea10 0x7fdd1c1eedf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c005040 tx=0x7fdd0c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.583+0000 7fdd12ffd640 1 -- 172.21.15.98:0/1897260658 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c003a00 con 0x7fdd1c1eea10 2023-11-28T17:49:56.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.584+0000 7fdd12ffd640 1 -- 172.21.15.98:0/1897260658 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4467 (secure 0 0 0) 0x7fdd0c003b60 con 0x7fdd1c1eea10 2023-11-28T17:49:56.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.584+0000 7fdd12ffd640 1 -- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c1eea10 msgr2=0x7fdd1c1eedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.584+0000 7fdd12ffd640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c1eea10 0x7fdd1c1eedf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c005040 tx=0x7fdd0c030000 comp rx=0 tx=0).stop 2023-11-28T17:49:56.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.584+0000 7fdd12ffd640 1 -- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c1eea10 msgr2=0x7fdd1c1eedf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:49:56.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.584+0000 7fdd12ffd640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c1eea10 0x7fdd1c1eedf0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c005040 tx=0x7fdd0c030000 comp rx=0 tx=0).stop 2023-11-28T17:49:56.585 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:49:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:49:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:49:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2023-11-28T17:49:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2023-11-28T17:49:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:49:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:49:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:49:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:49:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2023-11-28T17:49:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:49:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:49:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:49:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:49:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:49:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:49:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:49:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:49:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:49:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:49:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:49:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:49:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:49:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:49:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:49:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:49:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:49:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:49:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:49:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:49:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:49:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:49:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:49:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:49:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:49:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:49:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:49:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:49:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:49:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:49:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:49:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:49:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:49:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:49:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:49:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:49:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:49:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:49:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:49:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:49:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:49:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:49:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:49:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:49:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:49:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:49:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:49:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:49:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:49:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:49:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:49:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:49:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:49:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:49:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:49:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:49:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:49:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:49:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:49:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:49:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:49:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:49:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:49:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:49:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:49:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:49:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:49:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:49:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:49:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:49:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:49:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:49:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:49:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:49:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:49:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:49:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:49:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:49:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:49:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:49:56.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:49:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:49:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:49:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:49:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:49:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:49:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:49:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:49:56.595 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:49:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:49:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:49:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:49:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:49:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:49:56.596 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:49:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:49:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:49:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:49:56.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.598 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:49:56.598 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:49:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.584+0000 7fdd20eec640 1 -- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdcfc06b340 msgr2=0x7fdcfc06d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.584+0000 7fdd20eec640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fdcfc06b340 0x7fdcfc06d800 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c1ec090 tx=0x7fdd04008040 comp rx=0 tx=0).stop 2023-11-28T17:49:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.584+0000 7fdd20eec640 1 -- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c157ae0 msgr2=0x7fdd1c143a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.584+0000 7fdd20eec640 1 --2- 172.21.15.98:0/1897260658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd1c157ae0 0x7fdd1c143a00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c15a4d0 tx=0x7fdd1400b4a0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.584+0000 7fdd1ad9d640 1 -- 172.21.15.98:0/1897260658 reap_dead start 2023-11-28T17:49:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.584+0000 7fdd1ad9d640 1 -- 172.21.15.98:0/1897260658 reap_dead start 2023-11-28T17:49:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.584+0000 7fdd20eec640 1 -- 172.21.15.98:0/1897260658 shutdown_connections 2023-11-28T17:49:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.584+0000 7fdd20eec640 1 -- 172.21.15.98:0/1897260658 >> 172.21.15.98:0/1897260658 conn(0x7fdd1c06c0f0 msgr2=0x7fdd1c06dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.585+0000 7fdd20eec640 1 -- 172.21.15.98:0/1897260658 shutdown_connections 2023-11-28T17:49:56.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.585+0000 7fdd20eec640 1 -- 172.21.15.98:0/1897260658 wait complete. 2023-11-28T17:49:56.606 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:49:56.606 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2023-11-28T17:49:56.607 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2023-11-28T17:49:56.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.624+0000 7f5326848640 1 Processor -- start 2023-11-28T17:49:56.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.625+0000 7f5326848640 1 -- start start 2023-11-28T17:49:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.626+0000 7f5326848640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53200ca280 0x7f53200ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.626+0000 7f5326848640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532006c060 con 0x7f53200cb070 2023-11-28T17:49:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.626+0000 7f5326848640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532006a660 con 0x7f53201540d0 2023-11-28T17:49:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.626+0000 7f5326848640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53200cbdd0 con 0x7f53200ca280 2023-11-28T17:49:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.626+0000 7f531f7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f53200cb070 0x7f532006b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33954/0 (socket says 172.21.15.98:33954) 2023-11-28T17:49:56.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.626+0000 7f531f7fe640 1 -- 172.21.15.98:0/4164729367 learned_addr learned my addr 172.21.15.98:0/4164729367 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:49:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.626+0000 7f5324de5640 1 --2- 172.21.15.98:0/4164729367 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53200ca280 0x7f53200ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.627+0000 7f531effd640 1 -- 172.21.15.98:0/4164729367 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285451733 0 0) 0x7f5314001110 con 0x7f53200cb070 2023-11-28T17:49:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.627+0000 7f531effd640 1 -- 172.21.15.98:0/4164729367 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5308003520 con 0x7f53200cb070 2023-11-28T17:49:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.627+0000 7f531effd640 1 -- 172.21.15.98:0/4164729367 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 144289134 0 0) 0x7f5308003520 con 0x7f53200cb070 2023-11-28T17:49:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.627+0000 7f531effd640 1 -- 172.21.15.98:0/4164729367 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53200ca280 msgr2=0x7f53200ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.627+0000 7f531effd640 1 --2- 172.21.15.98:0/4164729367 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53200ca280 0x7f53200ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.627+0000 7f531effd640 1 -- 172.21.15.98:0/4164729367 >> v1:172.21.15.113:6789/0 conn(0x7f53201540d0 legacy=0x7f532015dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.627+0000 7f531effd640 1 -- 172.21.15.98:0/4164729367 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532006a8b0 con 0x7f53200cb070 2023-11-28T17:49:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.627+0000 7f531effd640 1 -- 172.21.15.98:0/4164729367 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f5314001800 con 0x7f53200cb070 2023-11-28T17:49:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.627+0000 7f531effd640 1 -- 172.21.15.98:0/4164729367 >> v1:172.21.15.98:6789/0 conn(0x7f53200cb070 legacy=0x7f532006b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.627+0000 7f531effd640 1 --2- 172.21.15.98:0/4164729367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53080039a0 0x7f5308003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.627+0000 7f531effd640 1 --2- 172.21.15.98:0/4164729367 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53200ca280 0x7f53080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.627+0000 7f531effd640 1 --2- 172.21.15.98:0/4164729367 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5308004d00 0x7f5308005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.627+0000 7f531effd640 1 -- 172.21.15.98:0/4164729367 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53200cbdd0 con 0x7f53080039a0 2023-11-28T17:49:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.628+0000 7f531effd640 1 -- 172.21.15.98:0/4164729367 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530800b980 con 0x7f5308004d00 2023-11-28T17:49:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.628+0000 7f531effd640 1 -- 172.21.15.98:0/4164729367 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530800bb40 con 0x7f53200ca280 2023-11-28T17:49:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.628+0000 7f531ffff640 1 --2- 172.21.15.98:0/4164729367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53080039a0 0x7f5308003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.628+0000 7f5324de5640 1 --2- 172.21.15.98:0/4164729367 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5308004d00 0x7f5308005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.628+0000 7f531ffff640 1 -- 172.21.15.98:0/4164729367 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53200ca280 msgr2=0x7f53080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.628+0000 7f531ffff640 1 --2- 172.21.15.98:0/4164729367 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53200ca280 0x7f53080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.628+0000 7f531ffff640 1 -- 172.21.15.98:0/4164729367 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5308004d00 msgr2=0x7f5308005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:56.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.628+0000 7f531ffff640 1 --2- 172.21.15.98:0/4164729367 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5308004d00 0x7f5308005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.628+0000 7f531ffff640 1 -- 172.21.15.98:0/4164729367 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5308003520 con 0x7f53080039a0 2023-11-28T17:49:56.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.628+0000 7f5324de5640 1 --2- 172.21.15.98:0/4164729367 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5308004d00 0x7f5308005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:56.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.629+0000 7f531ffff640 1 --2- 172.21.15.98:0/4164729367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53080039a0 0x7f5308003d80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5310001860 tx=0x7f53080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=628471cdc086bb33 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.629+0000 7f531effd640 1 -- 172.21.15.98:0/4164729367 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5310033640 con 0x7f53080039a0 2023-11-28T17:49:56.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.629+0000 7f531effd640 1 -- 172.21.15.98:0/4164729367 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53100337c0 con 0x7f53080039a0 2023-11-28T17:49:56.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.629+0000 7f531effd640 1 -- 172.21.15.98:0/4164729367 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5310033b70 con 0x7f53080039a0 2023-11-28T17:49:56.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.630+0000 7f5326848640 1 -- 172.21.15.98:0/4164729367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53080039a0 msgr2=0x7f5308003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.630+0000 7f5326848640 1 --2- 172.21.15.98:0/4164729367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53080039a0 0x7f5308003d80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5310001860 tx=0x7f53080057f0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.630+0000 7f5324de5640 1 -- 172.21.15.98:0/4164729367 reap_dead start 2023-11-28T17:49:56.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.630+0000 7f5326848640 1 -- 172.21.15.98:0/4164729367 shutdown_connections 2023-11-28T17:49:56.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.630+0000 7f5326848640 1 -- 172.21.15.98:0/4164729367 >> 172.21.15.98:0/4164729367 conn(0x7f53200cf490 msgr2=0x7f53200cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:56.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.630+0000 7f5326848640 1 -- 172.21.15.98:0/4164729367 shutdown_connections 2023-11-28T17:49:56.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.630+0000 7f5326848640 1 -- 172.21.15.98:0/4164729367 wait complete. 2023-11-28T17:49:56.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.633+0000 7f5326848640 1 Processor -- start 2023-11-28T17:49:56.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.633+0000 7f5326848640 1 -- start start 2023-11-28T17:49:56.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.635+0000 7f5326848640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53080039a0 0x7f532014de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.635+0000 7f5326848640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f532015a610 0x7f5320149e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.635+0000 7f5326848640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53200c3aa0 0x7f53200c5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.635+0000 7f5326848640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53200c51b0 con 0x7f532015a610 2023-11-28T17:49:56.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.635+0000 7f5326848640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5320159320 con 0x7f53200c3aa0 2023-11-28T17:49:56.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.635+0000 7f5326848640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532014e3b0 con 0x7f53080039a0 2023-11-28T17:49:56.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.635+0000 7f531f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f532015a610 0x7f5320149e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.635+0000 7f531f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f532015a610 0x7f5320149e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59668/0 (socket says 172.21.15.98:59668) 2023-11-28T17:49:56.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.635+0000 7f531f7fe640 1 -- 172.21.15.98:0/300407018 learned_addr learned my addr 172.21.15.98:0/300407018 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:56.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.635+0000 7f531ffff640 1 --2- 172.21.15.98:0/300407018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53080039a0 0x7f532014de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.635+0000 7f5324de5640 1 --2- 172.21.15.98:0/300407018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53200c3aa0 0x7f53200c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.635+0000 7f531f7fe640 1 -- 172.21.15.98:0/300407018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53080039a0 msgr2=0x7f532014de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.635+0000 7f531f7fe640 1 --2- 172.21.15.98:0/300407018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53080039a0 0x7f532014de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.636+0000 7f531f7fe640 1 -- 172.21.15.98:0/300407018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53200c3aa0 msgr2=0x7f53200c5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.636+0000 7f531f7fe640 1 --2- 172.21.15.98:0/300407018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53200c3aa0 0x7f53200c5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.636+0000 7f531f7fe640 1 -- 172.21.15.98:0/300407018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532014a3d0 con 0x7f532015a610 2023-11-28T17:49:56.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.636+0000 7f5324de5640 1 --2- 172.21.15.98:0/300407018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53200c3aa0 0x7f53200c5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:49:56.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.636+0000 7f531f7fe640 1 --2- 172.21.15.98:0/300407018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f532015a610 0x7f5320149e90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f531400c770 tx=0x7f531400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.636+0000 7f531cff9640 1 -- 172.21.15.98:0/300407018 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53140074e0 con 0x7f532015a610 2023-11-28T17:49:56.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.636+0000 7f531cff9640 1 -- 172.21.15.98:0/300407018 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5314007640 con 0x7f532015a610 2023-11-28T17:49:56.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.636+0000 7f531cff9640 1 -- 172.21.15.98:0/300407018 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5314004a00 con 0x7f532015a610 2023-11-28T17:49:56.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.636+0000 7f5326848640 1 -- 172.21.15.98:0/300407018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5320159570 con 0x7f532015a610 2023-11-28T17:49:56.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.636+0000 7f5326848640 1 -- 172.21.15.98:0/300407018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5320159b30 con 0x7f532015a610 2023-11-28T17:49:56.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.638+0000 7f5326848640 1 -- 172.21.15.98:0/300407018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5320074f60 con 0x7f532015a610 2023-11-28T17:49:56.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.639+0000 7f531cff9640 1 -- 172.21.15.98:0/300407018 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f5314004b60 con 0x7f532015a610 2023-11-28T17:49:56.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.640+0000 7f531cff9640 1 --2- 172.21.15.98:0/300407018 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f530c06b2d0 0x7f530c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.640+0000 7f531cff9640 1 -- 172.21.15.98:0/300407018 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5314088ef0 con 0x7f532015a610 2023-11-28T17:49:56.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.640+0000 7f531ffff640 1 --2- 172.21.15.98:0/300407018 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f530c06b2d0 0x7f530c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.641+0000 7f531ffff640 1 --2- 172.21.15.98:0/300407018 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f530c06b2d0 0x7f530c06d790 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5310035000 tx=0x7f5310009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.644+0000 7f531cff9640 1 -- 172.21.15.98:0/300407018 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5314053c50 con 0x7f532015a610 2023-11-28T17:49:56.653 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi098 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2023-11-28T17:49:56.733 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=001eb7dd2d2c4a7db44673ab0344338c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi098 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2023-11-28T17:49:56.736 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2023-11-28T17:49:56.746 INFO:tasks.mon_thrash:joining mon_thrasher 2023-11-28T17:49:56.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.791+0000 7f882a87c640 1 Processor -- start 2023-11-28T17:49:56.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.791+0000 7f882a87c640 1 -- start start 2023-11-28T17:49:56.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.793+0000 7f882a87c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88240cb070 0x7f882406b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.793+0000 7f882a87c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882406a660 con 0x7f88240ca280 2023-11-28T17:49:56.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.793+0000 7f882a87c640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882406a8b0 con 0x7f88241540d0 2023-11-28T17:49:56.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.793+0000 7f882a87c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88240cbdd0 con 0x7f88240cb070 2023-11-28T17:49:56.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.793+0000 7f8828e19640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f88240ca280 0x7f88241645f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33956/0 (socket says 172.21.15.98:33956) 2023-11-28T17:49:56.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.793+0000 7f8828e19640 1 -- 172.21.15.98:0/3740986840 learned_addr learned my addr 172.21.15.98:0/3740986840 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:49:56.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f881bfff640 1 --2- 172.21.15.98:0/3740986840 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88240cb070 0x7f882406b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675040754 0 0) 0x7f882406a660 con 0x7f88240ca280 2023-11-28T17:49:56.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8810003520 con 0x7f88240ca280 2023-11-28T17:49:56.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 <== mon.1 v1:172.21.15.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375719935 0 0) 0x7f882406a8b0 con 0x7f88241540d0 2023-11-28T17:49:56.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 --> v1:172.21.15.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f882406a660 con 0x7f88241540d0 2023-11-28T17:49:56.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1049205883 0 0) 0x7f8810003520 con 0x7f88240ca280 2023-11-28T17:49:56.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88240cb070 msgr2=0x7f882406b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:56.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 --2- 172.21.15.98:0/3740986840 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88240cb070 0x7f882406b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 >> v1:172.21.15.113:6789/0 conn(0x7f88241540d0 legacy=0x7f882415dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882406ab90 con 0x7f88240ca280 2023-11-28T17:49:56.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f881c0019f0 con 0x7f88240ca280 2023-11-28T17:49:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 >> v1:172.21.15.98:6789/0 conn(0x7f88240ca280 legacy=0x7f88241645f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 --2- 172.21.15.98:0/3740986840 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88240cb070 0x7f8810003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 --2- 172.21.15.98:0/3740986840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88100040e0 0x7f88100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 --2- 172.21.15.98:0/3740986840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8810004d00 0x7f8810005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88240cbdd0 con 0x7f8810004d00 2023-11-28T17:49:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881000bac0 con 0x7f88100040e0 2023-11-28T17:49:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.794+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f881000bc80 con 0x7f88240cb070 2023-11-28T17:49:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.795+0000 7f8828e19640 1 --2- 172.21.15.98:0/3740986840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8810004d00 0x7f8810005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.795+0000 7f8823fff640 1 --2- 172.21.15.98:0/3740986840 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88240cb070 0x7f8810003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.795+0000 7f8828e19640 1 -- 172.21.15.98:0/3740986840 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88240cb070 msgr2=0x7f8810003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.795+0000 7f8828e19640 1 --2- 172.21.15.98:0/3740986840 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f88240cb070 0x7f8810003b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.795+0000 7f8828e19640 1 -- 172.21.15.98:0/3740986840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88100040e0 msgr2=0x7f88100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.795+0000 7f8828e19640 1 --2- 172.21.15.98:0/3740986840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88100040e0 0x7f88100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.795+0000 7f8828e19640 1 -- 172.21.15.98:0/3740986840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8810003520 con 0x7f8810004d00 2023-11-28T17:49:56.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.796+0000 7f8828e19640 1 --2- 172.21.15.98:0/3740986840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8810004d00 0x7f8810005180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f881c00a710 tx=0x7f88100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=866872dbccf9db5c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.796+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c034630 con 0x7f8810004d00 2023-11-28T17:49:56.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.796+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f881c02f070 con 0x7f8810004d00 2023-11-28T17:49:56.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.796+0000 7f88237fe640 1 -- 172.21.15.98:0/3740986840 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c034a30 con 0x7f8810004d00 2023-11-28T17:49:56.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.797+0000 7f882a87c640 1 -- 172.21.15.98:0/3740986840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8810004d00 msgr2=0x7f8810005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.797+0000 7f882a87c640 1 --2- 172.21.15.98:0/3740986840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8810004d00 0x7f8810005180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f881c00a710 tx=0x7f88100057f0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.797+0000 7f8828e19640 1 -- 172.21.15.98:0/3740986840 reap_dead start 2023-11-28T17:49:56.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.797+0000 7f882a87c640 1 -- 172.21.15.98:0/3740986840 shutdown_connections 2023-11-28T17:49:56.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.797+0000 7f882a87c640 1 -- 172.21.15.98:0/3740986840 >> 172.21.15.98:0/3740986840 conn(0x7f88240cf490 msgr2=0x7f88240cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:56.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.797+0000 7f882a87c640 1 -- 172.21.15.98:0/3740986840 shutdown_connections 2023-11-28T17:49:56.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.797+0000 7f882a87c640 1 -- 172.21.15.98:0/3740986840 wait complete. 2023-11-28T17:49:56.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.799+0000 7f882a87c640 1 Processor -- start 2023-11-28T17:49:56.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.799+0000 7f882a87c640 1 -- start start 2023-11-28T17:49:56.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.799+0000 7f882a87c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8810004d00 0x7f88241efb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.799+0000 7f882a87c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f882414c7d0 0x7f8824146cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f882a87c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88240c5570 0x7f88240c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f882a87c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88241f3300 con 0x7f882414c7d0 2023-11-28T17:49:56.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f882a87c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88241f35e0 con 0x7f88240c5570 2023-11-28T17:49:56.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f882a87c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88241f38c0 con 0x7f8810004d00 2023-11-28T17:49:56.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f881bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f882414c7d0 0x7f8824146cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f881bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f882414c7d0 0x7f8824146cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59690/0 (socket says 172.21.15.98:59690) 2023-11-28T17:49:56.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f881bfff640 1 -- 172.21.15.98:0/641589194 learned_addr learned my addr 172.21.15.98:0/641589194 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:56.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f8823fff640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8810004d00 0x7f88241efb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f8828e19640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88240c5570 0x7f88240c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f881bfff640 1 -- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8810004d00 msgr2=0x7f88241efb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f881bfff640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8810004d00 0x7f88241efb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f881bfff640 1 -- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88240c5570 msgr2=0x7f88240c5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f881bfff640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88240c5570 0x7f88240c5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f881bfff640 1 -- 172.21.15.98:0/641589194 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88241f3ba0 con 0x7f882414c7d0 2023-11-28T17:49:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f8823fff640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f8810004d00 0x7f88241efb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f8828e19640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88240c5570 0x7f88240c5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.800+0000 7f881bfff640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f882414c7d0 0x7f8824146cd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f880c00e7c0 tx=0x7f880c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.801+0000 7f88217fa640 1 -- 172.21.15.98:0/641589194 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880c00eed0 con 0x7f882414c7d0 2023-11-28T17:49:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.801+0000 7f88217fa640 1 -- 172.21.15.98:0/641589194 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880c00c8c0 con 0x7f882414c7d0 2023-11-28T17:49:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.801+0000 7f88217fa640 1 -- 172.21.15.98:0/641589194 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880c00cc70 con 0x7f882414c7d0 2023-11-28T17:49:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.801+0000 7f882a87c640 1 -- 172.21.15.98:0/641589194 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88241f3e80 con 0x7f882414c7d0 2023-11-28T17:49:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.801+0000 7f882a87c640 1 -- 172.21.15.98:0/641589194 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88240d0000 con 0x7f882414c7d0 2023-11-28T17:49:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.802+0000 7f882a87c640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8824152740 0x7f8824152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.802+0000 7f882a87c640 1 -- 172.21.15.98:0/641589194 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88240d0280 con 0x7f8824152740 2023-11-28T17:49:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.803+0000 7f8823fff640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8824152740 0x7f8824152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.803+0000 7f88217fa640 1 -- 172.21.15.98:0/641589194 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f880c013470 con 0x7f882414c7d0 2023-11-28T17:49:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.803+0000 7f8823fff640 1 -- 172.21.15.98:0/641589194 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88241f3a30 con 0x7f8824152740 2023-11-28T17:49:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.804+0000 7f8823fff640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8824152740 0x7f8824152b20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8814000c00 tx=0x7f881400ebf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.804+0000 7f88217fa640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f880406b2a0 0x7f880406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.804+0000 7f88217fa640 1 -- 172.21.15.98:0/641589194 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f880c089c90 con 0x7f882414c7d0 2023-11-28T17:49:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.804+0000 7f8828e19640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f880406b2a0 0x7f880406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.804+0000 7f88217fa640 1 -- 172.21.15.98:0/641589194 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8814005270 con 0x7f8824152740 2023-11-28T17:49:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.805+0000 7f8828e19640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f880406b2a0 0x7f880406d760 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f881c034000 tx=0x7f881c02f000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.805+0000 7f88217fa640 1 -- 172.21.15.98:0/641589194 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f881400d040 con 0x7f8824152740 2023-11-28T17:49:56.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.805+0000 7f88217fa640 1 -- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8824152740 msgr2=0x7f8824152b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.805+0000 7f88217fa640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8824152740 0x7f8824152b20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8814000c00 tx=0x7f881400ebf0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.805+0000 7f88217fa640 1 -- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8824152740 msgr2=0x7f8824152b20 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:49:56.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.805+0000 7f88217fa640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8824152740 0x7f8824152b20 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8814000c00 tx=0x7f881400ebf0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.812+0000 7f882a87c640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88240c3090 0x7f88240c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:56.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.812+0000 7f882a87c640 1 -- 172.21.15.98:0/641589194 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8824074110 con 0x7f88240c3090 2023-11-28T17:49:56.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.813+0000 7f8823fff640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88240c3090 0x7f88240c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:56.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.813+0000 7f8823fff640 1 -- 172.21.15.98:0/641589194 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f88240c6890 con 0x7f88240c3090 2023-11-28T17:49:56.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.814+0000 7f8823fff640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88240c3090 0x7f88240c34a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8814017000 tx=0x7f8814000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:56.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.814+0000 7f88217fa640 1 -- 172.21.15.98:0/641589194 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8814022070 con 0x7f88240c3090 2023-11-28T17:49:56.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.815+0000 7f88217fa640 1 -- 172.21.15.98:0/641589194 <== mon.1 v2:172.21.15.113:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4975 (secure 0 0 0) 0x7f881400cde0 con 0x7f88240c3090 2023-11-28T17:49:56.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.815+0000 7f88217fa640 1 -- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88240c3090 msgr2=0x7f88240c34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.815+0000 7f88217fa640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88240c3090 0x7f88240c34a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8814017000 tx=0x7f8814000ed0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.815+0000 7f88217fa640 1 -- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88240c3090 msgr2=0x7f88240c34a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:49:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.815+0000 7f88217fa640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88240c3090 0x7f88240c34a0 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8814017000 tx=0x7f8814000ed0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.816 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:49:56.816 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:49:56.817 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:49:56.817 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:49:56.817 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2023-11-28T17:49:56.817 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:49:56.817 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:49:56.817 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:49:56.817 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:49:56.817 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:56.818 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2023-11-28T17:49:56.818 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:49:56.818 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:49:56.818 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:49:56.818 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:49:56.818 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:49:56.818 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:49:56.818 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:49:56.819 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:49:56.819 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:49:56.819 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:49:56.819 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:49:56.819 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:49:56.819 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:49:56.819 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:56.819 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:49:56.820 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:49:56.820 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:49:56.820 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:49:56.820 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:49:56.820 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:49:56.820 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:49:56.820 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:49:56.820 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:49:56.821 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:49:56.821 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:49:56.821 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:49:56.821 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:56.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.821 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:49:56.821 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:49:56.821 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:49:56.822 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:49:56.822 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:49:56.822 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:49:56.822 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:49:56.822 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:49:56.822 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:49:56.822 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:49:56.822 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:49:56.823 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:49:56.823 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:49:56.823 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:49:56.823 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:49:56.823 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:49:56.823 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:49:56.823 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:49:56.823 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:49:56.824 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:49:56.824 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:49:56.824 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:49:56.824 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:49:56.824 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:49:56.824 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:49:56.824 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:49:56.824 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:49:56.824 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:56.825 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:49:56.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.825 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:49:56.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.825 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:49:56.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:49:56.825 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:49:56.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:49:56.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:49:56.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:49:56.826 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:56.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:49:56.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:49:56.827 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:56.827 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:56.827 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:56.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:49:56.827 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:49:56.827 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:49:56.827 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:49:56.828 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:49:56.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.828 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:49:56.828 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:49:56.828 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:49:56.828 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:49:56.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:49:56.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:49:56.829 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:56.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:49:56.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:49:56.829 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:56.830 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:56.830 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:56.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.830 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:49:56.830 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:49:56.830 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:49:56.830 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:49:56.830 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:49:56.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.831 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:49:56.831 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:49:56.831 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:49:56.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:49:56.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:49:56.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:49:56.832 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:56.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:49:56.832 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:49:56.832 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:56.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:56.832 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:56.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:49:56.833 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:49:56.833 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:49:56.833 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:49:56.833 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:49:56.833 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:56.833 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:56.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.834 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:49:56.834 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:49:56.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.834 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:56.834 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:56.834 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:49:56.834 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:56.834 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:56.835 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:49:56.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.835 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:56.835 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:56.835 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2023-11-28T17:49:56.835 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:56.835 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:56.835 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:49:56.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.836 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:56.836 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:56.836 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:49:56.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:56.836 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:56.836 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:49:56.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.837 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:56.837 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:56.837 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:49:56.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:56.837 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:56.837 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:49:56.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:56.837 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:56.838 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:56.838 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:49:56.838 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:56.838 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:56.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:56.838 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:49:56.838 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:49:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.815+0000 7f882a87c640 1 -- 172.21.15.98:0/641589194 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f880406b2a0 msgr2=0x7f880406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.815+0000 7f882a87c640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f880406b2a0 0x7f880406d760 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f881c034000 tx=0x7f881c02f000 comp rx=0 tx=0).stop 2023-11-28T17:49:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.815+0000 7f882a87c640 1 -- 172.21.15.98:0/641589194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f882414c7d0 msgr2=0x7f8824146cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.815+0000 7f882a87c640 1 --2- 172.21.15.98:0/641589194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f882414c7d0 0x7f8824146cd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f880c00e7c0 tx=0x7f880c00ec80 comp rx=0 tx=0).stop 2023-11-28T17:49:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.815+0000 7f8828e19640 1 -- 172.21.15.98:0/641589194 reap_dead start 2023-11-28T17:49:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.815+0000 7f8828e19640 1 -- 172.21.15.98:0/641589194 reap_dead start 2023-11-28T17:49:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.815+0000 7f882a87c640 1 -- 172.21.15.98:0/641589194 shutdown_connections 2023-11-28T17:49:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.815+0000 7f882a87c640 1 -- 172.21.15.98:0/641589194 >> 172.21.15.98:0/641589194 conn(0x7f88240cf490 msgr2=0x7f8824150ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.815+0000 7f882a87c640 1 -- 172.21.15.98:0/641589194 shutdown_connections 2023-11-28T17:49:56.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.815+0000 7f882a87c640 1 -- 172.21.15.98:0/641589194 wait complete. 2023-11-28T17:49:56.841 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:49:56.842 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2023-11-28T17:49:56.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2023-11-28T17:49:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.838+0000 7f5326848640 1 -- 172.21.15.98:0/300407018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f532006c060 con 0x7f532015a610 2023-11-28T17:49:56.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.839+0000 7f531cff9640 1 -- 172.21.15.98:0/300407018 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f532006c060 con 0x7f532015a610 2023-11-28T17:49:56.843 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-11-28T17:49:56.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.841+0000 7f5326848640 1 -- 172.21.15.98:0/300407018 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f530c06b2d0 msgr2=0x7f530c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.841+0000 7f5326848640 1 --2- 172.21.15.98:0/300407018 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f530c06b2d0 0x7f530c06d790 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5310035000 tx=0x7f5310009030 comp rx=0 tx=0).stop 2023-11-28T17:49:56.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.841+0000 7f5326848640 1 -- 172.21.15.98:0/300407018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f532015a610 msgr2=0x7f5320149e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:56.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.842+0000 7f5326848640 1 --2- 172.21.15.98:0/300407018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f532015a610 0x7f5320149e90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f531400c770 tx=0x7f531400cc30 comp rx=0 tx=0).stop 2023-11-28T17:49:56.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.842+0000 7f5326848640 1 -- 172.21.15.98:0/300407018 shutdown_connections 2023-11-28T17:49:56.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.842+0000 7f5326848640 1 --2- 172.21.15.98:0/300407018 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f530c06b2d0 0x7f530c06d790 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.842+0000 7f5326848640 1 --2- 172.21.15.98:0/300407018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f53200c3aa0 0x7f53200c5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.842+0000 7f5326848640 1 --2- 172.21.15.98:0/300407018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f532015a610 0x7f5320149e90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.842+0000 7f5326848640 1 --2- 172.21.15.98:0/300407018 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f53080039a0 0x7f532014de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:56.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.842+0000 7f5326848640 1 -- 172.21.15.98:0/300407018 >> 172.21.15.98:0/300407018 conn(0x7f53200cf490 msgr2=0x7f53200c78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:56.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.842+0000 7f5326848640 1 -- 172.21.15.98:0/300407018 shutdown_connections 2023-11-28T17:49:56.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:56.842+0000 7f5326848640 1 -- 172.21.15.98:0/300407018 wait complete. 2023-11-28T17:49:56.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:49:57.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.038+0000 7f7c7bc46640 1 Processor -- start 2023-11-28T17:49:57.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.038+0000 7f7c7bc46640 1 -- start start 2023-11-28T17:49:57.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.040+0000 7f7c7bc46640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c741540d0 0x7f7c7415dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.040+0000 7f7c7bc46640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7c740cb070 0x7f7c7406b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.040+0000 7f7c7bc46640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7406a660 con 0x7f7c740ca280 2023-11-28T17:49:57.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.040+0000 7f7c7bc46640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c740cbdd0 con 0x7f7c740cb070 2023-11-28T17:49:57.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.040+0000 7f7c7bc46640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c741544b0 con 0x7f7c741540d0 2023-11-28T17:49:57.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.040+0000 7f7c791e1640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7c740cb070 0x7f7c7406b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.040+0000 7f7c799e2640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c741540d0 0x7f7c7415dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.040+0000 7f7c791e1640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f7c740cb070 0x7f7c7406b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.98:36628/0 (socket says 172.21.15.98:36628) 2023-11-28T17:49:57.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.040+0000 7f7c791e1640 1 -- 172.21.15.98:0/4196817461 learned_addr learned my addr 172.21.15.98:0/4196817461 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:57.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.041+0000 7f7c791e1640 1 -- 172.21.15.98:0/4196817461 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c741540d0 msgr2=0x7f7c7415dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:57.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.041+0000 7f7c791e1640 1 --2- 172.21.15.98:0/4196817461 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c741540d0 0x7f7c7415dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.041+0000 7f7c791e1640 1 -- 172.21.15.98:0/4196817461 >> v1:172.21.15.113:6789/0 conn(0x7f7c740ca280 legacy=0x7f7c74159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.041+0000 7f7c791e1640 1 -- 172.21.15.98:0/4196817461 --> v2:172.21.15.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7415e1b0 con 0x7f7c740cb070 2023-11-28T17:49:57.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.042+0000 7f7c791e1640 1 --2- 172.21.15.98:0/4196817461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c740cb070 0x7f7c7406b950 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009870 tx=0x7f7c6402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12bfd3a594c13339 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:57.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.043+0000 7f7c789e0640 1 -- 172.21.15.98:0/4196817461 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c64030840 con 0x7f7c740cb070 2023-11-28T17:49:57.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.043+0000 7f7c789e0640 1 -- 172.21.15.98:0/4196817461 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c640309a0 con 0x7f7c740cb070 2023-11-28T17:49:57.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.043+0000 7f7c789e0640 1 -- 172.21.15.98:0/4196817461 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c64030d30 con 0x7f7c740cb070 2023-11-28T17:49:57.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.043+0000 7f7c7bc46640 1 -- 172.21.15.98:0/4196817461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c740cb070 msgr2=0x7f7c7406b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.043+0000 7f7c7bc46640 1 --2- 172.21.15.98:0/4196817461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c740cb070 0x7f7c7406b950 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009870 tx=0x7f7c6402b980 comp rx=0 tx=0).stop 2023-11-28T17:49:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.043+0000 7f7c7bc46640 1 -- 172.21.15.98:0/4196817461 shutdown_connections 2023-11-28T17:49:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.043+0000 7f7c7bc46640 1 --2- 172.21.15.98:0/4196817461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c740cb070 0x7f7c7406b950 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.043+0000 7f7c7bc46640 1 --2- 172.21.15.98:0/4196817461 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c741540d0 0x7f7c7415dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.043+0000 7f7c7bc46640 1 -- 172.21.15.98:0/4196817461 >> 172.21.15.98:0/4196817461 conn(0x7f7c740cf490 msgr2=0x7f7c740cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.044+0000 7f7c7bc46640 1 -- 172.21.15.98:0/4196817461 shutdown_connections 2023-11-28T17:49:57.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.044+0000 7f7c7bc46640 1 -- 172.21.15.98:0/4196817461 wait complete. 2023-11-28T17:49:57.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.045+0000 7f7c7bc46640 1 Processor -- start 2023-11-28T17:49:57.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.045+0000 7f7c7bc46640 1 -- start start 2023-11-28T17:49:57.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.046+0000 7f7c7bc46640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c740ca280 0x7f7c7415a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.046+0000 7f7c7bc46640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740cb070 0x7f7c740c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.046+0000 7f7c7bc46640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c741540d0 0x7f7c7415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.046+0000 7f7c7bc46640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c740c3200 con 0x7f7c740ca280 2023-11-28T17:49:57.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.046+0000 7f7c7bc46640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c740c4c00 con 0x7f7c741540d0 2023-11-28T17:49:57.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.046+0000 7f7c7bc46640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c74156360 con 0x7f7c740cb070 2023-11-28T17:49:57.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.046+0000 7f7c799e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c740ca280 0x7f7c7415a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.047+0000 7f7c799e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c740ca280 0x7f7c7415a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59702/0 (socket says 172.21.15.98:59702) 2023-11-28T17:49:57.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.047+0000 7f7c799e2640 1 -- 172.21.15.98:0/4060362390 learned_addr learned my addr 172.21.15.98:0/4060362390 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:57.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.047+0000 7f7c7a1e3640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c741540d0 0x7f7c7415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.047+0000 7f7c791e1640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740cb070 0x7f7c740c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.047+0000 7f7c799e2640 1 -- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740cb070 msgr2=0x7f7c740c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.047+0000 7f7c799e2640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740cb070 0x7f7c740c3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.047+0000 7f7c799e2640 1 -- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c741540d0 msgr2=0x7f7c7415c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.047+0000 7f7c799e2640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c741540d0 0x7f7c7415c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.047+0000 7f7c799e2640 1 -- 172.21.15.98:0/4060362390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c640094d0 con 0x7f7c740ca280 2023-11-28T17:49:57.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.047+0000 7f7c799e2640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c740ca280 0x7f7c7415a690 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000e870 tx=0x7f7c7000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:57.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.048+0000 7f7c6a7fc640 1 -- 172.21.15.98:0/4060362390 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c700094f0 con 0x7f7c740ca280 2023-11-28T17:49:57.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.048+0000 7f7c6a7fc640 1 -- 172.21.15.98:0/4060362390 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7000c940 con 0x7f7c740ca280 2023-11-28T17:49:57.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.048+0000 7f7c6a7fc640 1 -- 172.21.15.98:0/4060362390 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7000ccf0 con 0x7f7c740ca280 2023-11-28T17:49:57.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.048+0000 7f7c7bc46640 1 -- 172.21.15.98:0/4060362390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c740c5d20 con 0x7f7c740ca280 2023-11-28T17:49:57.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.048+0000 7f7c7bc46640 1 -- 172.21.15.98:0/4060362390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c740c5110 con 0x7f7c740ca280 2023-11-28T17:49:57.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.049+0000 7f7c7bc46640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740c2b80 0x7f7c740c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.049+0000 7f7c7bc46640 1 -- 172.21.15.98:0/4060362390 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c74074110 con 0x7f7c740c2b80 2023-11-28T17:49:57.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.049+0000 7f7c791e1640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740c2b80 0x7f7c740c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.050+0000 7f7c6a7fc640 1 -- 172.21.15.98:0/4060362390 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f7c70013470 con 0x7f7c740ca280 2023-11-28T17:49:57.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.050+0000 7f7c791e1640 1 -- 172.21.15.98:0/4060362390 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c74075db0 con 0x7f7c740c2b80 2023-11-28T17:49:57.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.050+0000 7f7c6a7fc640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7c5806b2a0 0x7f7c5806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.050+0000 7f7c6a7fc640 1 -- 172.21.15.98:0/4060362390 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7c70089d80 con 0x7f7c740ca280 2023-11-28T17:49:57.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.051+0000 7f7c791e1640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740c2b80 0x7f7c740c53f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7c740b5070 tx=0x7f7c640046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:57.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.051+0000 7f7c7a1e3640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7c5806b2a0 0x7f7c5806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.051+0000 7f7c6a7fc640 1 -- 172.21.15.98:0/4060362390 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c64039c30 con 0x7f7c740c2b80 2023-11-28T17:49:57.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.051+0000 7f7c7a1e3640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7c5806b2a0 0x7f7c5806d760 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0012e0 tx=0x7f7c6c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:57.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.052+0000 7f7c6a7fc640 1 -- 172.21.15.98:0/4060362390 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9324 (secure 0 0 0) 0x7f7c64039d90 con 0x7f7c740c2b80 2023-11-28T17:49:57.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.052+0000 7f7c6a7fc640 1 -- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740c2b80 msgr2=0x7f7c740c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.052+0000 7f7c6a7fc640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740c2b80 0x7f7c740c53f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7c740b5070 tx=0x7f7c640046c0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.052+0000 7f7c6a7fc640 1 -- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740c2b80 msgr2=0x7f7c740c53f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:49:57.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.052+0000 7f7c6a7fc640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740c2b80 0x7f7c740c53f0 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7c740b5070 tx=0x7f7c640046c0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.059+0000 7f7c7bc46640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740c37c0 0x7f7c7414f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.059+0000 7f7c7bc46640 1 -- 172.21.15.98:0/4060362390 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7406ae70 con 0x7f7c740c37c0 2023-11-28T17:49:57.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.059+0000 7f7c791e1640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740c37c0 0x7f7c7414f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.060+0000 7f7c791e1640 1 -- 172.21.15.98:0/4060362390 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7c7406ab90 con 0x7f7c740c37c0 2023-11-28T17:49:57.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.060+0000 7f7c791e1640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740c37c0 0x7f7c7414f710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009870 tx=0x7f7c64009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:57.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.061+0000 7f7c6a7fc640 1 -- 172.21.15.98:0/4060362390 <== mon.2 v2:172.21.15.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6404c910 con 0x7f7c740c37c0 2023-11-28T17:49:57.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.061+0000 7f7c6a7fc640 1 -- 172.21.15.98:0/4060362390 <== mon.2 v2:172.21.15.113:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4975 (secure 0 0 0) 0x7f7c6404ca70 con 0x7f7c740c37c0 2023-11-28T17:49:57.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.061+0000 7f7c6a7fc640 1 -- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740c37c0 msgr2=0x7f7c7414f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.061+0000 7f7c6a7fc640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740c37c0 0x7f7c7414f710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009870 tx=0x7f7c64009240 comp rx=0 tx=0).stop 2023-11-28T17:49:57.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.061+0000 7f7c6a7fc640 1 -- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740c37c0 msgr2=0x7f7c7414f710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:49:57.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.061+0000 7f7c6a7fc640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f7c740c37c0 0x7f7c7414f710 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.063 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-11-28T17:49:57.063 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:49:57.063 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:49:57.063 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2023-11-28T17:49:57.064 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 82, 2023-11-28T17:49:57.064 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2023-11-28T17:49:57.064 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2023-11-28T17:49:57.064 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2023-11-28T17:49:57.064 INFO:teuthology.orchestra.run.smithi098.stdout: 2 2023-11-28T17:49:57.064 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:57.064 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2023-11-28T17:49:57.064 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:49:57.065 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2449958758054445060", 2023-11-28T17:49:57.065 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2023-11-28T17:49:57.065 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:49:57.065 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:49:57.065 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:49:57.065 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:49:57.065 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:49:57.065 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:49:57.066 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:49:57.066 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:49:57.066 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:49:57.066 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:49:57.066 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:57.066 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540138322906710015", 2023-11-28T17:49:57.066 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2023-11-28T17:49:57.066 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:49:57.067 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:49:57.067 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:49:57.067 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:49:57.067 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:49:57.067 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:49:57.067 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:49:57.067 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:49:57.067 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:49:57.068 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:49:57.068 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:57.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:57.068 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2023-11-28T17:49:57.068 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2023-11-28T17:49:57.068 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2023-11-28T17:49:57.068 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2023-11-28T17:49:57.068 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2023-11-28T17:49:57.069 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "a317ea31-3452-4bbd-814c-526dd33c0124", 2023-11-28T17:49:57.069 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:49:57.069 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2023-11-28T17:40:09.105954Z", 2023-11-28T17:49:57.069 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 18, 2023-11-28T17:49:57.069 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "reef", 2023-11-28T17:49:57.069 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 1, 2023-11-28T17:49:57.069 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2023-11-28T17:49:57.069 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2023-11-28T17:49:57.070 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2023-11-28T17:49:57.070 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2023-11-28T17:49:57.070 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2023-11-28T17:49:57.070 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2023-11-28T17:49:57.070 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2023-11-28T17:49:57.070 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2023-11-28T17:49:57.070 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2023-11-28T17:49:57.070 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2023-11-28T17:49:57.071 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2023-11-28T17:49:57.071 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2023-11-28T17:49:57.071 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2023-11-28T17:49:57.071 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2023-11-28T17:49:57.071 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2023-11-28T17:49:57.071 INFO:teuthology.orchestra.run.smithi098.stdout: "reef" 2023-11-28T17:49:57.071 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:57.071 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2023-11-28T17:49:57.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:57.072 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2023-11-28T17:49:57.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:57.072 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2023-11-28T17:49:57.072 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2023-11-28T17:49:57.072 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:49:57.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:49:57.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:57.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:49:57.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2023-11-28T17:49:57.073 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:57.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:57.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:57.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:49:57.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2023-11-28T17:49:57.073 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:57.074 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:57.074 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:57.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:57.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2023-11-28T17:49:57.074 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2023-11-28T17:49:57.074 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:49:57.074 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:49:57.074 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:49:57.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:57.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:57.075 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2023-11-28T17:49:57.075 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2023-11-28T17:49:57.075 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:49:57.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:49:57.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:57.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:49:57.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3300", 2023-11-28T17:49:57.076 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:57.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:57.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:57.076 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:49:57.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789", 2023-11-28T17:49:57.076 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:57.076 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:57.077 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:57.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:57.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6789/0", 2023-11-28T17:49:57.077 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6789/0", 2023-11-28T17:49:57.077 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:49:57.077 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:49:57.077 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:49:57.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:57.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:57.078 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2023-11-28T17:49:57.078 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2023-11-28T17:49:57.078 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2023-11-28T17:49:57.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2023-11-28T17:49:57.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:57.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2023-11-28T17:49:57.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:3301", 2023-11-28T17:49:57.079 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:57.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:57.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:57.079 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-11-28T17:49:57.079 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790", 2023-11-28T17:49:57.079 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2023-11-28T17:49:57.079 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:57.079 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:57.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:57.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.113:6790/0", 2023-11-28T17:49:57.080 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.113:6790/0", 2023-11-28T17:49:57.080 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2023-11-28T17:49:57.080 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2023-11-28T17:49:57.080 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2023-11-28T17:49:57.080 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:57.080 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:57.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:57.081 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2023-11-28T17:49:57.081 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2023-11-28T17:49:57.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:57.081 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:57.081 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:57.081 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:49:57.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:57.082 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:57.082 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": [ 2023-11-28T17:49:57.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:57.082 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:57.082 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:57.082 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:49:57.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:57.083 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:57.083 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2023-11-28T17:49:57.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:57.083 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:57.083 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:57.083 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2023-11-28T17:49:57.083 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:57.083 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:57.084 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2023-11-28T17:49:57.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:57.084 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:57.084 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:57.084 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2023-11-28T17:49:57.084 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:57.084 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2023-11-28T17:49:57.085 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2023-11-28T17:49:57.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-11-28T17:49:57.085 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f01cfbffffdffff", 2023-11-28T17:49:57.085 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "luminous", 2023-11-28T17:49:57.085 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2023-11-28T17:49:57.085 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-11-28T17:49:57.085 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-11-28T17:49:57.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-11-28T17:49:57.086 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2023-11-28T17:49:57.086 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-11-28T17:49:57.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.062+0000 7f7c7bc46640 1 -- 172.21.15.98:0/4060362390 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7c5806b2a0 msgr2=0x7f7c5806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.062+0000 7f7c7bc46640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f7c5806b2a0 0x7f7c5806d760 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0012e0 tx=0x7f7c6c009040 comp rx=0 tx=0).stop 2023-11-28T17:49:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.062+0000 7f7c7bc46640 1 -- 172.21.15.98:0/4060362390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c740ca280 msgr2=0x7f7c7415a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.062+0000 7f7c7bc46640 1 --2- 172.21.15.98:0/4060362390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c740ca280 0x7f7c7415a690 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000e870 tx=0x7f7c7000ed30 comp rx=0 tx=0).stop 2023-11-28T17:49:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.062+0000 7f7c7a1e3640 1 -- 172.21.15.98:0/4060362390 reap_dead start 2023-11-28T17:49:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.062+0000 7f7c7a1e3640 1 -- 172.21.15.98:0/4060362390 reap_dead start 2023-11-28T17:49:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.062+0000 7f7c7bc46640 1 -- 172.21.15.98:0/4060362390 shutdown_connections 2023-11-28T17:49:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.062+0000 7f7c7bc46640 1 -- 172.21.15.98:0/4060362390 >> 172.21.15.98:0/4060362390 conn(0x7f7c740cf490 msgr2=0x7f7c740cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.062+0000 7f7c7bc46640 1 -- 172.21.15.98:0/4060362390 shutdown_connections 2023-11-28T17:49:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.062+0000 7f7c7bc46640 1 -- 172.21.15.98:0/4060362390 wait complete. 2023-11-28T17:49:57.089 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2023-11-28T17:49:57.089 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2023-11-28T17:49:57.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2023-11-28T17:49:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.103+0000 7f1a0e30f640 1 Processor -- start 2023-11-28T17:49:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.104+0000 7f1a0e30f640 1 -- start start 2023-11-28T17:49:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.105+0000 7f1a0e30f640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f1a08070a20 0x7f1a08154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.105+0000 7f1a0e30f640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a080c36a0 0x7f1a08155310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.105+0000 7f1a0e30f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a08158640 con 0x7f1a08070640 2023-11-28T17:49:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.105+0000 7f1a0e30f640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a080c7d60 con 0x7f1a08070a20 2023-11-28T17:49:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.105+0000 7f1a0e30f640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080ce300 con 0x7f1a080c36a0 2023-11-28T17:49:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.105+0000 7f1a07fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1a08070640 0x7f1a080c8f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33966/0 (socket says 172.21.15.98:33966) 2023-11-28T17:49:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.105+0000 7f1a07fff640 1 -- 172.21.15.98:0/145730918 learned_addr learned my addr 172.21.15.98:0/145730918 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:49:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.105+0000 7f1a0c8ac640 1 --2- 172.21.15.98:0/145730918 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a080c36a0 0x7f1a08155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.105+0000 7f19fffff640 1 --2- 172.21.15.98:0/145730918 >> v2:172.21.15.113:3300/0 conn(0x7f1a08070a20 0x7f1a08154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.105+0000 7f1a077fe640 1 -- 172.21.15.98:0/145730918 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757803095 0 0) 0x7f1a08158640 con 0x7f1a08070640 2023-11-28T17:49:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a077fe640 1 -- 172.21.15.98:0/145730918 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f4003520 con 0x7f1a08070640 2023-11-28T17:49:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a077fe640 1 -- 172.21.15.98:0/145730918 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2822638470 0 0) 0x7f19f4003520 con 0x7f1a08070640 2023-11-28T17:49:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a077fe640 1 -- 172.21.15.98:0/145730918 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a080c36a0 msgr2=0x7f1a08155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a077fe640 1 --2- 172.21.15.98:0/145730918 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a080c36a0 0x7f1a08155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a077fe640 1 -- 172.21.15.98:0/145730918 >> v2:172.21.15.113:3300/0 conn(0x7f1a08070a20 msgr2=0x7f1a08154dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a077fe640 1 --2- 172.21.15.98:0/145730918 >> v2:172.21.15.113:3300/0 conn(0x7f1a08070a20 0x7f1a08154dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a077fe640 1 -- 172.21.15.98:0/145730918 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a080c7f50 con 0x7f1a08070640 2023-11-28T17:49:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a077fe640 1 -- 172.21.15.98:0/145730918 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f19f8001670 con 0x7f1a08070640 2023-11-28T17:49:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f19fffff640 1 --2- 172.21.15.98:0/145730918 >> v2:172.21.15.113:3300/0 conn(0x7f1a08070a20 0x7f1a08154dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:49:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a077fe640 1 -- 172.21.15.98:0/145730918 >> v1:172.21.15.98:6789/0 conn(0x7f1a08070640 legacy=0x7f1a080c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a077fe640 1 --2- 172.21.15.98:0/145730918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19f4003970 0x7f19f4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a077fe640 1 --2- 172.21.15.98:0/145730918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f40047c0 0x7f19f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a077fe640 1 --2- 172.21.15.98:0/145730918 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a080c36a0 0x7f19f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a077fe640 1 -- 172.21.15.98:0/145730918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080ce300 con 0x7f19f40047c0 2023-11-28T17:49:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a077fe640 1 -- 172.21.15.98:0/145730918 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f400ba40 con 0x7f19f4003970 2023-11-28T17:49:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a077fe640 1 -- 172.21.15.98:0/145730918 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f400bc00 con 0x7f1a080c36a0 2023-11-28T17:49:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f19fffff640 1 --2- 172.21.15.98:0/145730918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f40047c0 0x7f19f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.106+0000 7f1a07fff640 1 --2- 172.21.15.98:0/145730918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19f4003970 0x7f19f4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.107+0000 7f19fffff640 1 -- 172.21.15.98:0/145730918 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a080c36a0 msgr2=0x7f19f4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.107+0000 7f19fffff640 1 --2- 172.21.15.98:0/145730918 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a080c36a0 0x7f19f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.107+0000 7f19fffff640 1 -- 172.21.15.98:0/145730918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19f4003970 msgr2=0x7f19f4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.107+0000 7f19fffff640 1 --2- 172.21.15.98:0/145730918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f19f4003970 0x7f19f4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.107+0000 7f19fffff640 1 -- 172.21.15.98:0/145730918 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f4003520 con 0x7f19f40047c0 2023-11-28T17:49:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.108+0000 7f19fffff640 1 --2- 172.21.15.98:0/145730918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f40047c0 0x7f19f4004c40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f19f0006fd0 tx=0x7f19f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15eff0e01fa271b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.108+0000 7f1a077fe640 1 -- 172.21.15.98:0/145730918 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f0030630 con 0x7f19f40047c0 2023-11-28T17:49:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.108+0000 7f1a077fe640 1 -- 172.21.15.98:0/145730918 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f0030790 con 0x7f19f40047c0 2023-11-28T17:49:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.108+0000 7f1a077fe640 1 -- 172.21.15.98:0/145730918 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f0030b00 con 0x7f19f40047c0 2023-11-28T17:49:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.109+0000 7f1a0e30f640 1 -- 172.21.15.98:0/145730918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f40047c0 msgr2=0x7f19f4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.109+0000 7f1a0e30f640 1 --2- 172.21.15.98:0/145730918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19f40047c0 0x7f19f4004c40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f19f0006fd0 tx=0x7f19f40057f0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.109+0000 7f1a0c8ac640 1 -- 172.21.15.98:0/145730918 reap_dead start 2023-11-28T17:49:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.109+0000 7f1a0e30f640 1 -- 172.21.15.98:0/145730918 shutdown_connections 2023-11-28T17:49:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.109+0000 7f1a0e30f640 1 -- 172.21.15.98:0/145730918 >> 172.21.15.98:0/145730918 conn(0x7f1a0806e620 msgr2=0x7f1a0806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.109+0000 7f1a0e30f640 1 -- 172.21.15.98:0/145730918 shutdown_connections 2023-11-28T17:49:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.109+0000 7f1a0e30f640 1 -- 172.21.15.98:0/145730918 wait complete. 2023-11-28T17:49:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.111+0000 7f1a0e30f640 1 Processor -- start 2023-11-28T17:49:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.111+0000 7f1a0e30f640 1 -- start start 2023-11-28T17:49:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.112+0000 7f1a0e30f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a080cc5b0 0x7f1a081eb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.112+0000 7f1a0e30f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a080cc990 0x7f1a08140b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.112+0000 7f1a0e30f640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a080b13f0 0x7f1a081410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.112+0000 7f1a0e30f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080b1210 con 0x7f1a080cc5b0 2023-11-28T17:49:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.112+0000 7f1a0e30f640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080b0c50 con 0x7f1a080cc990 2023-11-28T17:49:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.112+0000 7f1a0e30f640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a080b0f30 con 0x7f1a080b13f0 2023-11-28T17:49:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.112+0000 7f1a07fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a080cc5b0 0x7f1a081eb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.112+0000 7f1a07fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a080cc5b0 0x7f1a081eb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59714/0 (socket says 172.21.15.98:59714) 2023-11-28T17:49:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.112+0000 7f1a07fff640 1 -- 172.21.15.98:0/1488452588 learned_addr learned my addr 172.21.15.98:0/1488452588 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.112+0000 7f1a0c8ac640 1 --2- 172.21.15.98:0/1488452588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a080b13f0 0x7f1a081410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.112+0000 7f19fffff640 1 --2- 172.21.15.98:0/1488452588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a080cc990 0x7f1a08140b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.112+0000 7f1a07fff640 1 -- 172.21.15.98:0/1488452588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a080b13f0 msgr2=0x7f1a081410d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.112+0000 7f1a07fff640 1 --2- 172.21.15.98:0/1488452588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a080b13f0 0x7f1a081410d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.112+0000 7f1a07fff640 1 -- 172.21.15.98:0/1488452588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a080cc990 msgr2=0x7f1a08140b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.113+0000 7f1a07fff640 1 --2- 172.21.15.98:0/1488452588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a080cc990 0x7f1a08140b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.113+0000 7f1a07fff640 1 -- 172.21.15.98:0/1488452588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a080c6b40 con 0x7f1a080cc5b0 2023-11-28T17:49:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.113+0000 7f1a07fff640 1 --2- 172.21.15.98:0/1488452588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a080cc5b0 0x7f1a081eb460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f19f8005fd0 tx=0x7f19f800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.113+0000 7f1a057fa640 1 -- 172.21.15.98:0/1488452588 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f800c8d0 con 0x7f1a080cc5b0 2023-11-28T17:49:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.113+0000 7f1a057fa640 1 -- 172.21.15.98:0/1488452588 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f800ca50 con 0x7f1a080cc5b0 2023-11-28T17:49:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.113+0000 7f1a057fa640 1 -- 172.21.15.98:0/1488452588 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f800ce00 con 0x7f1a080cc5b0 2023-11-28T17:49:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.113+0000 7f1a0e30f640 1 -- 172.21.15.98:0/1488452588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a080c6e20 con 0x7f1a080cc5b0 2023-11-28T17:49:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.113+0000 7f1a0e30f640 1 -- 172.21.15.98:0/1488452588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a080c7620 con 0x7f1a080cc5b0 2023-11-28T17:49:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.115+0000 7f1a0e30f640 1 -- 172.21.15.98:0/1488452588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a080c7340 con 0x7f1a080cc5b0 2023-11-28T17:49:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.115+0000 7f1a057fa640 1 -- 172.21.15.98:0/1488452588 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f19f801d460 con 0x7f1a080cc5b0 2023-11-28T17:49:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.116+0000 7f1a057fa640 1 --2- 172.21.15.98:0/1488452588 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19e806b2d0 0x7f19e806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.116+0000 7f1a057fa640 1 -- 172.21.15.98:0/1488452588 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f19f8089d90 con 0x7f1a080cc5b0 2023-11-28T17:49:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.117+0000 7f19fffff640 1 --2- 172.21.15.98:0/1488452588 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19e806b2d0 0x7f19e806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.117+0000 7f19fffff640 1 --2- 172.21.15.98:0/1488452588 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19e806b2d0 0x7f19e806d790 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f19f0036000 tx=0x7f19f0031000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.122+0000 7f1a057fa640 1 -- 172.21.15.98:0/1488452588 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a080c7340 con 0x7f1a080cc5b0 2023-11-28T17:49:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.270+0000 7f5312b5c640 1 Processor -- start 2023-11-28T17:49:57.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.270+0000 7f5312b5c640 1 -- start start 2023-11-28T17:49:57.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.271+0000 7f5312b5c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f530c1519b0 0x7f530c070210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.271+0000 7f5312b5c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f530c151d90 0x7f530c070750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.271+0000 7f5312b5c640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c0c8ed0 con 0x7f530c0caa40 2023-11-28T17:49:57.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.271+0000 7f5312b5c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f530c0c28c0 con 0x7f530c1519b0 2023-11-28T17:49:57.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.271+0000 7f5312b5c640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f530c0c91b0 con 0x7f530c151d90 2023-11-28T17:49:57.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.271+0000 7f5311b5a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f530c1519b0 0x7f530c070210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.272+0000 7f5311b5a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f530c1519b0 0x7f530c070210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59730/0 (socket says 172.21.15.98:59730) 2023-11-28T17:49:57.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.272+0000 7f5311b5a640 1 -- 172.21.15.98:0/1598449801 learned_addr learned my addr 172.21.15.98:0/1598449801 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:57.273 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.272+0000 7f5311359640 1 --2- 172.21.15.98:0/1598449801 >> v2:172.21.15.113:3300/0 conn(0x7f530c151d90 0x7f530c070750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.272+0000 7f5311b5a640 1 -- 172.21.15.98:0/1598449801 >> v2:172.21.15.113:3300/0 conn(0x7f530c151d90 msgr2=0x7f530c070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:57.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.272+0000 7f5311b5a640 1 --2- 172.21.15.98:0/1598449801 >> v2:172.21.15.113:3300/0 conn(0x7f530c151d90 0x7f530c070750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.272+0000 7f5311b5a640 1 -- 172.21.15.98:0/1598449801 >> v1:172.21.15.113:6789/0 conn(0x7f530c0caa40 legacy=0x7f530c157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:57.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.272+0000 7f5311b5a640 1 -- 172.21.15.98:0/1598449801 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c0c9490 con 0x7f530c1519b0 2023-11-28T17:49:57.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.273+0000 7f5311b5a640 1 --2- 172.21.15.98:0/1598449801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1519b0 0x7f530c070210 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f530c0c4650 tx=0x7f530802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f650237e3557ccdc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:57.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.273+0000 7f5310b58640 1 -- 172.21.15.98:0/1598449801 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308031870 con 0x7f530c1519b0 2023-11-28T17:49:57.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.273+0000 7f5310b58640 1 -- 172.21.15.98:0/1598449801 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53080319d0 con 0x7f530c1519b0 2023-11-28T17:49:57.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.273+0000 7f5310b58640 1 -- 172.21.15.98:0/1598449801 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308031d60 con 0x7f530c1519b0 2023-11-28T17:49:57.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.274+0000 7f5312b5c640 1 -- 172.21.15.98:0/1598449801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1519b0 msgr2=0x7f530c070210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.274+0000 7f5312b5c640 1 --2- 172.21.15.98:0/1598449801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1519b0 0x7f530c070210 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f530c0c4650 tx=0x7f530802b910 comp rx=0 tx=0).stop 2023-11-28T17:49:57.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.274+0000 7f5312b5c640 1 -- 172.21.15.98:0/1598449801 shutdown_connections 2023-11-28T17:49:57.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.274+0000 7f5312b5c640 1 --2- 172.21.15.98:0/1598449801 >> v2:172.21.15.113:3300/0 conn(0x7f530c151d90 0x7f530c070750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.274+0000 7f5312b5c640 1 --2- 172.21.15.98:0/1598449801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1519b0 0x7f530c070210 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.274+0000 7f5312b5c640 1 -- 172.21.15.98:0/1598449801 >> 172.21.15.98:0/1598449801 conn(0x7f530c0722b0 msgr2=0x7f530c072690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:57.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.274+0000 7f5312b5c640 1 -- 172.21.15.98:0/1598449801 shutdown_connections 2023-11-28T17:49:57.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.274+0000 7f5312b5c640 1 -- 172.21.15.98:0/1598449801 wait complete. 2023-11-28T17:49:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.276+0000 7f5312b5c640 1 Processor -- start 2023-11-28T17:49:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.276+0000 7f5312b5c640 1 -- start start 2023-11-28T17:49:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.277+0000 7f5312b5c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f530c0caa40 0x7f530c0b2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.277+0000 7f5312b5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1519b0 0x7f530c15a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.277+0000 7f5312b5c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f530c151d90 0x7f530c15abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.277+0000 7f5312b5c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c1efa60 con 0x7f530c1519b0 2023-11-28T17:49:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.277+0000 7f5312b5c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c1eff90 con 0x7f530c151d90 2023-11-28T17:49:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.277+0000 7f5312b5c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530c159f00 con 0x7f530c0caa40 2023-11-28T17:49:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.277+0000 7f5311359640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1519b0 0x7f530c15a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.277+0000 7f5311359640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1519b0 0x7f530c15a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59744/0 (socket says 172.21.15.98:59744) 2023-11-28T17:49:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.277+0000 7f5311359640 1 -- 172.21.15.98:0/684440051 learned_addr learned my addr 172.21.15.98:0/684440051 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.277+0000 7f531235b640 1 --2- 172.21.15.98:0/684440051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f530c151d90 0x7f530c15abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.278+0000 7f5311359640 1 -- 172.21.15.98:0/684440051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f530c0caa40 msgr2=0x7f530c0b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.278+0000 7f5311359640 1 --2- 172.21.15.98:0/684440051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f530c0caa40 0x7f530c0b2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.278+0000 7f5311359640 1 -- 172.21.15.98:0/684440051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f530c151d90 msgr2=0x7f530c15abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.278+0000 7f5311359640 1 --2- 172.21.15.98:0/684440051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f530c151d90 0x7f530c15abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.278+0000 7f5311359640 1 -- 172.21.15.98:0/684440051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53080094d0 con 0x7f530c1519b0 2023-11-28T17:49:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.278+0000 7f5311359640 1 --2- 172.21.15.98:0/684440051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1519b0 0x7f530c15a670 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00d5c0 tx=0x7f52fc00da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.278+0000 7f53027fc640 1 -- 172.21.15.98:0/684440051 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52fc0125f0 con 0x7f530c1519b0 2023-11-28T17:49:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.278+0000 7f53027fc640 1 -- 172.21.15.98:0/684440051 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52fc012750 con 0x7f530c1519b0 2023-11-28T17:49:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.278+0000 7f53027fc640 1 -- 172.21.15.98:0/684440051 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52fc012b00 con 0x7f530c1519b0 2023-11-28T17:49:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.278+0000 7f5312b5c640 1 -- 172.21.15.98:0/684440051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530c14f840 con 0x7f530c1519b0 2023-11-28T17:49:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.278+0000 7f5312b5c640 1 -- 172.21.15.98:0/684440051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530c0cb220 con 0x7f530c1519b0 2023-11-28T17:49:57.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.280+0000 7f5312b5c640 1 -- 172.21.15.98:0/684440051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530c074db0 con 0x7f530c1519b0 2023-11-28T17:49:57.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.280+0000 7f53027fc640 1 -- 172.21.15.98:0/684440051 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f52fc012c60 con 0x7f530c1519b0 2023-11-28T17:49:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.281+0000 7f53027fc640 1 --2- 172.21.15.98:0/684440051 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f52f006b2a0 0x7f52f006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.281+0000 7f53027fc640 1 -- 172.21.15.98:0/684440051 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f52fc08a1e0 con 0x7f530c1519b0 2023-11-28T17:49:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.281+0000 7f5311b5a640 1 --2- 172.21.15.98:0/684440051 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f52f006b2a0 0x7f52f006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.282+0000 7f5311b5a640 1 --2- 172.21.15.98:0/684440051 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f52f006b2a0 0x7f52f006d760 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5308036000 tx=0x7f5308030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.285+0000 7f53027fc640 1 -- 172.21.15.98:0/684440051 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52fc054f30 con 0x7f530c1519b0 2023-11-28T17:49:57.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.317+0000 7f1a0e30f640 1 -- 172.21.15.98:0/1488452588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1a080c7340 con 0x7f1a080cc5b0 2023-11-28T17:49:57.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.318+0000 7f1a057fa640 1 -- 172.21.15.98:0/1488452588 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1a080c7340 con 0x7f1a080cc5b0 2023-11-28T17:49:57.319 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-11-28T17:49:57.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.320+0000 7f1a0e30f640 1 -- 172.21.15.98:0/1488452588 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19e806b2d0 msgr2=0x7f19e806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.320+0000 7f1a0e30f640 1 --2- 172.21.15.98:0/1488452588 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19e806b2d0 0x7f19e806d790 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f19f0036000 tx=0x7f19f0031000 comp rx=0 tx=0).stop 2023-11-28T17:49:57.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.320+0000 7f1a0e30f640 1 -- 172.21.15.98:0/1488452588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a080cc5b0 msgr2=0x7f1a081eb460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.320+0000 7f1a0e30f640 1 --2- 172.21.15.98:0/1488452588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a080cc5b0 0x7f1a081eb460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f19f8005fd0 tx=0x7f19f800ed50 comp rx=0 tx=0).stop 2023-11-28T17:49:57.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.320+0000 7f1a0e30f640 1 -- 172.21.15.98:0/1488452588 shutdown_connections 2023-11-28T17:49:57.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.320+0000 7f1a0e30f640 1 --2- 172.21.15.98:0/1488452588 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f1a080b13f0 0x7f1a081410d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.320+0000 7f1a0e30f640 1 --2- 172.21.15.98:0/1488452588 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f19e806b2d0 0x7f19e806d790 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.320+0000 7f1a0e30f640 1 --2- 172.21.15.98:0/1488452588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a080cc990 0x7f1a08140b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.320+0000 7f1a0e30f640 1 --2- 172.21.15.98:0/1488452588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a080cc5b0 0x7f1a081eb460 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.320+0000 7f1a0e30f640 1 -- 172.21.15.98:0/1488452588 >> 172.21.15.98:0/1488452588 conn(0x7f1a0806e620 msgr2=0x7f1a08150370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:57.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.321+0000 7f1a0e30f640 1 -- 172.21.15.98:0/1488452588 shutdown_connections 2023-11-28T17:49:57.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.321+0000 7f1a0e30f640 1 -- 172.21.15.98:0/1488452588 wait complete. 2023-11-28T17:49:57.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:49:57.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.547+0000 7f5312b5c640 1 -- 172.21.15.98:0/684440051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f530c0afa10 con 0x7f530c1519b0 2023-11-28T17:49:57.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.549+0000 7f53027fc640 1 -- 172.21.15.98:0/684440051 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f52fc058550 con 0x7f530c1519b0 2023-11-28T17:49:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.552+0000 7f5312b5c640 1 -- 172.21.15.98:0/684440051 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f52f006b2a0 msgr2=0x7f52f006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.552+0000 7f5312b5c640 1 --2- 172.21.15.98:0/684440051 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f52f006b2a0 0x7f52f006d760 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5308036000 tx=0x7f5308030000 comp rx=0 tx=0).stop 2023-11-28T17:49:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.552+0000 7f5312b5c640 1 -- 172.21.15.98:0/684440051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1519b0 msgr2=0x7f530c15a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.552+0000 7f5312b5c640 1 --2- 172.21.15.98:0/684440051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1519b0 0x7f530c15a670 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00d5c0 tx=0x7f52fc00da80 comp rx=0 tx=0).stop 2023-11-28T17:49:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.552+0000 7f5312b5c640 1 -- 172.21.15.98:0/684440051 shutdown_connections 2023-11-28T17:49:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.552+0000 7f5312b5c640 1 --2- 172.21.15.98:0/684440051 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f52f006b2a0 0x7f52f006d760 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.552+0000 7f5312b5c640 1 --2- 172.21.15.98:0/684440051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f530c151d90 0x7f530c15abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.552+0000 7f5312b5c640 1 --2- 172.21.15.98:0/684440051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f530c1519b0 0x7f530c15a670 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.552+0000 7f5312b5c640 1 --2- 172.21.15.98:0/684440051 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f530c0caa40 0x7f530c0b2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.552+0000 7f5312b5c640 1 -- 172.21.15.98:0/684440051 >> 172.21.15.98:0/684440051 conn(0x7f530c0722b0 msgr2=0x7f530c156c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.552+0000 7f5312b5c640 1 -- 172.21.15.98:0/684440051 shutdown_connections 2023-11-28T17:49:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.552+0000 7f5312b5c640 1 -- 172.21.15.98:0/684440051 wait complete. 2023-11-28T17:49:57.571 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2023-11-28T17:49:57.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.580+0000 7fbbbd497640 1 Processor -- start 2023-11-28T17:49:57.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.580+0000 7fbbbd497640 1 -- start start 2023-11-28T17:49:57.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.581+0000 7fbbbd497640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fbbb80c4760 0x7fbbb80c2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.581+0000 7fbbbd497640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbbb815a0f0 0x7fbbb815a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.581+0000 7fbbbd497640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb8156980 con 0x7fbbb80c4380 2023-11-28T17:49:57.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.581+0000 7fbbbd497640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbb80c2a60 con 0x7fbbb815a0f0 2023-11-28T17:49:57.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.581+0000 7fbbbd497640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbb815aa60 con 0x7fbbb80c4760 2023-11-28T17:49:57.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.582+0000 7fbbb77fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbbb815a0f0 0x7fbbb815a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.582+0000 7fbbb77fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbbb815a0f0 0x7fbbb815a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59756/0 (socket says 172.21.15.98:59756) 2023-11-28T17:49:57.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.582+0000 7fbbb77fe640 1 -- 172.21.15.98:0/1951226817 learned_addr learned my addr 172.21.15.98:0/1951226817 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:57.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.582+0000 7fbbb77fe640 1 -- 172.21.15.98:0/1951226817 >> v2:172.21.15.113:3300/0 conn(0x7fbbb80c4760 msgr2=0x7fbbb80c2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:57.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.582+0000 7fbbb77fe640 1 --2- 172.21.15.98:0/1951226817 >> v2:172.21.15.113:3300/0 conn(0x7fbbb80c4760 0x7fbbb80c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.582+0000 7fbbb77fe640 1 -- 172.21.15.98:0/1951226817 >> v1:172.21.15.113:6789/0 conn(0x7fbbb80c4380 legacy=0x7fbbb80c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:57.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.582+0000 7fbbb77fe640 1 -- 172.21.15.98:0/1951226817 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb8156c40 con 0x7fbbb815a0f0 2023-11-28T17:49:57.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.583+0000 7fbbb77fe640 1 --2- 172.21.15.98:0/1951226817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb815a0f0 0x7fbbb815a520 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbbb806b9f0 tx=0x7fbbb002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f888fb3df3b625 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:57.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.583+0000 7fbbb67fc640 1 -- 172.21.15.98:0/1951226817 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb002cb20 con 0x7fbbb815a0f0 2023-11-28T17:49:57.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.583+0000 7fbbb67fc640 1 -- 172.21.15.98:0/1951226817 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb002cc80 con 0x7fbbb815a0f0 2023-11-28T17:49:57.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.583+0000 7fbbb67fc640 1 -- 172.21.15.98:0/1951226817 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb00343f0 con 0x7fbbb815a0f0 2023-11-28T17:49:57.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.584+0000 7fbbbd497640 1 -- 172.21.15.98:0/1951226817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb815a0f0 msgr2=0x7fbbb815a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.584+0000 7fbbbd497640 1 --2- 172.21.15.98:0/1951226817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb815a0f0 0x7fbbb815a520 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbbb806b9f0 tx=0x7fbbb002bf80 comp rx=0 tx=0).stop 2023-11-28T17:49:57.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.584+0000 7fbbbd497640 1 -- 172.21.15.98:0/1951226817 shutdown_connections 2023-11-28T17:49:57.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.584+0000 7fbbbd497640 1 --2- 172.21.15.98:0/1951226817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb815a0f0 0x7fbbb815a520 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.584+0000 7fbbbd497640 1 --2- 172.21.15.98:0/1951226817 >> v2:172.21.15.113:3300/0 conn(0x7fbbb80c4760 0x7fbbb80c2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.584+0000 7fbbbd497640 1 -- 172.21.15.98:0/1951226817 >> 172.21.15.98:0/1951226817 conn(0x7fbbb806c0f0 msgr2=0x7fbbb806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:57.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.584+0000 7fbbbd497640 1 -- 172.21.15.98:0/1951226817 shutdown_connections 2023-11-28T17:49:57.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.584+0000 7fbbbd497640 1 -- 172.21.15.98:0/1951226817 wait complete. 2023-11-28T17:49:57.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.586+0000 7fbbbd497640 1 Processor -- start 2023-11-28T17:49:57.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.586+0000 7fbbbd497640 1 -- start start 2023-11-28T17:49:57.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.587+0000 7fbbbd497640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb80c4380 0x7fbbb814b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.587+0000 7fbbbd497640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbb80c4760 0x7fbbb81ef670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.587+0000 7fbbbd497640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbbb815a0f0 0x7fbbb8155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.587+0000 7fbbbd497640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb81556d0 con 0x7fbbb80c4380 2023-11-28T17:49:57.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.587+0000 7fbbbd497640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb8154d00 con 0x7fbbb80c4760 2023-11-28T17:49:57.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.587+0000 7fbbbd497640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb81546f0 con 0x7fbbb815a0f0 2023-11-28T17:49:57.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.587+0000 7fbbb6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb80c4380 0x7fbbb814b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.587+0000 7fbbb6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb80c4380 0x7fbbb814b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59772/0 (socket says 172.21.15.98:59772) 2023-11-28T17:49:57.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.587+0000 7fbbb6ffd640 1 -- 172.21.15.98:0/4202298065 learned_addr learned my addr 172.21.15.98:0/4202298065 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:57.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.587+0000 7fbbaffff640 1 --2- 172.21.15.98:0/4202298065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbb80c4760 0x7fbbb81ef670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.588+0000 7fbbb6ffd640 1 -- 172.21.15.98:0/4202298065 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbbb815a0f0 msgr2=0x7fbbb8155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.588+0000 7fbbb6ffd640 1 --2- 172.21.15.98:0/4202298065 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbbb815a0f0 0x7fbbb8155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.588+0000 7fbbb6ffd640 1 -- 172.21.15.98:0/4202298065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbb80c4760 msgr2=0x7fbbb81ef670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.588+0000 7fbbb6ffd640 1 --2- 172.21.15.98:0/4202298065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbb80c4760 0x7fbbb81ef670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.588+0000 7fbbb6ffd640 1 -- 172.21.15.98:0/4202298065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb0009b30 con 0x7fbbb80c4380 2023-11-28T17:49:57.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.588+0000 7fbbb6ffd640 1 --2- 172.21.15.98:0/4202298065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb80c4380 0x7fbbb814b140 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbba800c410 tx=0x7fbba800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:57.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.588+0000 7fbbaf7fe640 1 -- 172.21.15.98:0/4202298065 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba80035f0 con 0x7fbbb80c4380 2023-11-28T17:49:57.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.588+0000 7fbbaf7fe640 1 -- 172.21.15.98:0/4202298065 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba8003750 con 0x7fbbb80c4380 2023-11-28T17:49:57.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.588+0000 7fbbaf7fe640 1 -- 172.21.15.98:0/4202298065 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba8003b00 con 0x7fbbb80c4380 2023-11-28T17:49:57.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.588+0000 7fbbbd497640 1 -- 172.21.15.98:0/4202298065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb8156160 con 0x7fbbb80c4380 2023-11-28T17:49:57.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.588+0000 7fbbbd497640 1 -- 172.21.15.98:0/4202298065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb8154990 con 0x7fbbb80c4380 2023-11-28T17:49:57.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.590+0000 7fbbbd497640 1 -- 172.21.15.98:0/4202298065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb8154b50 con 0x7fbbb80c4380 2023-11-28T17:49:57.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.590+0000 7fbbaf7fe640 1 -- 172.21.15.98:0/4202298065 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fbba8003c60 con 0x7fbbb80c4380 2023-11-28T17:49:57.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.591+0000 7fbbaf7fe640 1 --2- 172.21.15.98:0/4202298065 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbb9806b2a0 0x7fbb9806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:57.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.591+0000 7fbbaf7fe640 1 -- 172.21.15.98:0/4202298065 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbba8088e40 con 0x7fbbb80c4380 2023-11-28T17:49:57.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.591+0000 7fbbaffff640 1 --2- 172.21.15.98:0/4202298065 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbb9806b2a0 0x7fbb9806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:57.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.592+0000 7fbbaffff640 1 --2- 172.21.15.98:0/4202298065 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbb9806b2a0 0x7fbb9806d760 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80c27c0 tx=0x7fbba0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:57.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.596+0000 7fbbaf7fe640 1 -- 172.21.15.98:0/4202298065 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbba8053c30 con 0x7fbbb80c4380 2023-11-28T17:49:57.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.867+0000 7fbbbd497640 1 -- 172.21.15.98:0/4202298065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbbb8074f60 con 0x7fbbb80c4380 2023-11-28T17:49:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.868+0000 7fbbaf7fe640 1 -- 172.21.15.98:0/4202298065 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbba8057250 con 0x7fbbb80c4380 2023-11-28T17:49:57.870 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-11-28T17:49:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.871+0000 7fbbbd497640 1 -- 172.21.15.98:0/4202298065 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbb9806b2a0 msgr2=0x7fbb9806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.871+0000 7fbbbd497640 1 --2- 172.21.15.98:0/4202298065 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbb9806b2a0 0x7fbb9806d760 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80c27c0 tx=0x7fbba0008040 comp rx=0 tx=0).stop 2023-11-28T17:49:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.871+0000 7fbbbd497640 1 -- 172.21.15.98:0/4202298065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb80c4380 msgr2=0x7fbbb814b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.871+0000 7fbbbd497640 1 --2- 172.21.15.98:0/4202298065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb80c4380 0x7fbbb814b140 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbba800c410 tx=0x7fbba800c8d0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.871+0000 7fbbbd497640 1 -- 172.21.15.98:0/4202298065 shutdown_connections 2023-11-28T17:49:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.871+0000 7fbbbd497640 1 --2- 172.21.15.98:0/4202298065 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fbbb815a0f0 0x7fbbb8155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.871+0000 7fbbbd497640 1 --2- 172.21.15.98:0/4202298065 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fbb9806b2a0 0x7fbb9806d760 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.871+0000 7fbbbd497640 1 --2- 172.21.15.98:0/4202298065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbbb80c4760 0x7fbbb81ef670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.871+0000 7fbbbd497640 1 --2- 172.21.15.98:0/4202298065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbb80c4380 0x7fbbb814b140 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.871+0000 7fbbbd497640 1 -- 172.21.15.98:0/4202298065 >> 172.21.15.98:0/4202298065 conn(0x7fbbb806c0f0 msgr2=0x7fbbb806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.871+0000 7fbbbd497640 1 -- 172.21.15.98:0/4202298065 shutdown_connections 2023-11-28T17:49:57.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:57.871+0000 7fbbbd497640 1 -- 172.21.15.98:0/4202298065 wait complete. 2023-11-28T17:49:57.891 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:49:57.891 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:49:57.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:49:58.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.177+0000 7fde7e62e640 1 Processor -- start 2023-11-28T17:49:58.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.177+0000 7fde7e62e640 1 -- start start 2023-11-28T17:49:58.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.178+0000 7fde7e62e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fde78150ac0 0x7fde780716e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:58.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.178+0000 7fde7e62e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde7806b1e0 0x7fde78071c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:58.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.178+0000 7fde7e62e640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde78150ea0 con 0x7fde780c71f0 2023-11-28T17:49:58.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.178+0000 7fde7e62e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7815ad90 con 0x7fde78150ac0 2023-11-28T17:49:58.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.178+0000 7fde7e62e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde781567a0 con 0x7fde7806b1e0 2023-11-28T17:49:58.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.179+0000 7fde7d62c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fde78150ac0 0x7fde780716e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:58.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.179+0000 7fde7d62c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fde78150ac0 0x7fde780716e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59776/0 (socket says 172.21.15.98:59776) 2023-11-28T17:49:58.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.179+0000 7fde7d62c640 1 -- 172.21.15.98:0/1776500488 learned_addr learned my addr 172.21.15.98:0/1776500488 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:58.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.179+0000 7fde7ce2b640 1 --2- 172.21.15.98:0/1776500488 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde7806b1e0 0x7fde78071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:58.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.179+0000 7fde7d62c640 1 -- 172.21.15.98:0/1776500488 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde7806b1e0 msgr2=0x7fde78071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:58.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.179+0000 7fde7d62c640 1 --2- 172.21.15.98:0/1776500488 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde7806b1e0 0x7fde78071c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.179+0000 7fde7d62c640 1 -- 172.21.15.98:0/1776500488 >> v1:172.21.15.113:6789/0 conn(0x7fde780c71f0 legacy=0x7fde78156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:58.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.179+0000 7fde7d62c640 1 -- 172.21.15.98:0/1776500488 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde78159f30 con 0x7fde78150ac0 2023-11-28T17:49:58.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.180+0000 7fde7d62c640 1 --2- 172.21.15.98:0/1776500488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde78150ac0 0x7fde780716e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fde74009870 tx=0x7fde7402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad928f302bbe516 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:58.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.180+0000 7fde6ffff640 1 -- 172.21.15.98:0/1776500488 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74030840 con 0x7fde78150ac0 2023-11-28T17:49:58.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.180+0000 7fde6ffff640 1 -- 172.21.15.98:0/1776500488 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde740309a0 con 0x7fde78150ac0 2023-11-28T17:49:58.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.180+0000 7fde6ffff640 1 -- 172.21.15.98:0/1776500488 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74030d30 con 0x7fde78150ac0 2023-11-28T17:49:58.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.181+0000 7fde7e62e640 1 -- 172.21.15.98:0/1776500488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde78150ac0 msgr2=0x7fde780716e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:58.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.181+0000 7fde7e62e640 1 --2- 172.21.15.98:0/1776500488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde78150ac0 0x7fde780716e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fde74009870 tx=0x7fde7402b980 comp rx=0 tx=0).stop 2023-11-28T17:49:58.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.181+0000 7fde7e62e640 1 -- 172.21.15.98:0/1776500488 shutdown_connections 2023-11-28T17:49:58.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.181+0000 7fde7e62e640 1 --2- 172.21.15.98:0/1776500488 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde7806b1e0 0x7fde78071c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.181+0000 7fde7e62e640 1 --2- 172.21.15.98:0/1776500488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde78150ac0 0x7fde780716e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.181+0000 7fde7e62e640 1 -- 172.21.15.98:0/1776500488 >> 172.21.15.98:0/1776500488 conn(0x7fde78066230 msgr2=0x7fde7806a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:58.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.181+0000 7fde7e62e640 1 -- 172.21.15.98:0/1776500488 shutdown_connections 2023-11-28T17:49:58.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.181+0000 7fde7e62e640 1 -- 172.21.15.98:0/1776500488 wait complete. 2023-11-28T17:49:58.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.183+0000 7fde7e62e640 1 Processor -- start 2023-11-28T17:49:58.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.183+0000 7fde7e62e640 1 -- start start 2023-11-28T17:49:58.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7e62e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde7806b1e0 0x7fde7814f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:58.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7e62e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde780c71f0 0x7fde781ef3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:58.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7e62e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde78150ac0 0x7fde781ef900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:58.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7e62e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7814ddd0 con 0x7fde780c71f0 2023-11-28T17:49:58.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7e62e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde781f3870 con 0x7fde7806b1e0 2023-11-28T17:49:58.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7e62e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7813eed0 con 0x7fde78150ac0 2023-11-28T17:49:58.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7ce2b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde780c71f0 0x7fde781ef3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:58.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7ce2b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde780c71f0 0x7fde781ef3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59786/0 (socket says 172.21.15.98:59786) 2023-11-28T17:49:58.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7ce2b640 1 -- 172.21.15.98:0/174465499 learned_addr learned my addr 172.21.15.98:0/174465499 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:58.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7d62c640 1 --2- 172.21.15.98:0/174465499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde7806b1e0 0x7fde7814f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:58.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7de2d640 1 --2- 172.21.15.98:0/174465499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde78150ac0 0x7fde781ef900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:58.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7ce2b640 1 -- 172.21.15.98:0/174465499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde78150ac0 msgr2=0x7fde781ef900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:58.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7ce2b640 1 --2- 172.21.15.98:0/174465499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde78150ac0 0x7fde781ef900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7ce2b640 1 -- 172.21.15.98:0/174465499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde7806b1e0 msgr2=0x7fde7814f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:58.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7ce2b640 1 --2- 172.21.15.98:0/174465499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde7806b1e0 0x7fde7814f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.184+0000 7fde7ce2b640 1 -- 172.21.15.98:0/174465499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde740094d0 con 0x7fde780c71f0 2023-11-28T17:49:58.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.185+0000 7fde7ce2b640 1 --2- 172.21.15.98:0/174465499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde780c71f0 0x7fde781ef3c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fde6800c7f0 tx=0x7fde6800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:58.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.185+0000 7fde6dffb640 1 -- 172.21.15.98:0/174465499 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde68004220 con 0x7fde780c71f0 2023-11-28T17:49:58.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.185+0000 7fde6dffb640 1 -- 172.21.15.98:0/174465499 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde68004380 con 0x7fde780c71f0 2023-11-28T17:49:58.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.185+0000 7fde6dffb640 1 -- 172.21.15.98:0/174465499 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde68007860 con 0x7fde780c71f0 2023-11-28T17:49:58.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.186+0000 7fde7e62e640 1 -- 172.21.15.98:0/174465499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde781f2ab0 con 0x7fde780c71f0 2023-11-28T17:49:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.186+0000 7fde7e62e640 1 -- 172.21.15.98:0/174465499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde781f3070 con 0x7fde780c71f0 2023-11-28T17:49:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.187+0000 7fde7e62e640 1 -- 172.21.15.98:0/174465499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde78074db0 con 0x7fde780c71f0 2023-11-28T17:49:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.187+0000 7fde6dffb640 1 -- 172.21.15.98:0/174465499 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fde68007bd0 con 0x7fde780c71f0 2023-11-28T17:49:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.188+0000 7fde6dffb640 1 --2- 172.21.15.98:0/174465499 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fde5c06b2d0 0x7fde5c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.188+0000 7fde6dffb640 1 -- 172.21.15.98:0/174465499 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fde68089b80 con 0x7fde780c71f0 2023-11-28T17:49:58.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.188+0000 7fde7d62c640 1 --2- 172.21.15.98:0/174465499 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fde5c06b2d0 0x7fde5c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:58.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.189+0000 7fde7d62c640 1 --2- 172.21.15.98:0/174465499 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fde5c06b2d0 0x7fde5c06d790 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fde74036000 tx=0x7fde7403b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:58.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.193+0000 7fde6dffb640 1 -- 172.21.15.98:0/174465499 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde68054970 con 0x7fde780c71f0 2023-11-28T17:49:58.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.393+0000 7fde7e62e640 1 -- 172.21.15.98:0/174465499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde780c6fd0 con 0x7fde780c71f0 2023-11-28T17:49:58.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.401+0000 7fde6dffb640 1 -- 172.21.15.98:0/174465499 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16766 (secure 0 0 0) 0x7fde68057f90 con 0x7fde780c71f0 2023-11-28T17:49:58.402 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-11-28T17:49:58.402 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"fsid":"a317ea31-3452-4bbd-814c-526dd33c0124","created":"2023-11-28T17:40:57.434851+0000","modified":"2023-11-28T17:42:18.305147+0000","last_up_change":"2023-11-28T17:41:09.556993+0000","last_in_change":"2023-11-28T17:41:01.035090+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:41:10.303280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:41:14.890871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:41:15.704103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"189b7858-08c8-45d8-8834-fa4a3620bbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6810","nonce":2188704385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6813","nonce":2188704385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6819","nonce":2188704385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2188704385},{"type":"v1","addr":"172.21.15.98:6816","nonce":2188704385}]},"public_addr":"172.21.15.98:6810/2188704385","cluster_addr":"172.21.15.98:6813/2188704385","heartbeat_back_addr":"172.21.15.98:6819/2188704385","heartbeat_front_addr":"172.21.15.98:6816/2188704385","state":["exists","up"]},{"osd":1,"uuid":"f5a771a0-ab90-45ff-80f9-f62f0ef1a954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6801","nonce":3918226860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6803","nonce":3918226860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6807","nonce":3918226860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":3918226860},{"type":"v1","addr":"172.21.15.98:6805","nonce":3918226860}]},"public_addr":"172.21.15.98:6801/3918226860","cluster_addr":"172.21.15.98:6803/3918226860","heartbeat_back_addr":"172.21.15.98:6807/3918226860","heartbeat_front_addr":"172.21.15.98:6805/3918226860","state":["exists","up"]},{"osd":2,"uuid":"ea38def8-88ca-44ee-8e2e-b90920eec720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6809","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6812","nonce":2691532400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6815","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6818","nonce":2691532400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6823","nonce":2691532400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2691532400},{"type":"v1","addr":"172.21.15.98:6821","nonce":2691532400}]},"public_addr":"172.21.15.98:6812/2691532400","cluster_addr":"172.21.15.98:6818/2691532400","heartbeat_back_addr":"172.21.15.98:6823/2691532400","heartbeat_front_addr":"172.21.15.98:6821/2691532400","state":["exists","up"]},{"osd":3,"uuid":"a8373586-43f6-4c83-b0c5-d26889c25963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6825","nonce":1905487008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6827","nonce":1905487008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6831","nonce":1905487008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":1905487008},{"type":"v1","addr":"172.21.15.98:6829","nonce":1905487008}]},"public_addr":"172.21.15.98:6825/1905487008","cluster_addr":"172.21.15.98:6827/1905487008","heartbeat_back_addr":"172.21.15.98:6831/1905487008","heartbeat_front_addr":"172.21.15.98:6829/1905487008","state":["exists","up"]},{"osd":4,"uuid":"fcf4f283-d0e0-4860-94f2-20301fd75204","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6801","nonce":2974677215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6803","nonce":2974677215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6807","nonce":2974677215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":2974677215},{"type":"v1","addr":"172.21.15.113:6805","nonce":2974677215}]},"public_addr":"172.21.15.113:6801/2974677215","cluster_addr":"172.21.15.113:6803/2974677215","heartbeat_back_addr":"172.21.15.113:6807/2974677215","heartbeat_front_addr":"172.21.15.113:6805/2974677215","state":["exists","up"]},{"osd":5,"uuid":"b8b1086a-77a2-459a-b541-154e955258c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6817","nonce":2685000355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6821","nonce":2685000355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6829","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6831","nonce":2685000355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6823","nonce":2685000355},{"type":"v1","addr":"172.21.15.113:6826","nonce":2685000355}]},"public_addr":"172.21.15.113:6817/2685000355","cluster_addr":"172.21.15.113:6821/2685000355","heartbeat_back_addr":"172.21.15.113:6831/2685000355","heartbeat_front_addr":"172.21.15.113:6826/2685000355","state":["exists","up"]},{"osd":6,"uuid":"a6a41999-a583-4c92-8675-5c5336e9319a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6809","nonce":2764300831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6811","nonce":2764300831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6815","nonce":2764300831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":2764300831},{"type":"v1","addr":"172.21.15.113:6813","nonce":2764300831}]},"public_addr":"172.21.15.113:6809/2764300831","cluster_addr":"172.21.15.113:6811/2764300831","heartbeat_back_addr":"172.21.15.113:6815/2764300831","heartbeat_front_addr":"172.21.15.113:6813/2764300831","state":["exists","up"]},{"osd":7,"uuid":"7ab8ecaf-289c-444c-a54d-ffa2cacfa663","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6819","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6820","nonce":1656248400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6824","nonce":1656248400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6830","nonce":1656248400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6825","nonce":1656248400},{"type":"v1","addr":"172.21.15.113:6827","nonce":1656248400}]},"public_addr":"172.21.15.113:6820/1656248400","cluster_addr":"172.21.15.113:6824/1656248400","heartbeat_back_addr":"172.21.15.113:6830/1656248400","heartbeat_front_addr":"172.21.15.113:6827/1656248400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.379549+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.312196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.117494+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.388128+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.360608+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.411101+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.355995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:41:06.390362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.29","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:49:58.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.404+0000 7fde7e62e640 1 -- 172.21.15.98:0/174465499 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fde5c06b2d0 msgr2=0x7fde5c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:58.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.404+0000 7fde7e62e640 1 --2- 172.21.15.98:0/174465499 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fde5c06b2d0 0x7fde5c06d790 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fde74036000 tx=0x7fde7403b040 comp rx=0 tx=0).stop 2023-11-28T17:49:58.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.404+0000 7fde7e62e640 1 -- 172.21.15.98:0/174465499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde780c71f0 msgr2=0x7fde781ef3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:58.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.404+0000 7fde7e62e640 1 --2- 172.21.15.98:0/174465499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde780c71f0 0x7fde781ef3c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fde6800c7f0 tx=0x7fde6800ccb0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.404+0000 7fde7e62e640 1 -- 172.21.15.98:0/174465499 shutdown_connections 2023-11-28T17:49:58.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.404+0000 7fde7e62e640 1 --2- 172.21.15.98:0/174465499 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fde78150ac0 0x7fde781ef900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.404+0000 7fde7e62e640 1 --2- 172.21.15.98:0/174465499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde780c71f0 0x7fde781ef3c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.404+0000 7fde7e62e640 1 --2- 172.21.15.98:0/174465499 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fde5c06b2d0 0x7fde5c06d790 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.404+0000 7fde7e62e640 1 --2- 172.21.15.98:0/174465499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde7806b1e0 0x7fde7814f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.404+0000 7fde7e62e640 1 -- 172.21.15.98:0/174465499 >> 172.21.15.98:0/174465499 conn(0x7fde78066230 msgr2=0x7fde780c7700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:58.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.404+0000 7fde7e62e640 1 -- 172.21.15.98:0/174465499 shutdown_connections 2023-11-28T17:49:58.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.404+0000 7fde7e62e640 1 -- 172.21.15.98:0/174465499 wait complete. 2023-11-28T17:49:58.428 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:41:10.303280+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:41:14.890871+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:41:15.704103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:49:58.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:49:58.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.632+0000 7fa5cee9b640 1 Processor -- start 2023-11-28T17:49:58.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.633+0000 7fa5cee9b640 1 -- start start 2023-11-28T17:49:58.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.634+0000 7fa5cee9b640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7fa5c80c4760 0x7fa5c80c2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.634+0000 7fa5cee9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa5c815a0f0 0x7fa5c815a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.634+0000 7fa5cee9b640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c8156980 con 0x7fa5c80c4380 2023-11-28T17:49:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.634+0000 7fa5cee9b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c80c2a60 con 0x7fa5c815a0f0 2023-11-28T17:49:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.634+0000 7fa5cee9b640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c815aa60 con 0x7fa5c80c4760 2023-11-28T17:49:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.634+0000 7fa5cd438640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa5c815a0f0 0x7fa5c815a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.635+0000 7fa5cd438640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa5c815a0f0 0x7fa5c815a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59800/0 (socket says 172.21.15.98:59800) 2023-11-28T17:49:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.635+0000 7fa5cd438640 1 -- 172.21.15.98:0/3142956563 learned_addr learned my addr 172.21.15.98:0/3142956563 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.635+0000 7fa5c7fff640 1 --2- 172.21.15.98:0/3142956563 >> v2:172.21.15.113:3300/0 conn(0x7fa5c80c4760 0x7fa5c80c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.635+0000 7fa5cd438640 1 -- 172.21.15.98:0/3142956563 >> v2:172.21.15.113:3300/0 conn(0x7fa5c80c4760 msgr2=0x7fa5c80c2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.635+0000 7fa5cd438640 1 --2- 172.21.15.98:0/3142956563 >> v2:172.21.15.113:3300/0 conn(0x7fa5c80c4760 0x7fa5c80c2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.635+0000 7fa5cd438640 1 -- 172.21.15.98:0/3142956563 >> v1:172.21.15.113:6789/0 conn(0x7fa5c80c4380 legacy=0x7fa5c80c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.635+0000 7fa5cd438640 1 -- 172.21.15.98:0/3142956563 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c8156c40 con 0x7fa5c815a0f0 2023-11-28T17:49:58.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.636+0000 7fa5cd438640 1 --2- 172.21.15.98:0/3142956563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c815a0f0 0x7fa5c815a520 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa5c806b9f0 tx=0x7fa5c002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96770a0068c4036 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:58.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.637+0000 7fa5c77fe640 1 -- 172.21.15.98:0/3142956563 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0030860 con 0x7fa5c815a0f0 2023-11-28T17:49:58.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.637+0000 7fa5c77fe640 1 -- 172.21.15.98:0/3142956563 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c00309c0 con 0x7fa5c815a0f0 2023-11-28T17:49:58.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.637+0000 7fa5c77fe640 1 -- 172.21.15.98:0/3142956563 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0030d50 con 0x7fa5c815a0f0 2023-11-28T17:49:58.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.637+0000 7fa5cee9b640 1 -- 172.21.15.98:0/3142956563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c815a0f0 msgr2=0x7fa5c815a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:58.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.637+0000 7fa5cee9b640 1 --2- 172.21.15.98:0/3142956563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c815a0f0 0x7fa5c815a520 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa5c806b9f0 tx=0x7fa5c002b9a0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.638+0000 7fa5cee9b640 1 -- 172.21.15.98:0/3142956563 shutdown_connections 2023-11-28T17:49:58.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.638+0000 7fa5cee9b640 1 --2- 172.21.15.98:0/3142956563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c815a0f0 0x7fa5c815a520 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.638+0000 7fa5cee9b640 1 --2- 172.21.15.98:0/3142956563 >> v2:172.21.15.113:3300/0 conn(0x7fa5c80c4760 0x7fa5c80c2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.638+0000 7fa5cee9b640 1 -- 172.21.15.98:0/3142956563 >> 172.21.15.98:0/3142956563 conn(0x7fa5c806c0f0 msgr2=0x7fa5c806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:58.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.638+0000 7fa5cee9b640 1 -- 172.21.15.98:0/3142956563 shutdown_connections 2023-11-28T17:49:58.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.638+0000 7fa5cee9b640 1 -- 172.21.15.98:0/3142956563 wait complete. 2023-11-28T17:49:58.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.639+0000 7fa5cee9b640 1 Processor -- start 2023-11-28T17:49:58.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.640+0000 7fa5cee9b640 1 -- start start 2023-11-28T17:49:58.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.640+0000 7fa5cee9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c80c4380 0x7fa5c8151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.640+0000 7fa5cee9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c80c4760 0x7fa5c8158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.640+0000 7fa5cee9b640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c815a0f0 0x7fa5c8154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.640+0000 7fa5cee9b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c8159e20 con 0x7fa5c80c4380 2023-11-28T17:49:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.640+0000 7fa5cee9b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c813f0e0 con 0x7fa5c80c4760 2023-11-28T17:49:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.640+0000 7fa5cee9b640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c81556d0 con 0x7fa5c815a0f0 2023-11-28T17:49:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.641+0000 7fa5ccc37640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c80c4380 0x7fa5c8151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.641+0000 7fa5ccc37640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c80c4380 0x7fa5c8151df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59802/0 (socket says 172.21.15.98:59802) 2023-11-28T17:49:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.641+0000 7fa5ccc37640 1 -- 172.21.15.98:0/1878094765 learned_addr learned my addr 172.21.15.98:0/1878094765 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.641+0000 7fa5c7fff640 1 --2- 172.21.15.98:0/1878094765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c80c4760 0x7fa5c8158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.641+0000 7fa5cd438640 1 --2- 172.21.15.98:0/1878094765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c815a0f0 0x7fa5c8154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.641+0000 7fa5ccc37640 1 -- 172.21.15.98:0/1878094765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c815a0f0 msgr2=0x7fa5c8154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.641+0000 7fa5ccc37640 1 --2- 172.21.15.98:0/1878094765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c815a0f0 0x7fa5c8154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.641+0000 7fa5ccc37640 1 -- 172.21.15.98:0/1878094765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c80c4760 msgr2=0x7fa5c8158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.641+0000 7fa5ccc37640 1 --2- 172.21.15.98:0/1878094765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c80c4760 0x7fa5c8158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.641+0000 7fa5ccc37640 1 -- 172.21.15.98:0/1878094765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c00094d0 con 0x7fa5c80c4380 2023-11-28T17:49:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.641+0000 7fa5cd438640 1 --2- 172.21.15.98:0/1878094765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c815a0f0 0x7fa5c8154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.642+0000 7fa5ccc37640 1 --2- 172.21.15.98:0/1878094765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c80c4380 0x7fa5c8151df0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000c460 tx=0x7fa5b000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.642+0000 7fa5c57fa640 1 -- 172.21.15.98:0/1878094765 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0011790 con 0x7fa5c80c4380 2023-11-28T17:49:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.642+0000 7fa5c57fa640 1 -- 172.21.15.98:0/1878094765 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b0011930 con 0x7fa5c80c4380 2023-11-28T17:49:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.642+0000 7fa5c57fa640 1 -- 172.21.15.98:0/1878094765 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0011ce0 con 0x7fa5c80c4380 2023-11-28T17:49:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.642+0000 7fa5cee9b640 1 -- 172.21.15.98:0/1878094765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c8151350 con 0x7fa5c80c4380 2023-11-28T17:49:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.642+0000 7fa5cee9b640 1 -- 172.21.15.98:0/1878094765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c81553f0 con 0x7fa5c80c4380 2023-11-28T17:49:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.643+0000 7fa5cee9b640 1 -- 172.21.15.98:0/1878094765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c8074f60 con 0x7fa5c80c4380 2023-11-28T17:49:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.644+0000 7fa5c57fa640 1 -- 172.21.15.98:0/1878094765 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa5b001c460 con 0x7fa5c80c4380 2023-11-28T17:49:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.645+0000 7fa5c57fa640 1 --2- 172.21.15.98:0/1878094765 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa5a806b2d0 0x7fa5a806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.645+0000 7fa5c57fa640 1 -- 172.21.15.98:0/1878094765 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa5b0089c10 con 0x7fa5c80c4380 2023-11-28T17:49:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.645+0000 7fa5c7fff640 1 --2- 172.21.15.98:0/1878094765 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa5a806b2d0 0x7fa5a806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.646+0000 7fa5c7fff640 1 --2- 172.21.15.98:0/1878094765 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa5a806b2d0 0x7fa5a806d790 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8159830 tx=0x7fa5b8003990 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:58.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.650+0000 7fa5c57fa640 1 -- 172.21.15.98:0/1878094765 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa5b0054960 con 0x7fa5c80c4380 2023-11-28T17:49:58.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.841+0000 7fa5cee9b640 1 -- 172.21.15.98:0/1878094765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5c806e350 con 0x7fa5c80c4380 2023-11-28T17:49:58.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.842+0000 7fa5c57fa640 1 -- 172.21.15.98:0/1878094765 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa5b0057f80 con 0x7fa5c80c4380 2023-11-28T17:49:58.842 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-11-28T17:49:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.844+0000 7fa5cee9b640 1 -- 172.21.15.98:0/1878094765 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa5a806b2d0 msgr2=0x7fa5a806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.844+0000 7fa5cee9b640 1 --2- 172.21.15.98:0/1878094765 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa5a806b2d0 0x7fa5a806d790 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8159830 tx=0x7fa5b8003990 comp rx=0 tx=0).stop 2023-11-28T17:49:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.844+0000 7fa5cee9b640 1 -- 172.21.15.98:0/1878094765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c80c4380 msgr2=0x7fa5c8151df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.844+0000 7fa5cee9b640 1 --2- 172.21.15.98:0/1878094765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c80c4380 0x7fa5c8151df0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000c460 tx=0x7fa5b000c920 comp rx=0 tx=0).stop 2023-11-28T17:49:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.844+0000 7fa5cee9b640 1 -- 172.21.15.98:0/1878094765 shutdown_connections 2023-11-28T17:49:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.844+0000 7fa5cee9b640 1 --2- 172.21.15.98:0/1878094765 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7fa5c815a0f0 0x7fa5c8154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.844+0000 7fa5cee9b640 1 --2- 172.21.15.98:0/1878094765 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7fa5a806b2d0 0x7fa5a806d790 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.844+0000 7fa5cee9b640 1 --2- 172.21.15.98:0/1878094765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c80c4760 0x7fa5c8158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.844+0000 7fa5cee9b640 1 --2- 172.21.15.98:0/1878094765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c80c4380 0x7fa5c8151df0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.844+0000 7fa5cee9b640 1 -- 172.21.15.98:0/1878094765 >> 172.21.15.98:0/1878094765 conn(0x7fa5c806c0f0 msgr2=0x7fa5c81504b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.844+0000 7fa5cee9b640 1 -- 172.21.15.98:0/1878094765 shutdown_connections 2023-11-28T17:49:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:58.844+0000 7fa5cee9b640 1 -- 172.21.15.98:0/1878094765 wait complete. 2023-11-28T17:49:58.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:49:59.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.075+0000 7f9c249ec640 1 Processor -- start 2023-11-28T17:49:59.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.075+0000 7f9c249ec640 1 -- start start 2023-11-28T17:49:59.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.077+0000 7f9c249ec640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9c20157ae0 0x7f9c20151910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.077+0000 7f9c249ec640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9c200c5f70 0x7f9c20151e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.077+0000 7f9c249ec640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c200cc790 con 0x7f9c200c5b90 2023-11-28T17:49:59.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.077+0000 7f9c249ec640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c200c71e0 con 0x7f9c20157ae0 2023-11-28T17:49:59.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.077+0000 7f9c249ec640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2006e280 con 0x7f9c200c5f70 2023-11-28T17:49:59.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.077+0000 7f9c1dd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9c20157ae0 0x7f9c20151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.077+0000 7f9c1dd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9c20157ae0 0x7f9c20151910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59806/0 (socket says 172.21.15.98:59806) 2023-11-28T17:49:59.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.077+0000 7f9c1dd9b640 1 -- 172.21.15.98:0/3187651321 learned_addr learned my addr 172.21.15.98:0/3187651321 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:59.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.078+0000 7f9c1ed9d640 1 --2- 172.21.15.98:0/3187651321 >> v2:172.21.15.113:3300/0 conn(0x7f9c200c5f70 0x7f9c20151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.078+0000 7f9c1dd9b640 1 -- 172.21.15.98:0/3187651321 >> v2:172.21.15.113:3300/0 conn(0x7f9c200c5f70 msgr2=0x7f9c20151e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:59.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.078+0000 7f9c1dd9b640 1 --2- 172.21.15.98:0/3187651321 >> v2:172.21.15.113:3300/0 conn(0x7f9c200c5f70 0x7f9c20151e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.078+0000 7f9c1dd9b640 1 -- 172.21.15.98:0/3187651321 >> v1:172.21.15.113:6789/0 conn(0x7f9c200c5b90 legacy=0x7f9c200ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:59.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.078+0000 7f9c1dd9b640 1 -- 172.21.15.98:0/3187651321 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2006e560 con 0x7f9c20157ae0 2023-11-28T17:49:59.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.079+0000 7f9c1dd9b640 1 --2- 172.21.15.98:0/3187651321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c20157ae0 0x7f9c20151910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9c200c8660 tx=0x7f9c0802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19d115e174dd8bb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:59.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.079+0000 7f9c1d59a640 1 -- 172.21.15.98:0/3187651321 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08031870 con 0x7f9c20157ae0 2023-11-28T17:49:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.079+0000 7f9c1d59a640 1 -- 172.21.15.98:0/3187651321 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c080319d0 con 0x7f9c20157ae0 2023-11-28T17:49:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.079+0000 7f9c1d59a640 1 -- 172.21.15.98:0/3187651321 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08031d60 con 0x7f9c20157ae0 2023-11-28T17:49:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.080+0000 7f9c249ec640 1 -- 172.21.15.98:0/3187651321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c20157ae0 msgr2=0x7f9c20151910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.080+0000 7f9c249ec640 1 --2- 172.21.15.98:0/3187651321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c20157ae0 0x7f9c20151910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9c200c8660 tx=0x7f9c0802b910 comp rx=0 tx=0).stop 2023-11-28T17:49:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.080+0000 7f9c249ec640 1 -- 172.21.15.98:0/3187651321 shutdown_connections 2023-11-28T17:49:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.080+0000 7f9c249ec640 1 --2- 172.21.15.98:0/3187651321 >> v2:172.21.15.113:3300/0 conn(0x7f9c200c5f70 0x7f9c20151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.080+0000 7f9c249ec640 1 --2- 172.21.15.98:0/3187651321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c20157ae0 0x7f9c20151910 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.080+0000 7f9c249ec640 1 -- 172.21.15.98:0/3187651321 >> 172.21.15.98:0/3187651321 conn(0x7f9c2006ef10 msgr2=0x7f9c2006f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:59.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.081+0000 7f9c249ec640 1 -- 172.21.15.98:0/3187651321 shutdown_connections 2023-11-28T17:49:59.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.081+0000 7f9c249ec640 1 -- 172.21.15.98:0/3187651321 wait complete. 2023-11-28T17:49:59.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.082+0000 7f9c249ec640 1 Processor -- start 2023-11-28T17:49:59.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.082+0000 7f9c249ec640 1 -- start start 2023-11-28T17:49:59.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.083+0000 7f9c249ec640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c200c5b90 0x7f9c201541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.083+0000 7f9c249ec640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c200c5f70 0x7f9c201f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.083+0000 7f9c249ec640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c20157ae0 0x7f9c20150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.083+0000 7f9c249ec640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2014d860 con 0x7f9c200c5f70 2023-11-28T17:49:59.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.083+0000 7f9c249ec640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c20154720 con 0x7f9c20157ae0 2023-11-28T17:49:59.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.083+0000 7f9c249ec640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c20151f50 con 0x7f9c200c5b90 2023-11-28T17:49:59.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.084+0000 7f9c1dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c200c5f70 0x7f9c201f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.084+0000 7f9c1dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c200c5f70 0x7f9c201f2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59810/0 (socket says 172.21.15.98:59810) 2023-11-28T17:49:59.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.084+0000 7f9c1dd9b640 1 -- 172.21.15.98:0/2395622073 learned_addr learned my addr 172.21.15.98:0/2395622073 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:59.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.084+0000 7f9c1ed9d640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c20157ae0 0x7f9c20150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.084+0000 7f9c1e59c640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c200c5b90 0x7f9c201541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.084+0000 7f9c1dd9b640 1 -- 172.21.15.98:0/2395622073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c200c5b90 msgr2=0x7f9c201541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:59.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.084+0000 7f9c1dd9b640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c200c5b90 0x7f9c201541e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.084+0000 7f9c1dd9b640 1 -- 172.21.15.98:0/2395622073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c20157ae0 msgr2=0x7f9c20150720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:59.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.084+0000 7f9c1dd9b640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c20157ae0 0x7f9c20150720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.084+0000 7f9c1dd9b640 1 -- 172.21.15.98:0/2395622073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c080094d0 con 0x7f9c200c5f70 2023-11-28T17:49:59.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.084+0000 7f9c1e59c640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c200c5b90 0x7f9c201541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:59.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.084+0000 7f9c1ed9d640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c20157ae0 0x7f9c20150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:59.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.085+0000 7f9c1dd9b640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c200c5f70 0x7f9c201f2c30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9c08000c00 tx=0x7f9c080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:59.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.085+0000 7f9c16ffd640 1 -- 172.21.15.98:0/2395622073 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0803abe0 con 0x7f9c200c5f70 2023-11-28T17:49:59.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.085+0000 7f9c16ffd640 1 -- 172.21.15.98:0/2395622073 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0803ad40 con 0x7f9c200c5f70 2023-11-28T17:49:59.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.085+0000 7f9c16ffd640 1 -- 172.21.15.98:0/2395622073 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08044610 con 0x7f9c200c5f70 2023-11-28T17:49:59.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.085+0000 7f9c249ec640 1 -- 172.21.15.98:0/2395622073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c201f0080 con 0x7f9c200c5f70 2023-11-28T17:49:59.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.085+0000 7f9c249ec640 1 -- 172.21.15.98:0/2395622073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c201ef750 con 0x7f9c200c5f70 2023-11-28T17:49:59.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.086+0000 7f9c249ec640 1 -- 172.21.15.98:0/2395622073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c201f0230 con 0x7f9c200c5f70 2023-11-28T17:49:59.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.087+0000 7f9c16ffd640 1 -- 172.21.15.98:0/2395622073 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f9c08044770 con 0x7f9c200c5f70 2023-11-28T17:49:59.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.088+0000 7f9c16ffd640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9c0006b2a0 0x7f9c0006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.088+0000 7f9c16ffd640 1 -- 172.21.15.98:0/2395622073 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9c0802d040 con 0x7f9c200c5f70 2023-11-28T17:49:59.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.088+0000 7f9c1e59c640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9c0006b2a0 0x7f9c0006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.089+0000 7f9c1e59c640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9c0006b2a0 0x7f9c0006d760 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9c10003eb0 tx=0x7f9c10009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:59.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.092+0000 7f9c16ffd640 1 -- 172.21.15.98:0/2395622073 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9c08077230 con 0x7f9c200c5f70 2023-11-28T17:49:59.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.280+0000 7f9c249ec640 1 -- 172.21.15.98:0/2395622073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9c200c99a0 con 0x7f9c200c5f70 2023-11-28T17:49:59.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.281+0000 7f9c16ffd640 1 -- 172.21.15.98:0/2395622073 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9c0807a850 con 0x7f9c200c5f70 2023-11-28T17:49:59.281 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-11-28T17:49:59.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.283+0000 7f9c249ec640 1 -- 172.21.15.98:0/2395622073 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9c0006b2a0 msgr2=0x7f9c0006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:59.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.284+0000 7f9c249ec640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9c0006b2a0 0x7f9c0006d760 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9c10003eb0 tx=0x7f9c10009040 comp rx=0 tx=0).stop 2023-11-28T17:49:59.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.284+0000 7f9c249ec640 1 -- 172.21.15.98:0/2395622073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c200c5f70 msgr2=0x7f9c201f2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:59.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.284+0000 7f9c249ec640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c200c5f70 0x7f9c201f2c30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9c08000c00 tx=0x7f9c080046c0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.284+0000 7f9c249ec640 1 -- 172.21.15.98:0/2395622073 shutdown_connections 2023-11-28T17:49:59.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.284+0000 7f9c249ec640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f9c0006b2a0 0x7f9c0006d760 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.284+0000 7f9c249ec640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9c20157ae0 0x7f9c20150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.284+0000 7f9c249ec640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c200c5f70 0x7f9c201f2c30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.284+0000 7f9c249ec640 1 --2- 172.21.15.98:0/2395622073 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9c200c5b90 0x7f9c201541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.284+0000 7f9c249ec640 1 -- 172.21.15.98:0/2395622073 >> 172.21.15.98:0/2395622073 conn(0x7f9c2006ef10 msgr2=0x7f9c2006b530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:59.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.284+0000 7f9c249ec640 1 -- 172.21.15.98:0/2395622073 shutdown_connections 2023-11-28T17:49:59.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.284+0000 7f9c249ec640 1 -- 172.21.15.98:0/2395622073 wait complete. 2023-11-28T17:49:59.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:49:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.518+0000 7f221c87c640 1 Processor -- start 2023-11-28T17:49:59.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.518+0000 7f221c87c640 1 -- start start 2023-11-28T17:49:59.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.520+0000 7f221c87c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f22180c4380 0x7f22180c2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.520+0000 7f221c87c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f221815a0f0 0x7f221815a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.520+0000 7f221c87c640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221815aaa0 con 0x7f22180c4760 2023-11-28T17:49:59.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.520+0000 7f221c87c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22180c2a60 con 0x7f22180c4380 2023-11-28T17:49:59.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.520+0000 7f221c87c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2218156980 con 0x7f221815a0f0 2023-11-28T17:49:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.520+0000 7f221659c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f22180c4380 0x7f22180c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.520+0000 7f221659c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f22180c4380 0x7f22180c2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59814/0 (socket says 172.21.15.98:59814) 2023-11-28T17:49:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.520+0000 7f221659c640 1 -- 172.21.15.98:0/2004814798 learned_addr learned my addr 172.21.15.98:0/2004814798 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.521+0000 7f2216d9d640 1 --2- 172.21.15.98:0/2004814798 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f221815a0f0 0x7f221815a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.521+0000 7f221659c640 1 -- 172.21.15.98:0/2004814798 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f221815a0f0 msgr2=0x7f221815a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.521+0000 7f221659c640 1 --2- 172.21.15.98:0/2004814798 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f221815a0f0 0x7f221815a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.521+0000 7f221659c640 1 -- 172.21.15.98:0/2004814798 >> v1:172.21.15.113:6789/0 conn(0x7f22180c4760 legacy=0x7f22180c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.521+0000 7f221659c640 1 -- 172.21.15.98:0/2004814798 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2218156c40 con 0x7f22180c4380 2023-11-28T17:49:59.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.521+0000 7f2216d9d640 1 --2- 172.21.15.98:0/2004814798 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f221815a0f0 0x7f221815a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:59.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.522+0000 7f221659c640 1 --2- 172.21.15.98:0/2004814798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22180c4380 0x7f22180c2520 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f22080098a0 tx=0x7f220802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9582fa71be071fdc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:59.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.522+0000 7f2215d9b640 1 -- 172.21.15.98:0/2004814798 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2208030840 con 0x7f22180c4380 2023-11-28T17:49:59.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.522+0000 7f2215d9b640 1 -- 172.21.15.98:0/2004814798 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22080309a0 con 0x7f22180c4380 2023-11-28T17:49:59.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.522+0000 7f2215d9b640 1 -- 172.21.15.98:0/2004814798 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2208030d30 con 0x7f22180c4380 2023-11-28T17:49:59.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.523+0000 7f221c87c640 1 -- 172.21.15.98:0/2004814798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22180c4380 msgr2=0x7f22180c2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:59.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.523+0000 7f221c87c640 1 --2- 172.21.15.98:0/2004814798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22180c4380 0x7f22180c2520 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f22080098a0 tx=0x7f220802b980 comp rx=0 tx=0).stop 2023-11-28T17:49:59.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.523+0000 7f221c87c640 1 -- 172.21.15.98:0/2004814798 shutdown_connections 2023-11-28T17:49:59.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.523+0000 7f221c87c640 1 --2- 172.21.15.98:0/2004814798 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f221815a0f0 0x7f221815a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.523+0000 7f221c87c640 1 --2- 172.21.15.98:0/2004814798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22180c4380 0x7f22180c2520 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.523+0000 7f221c87c640 1 -- 172.21.15.98:0/2004814798 >> 172.21.15.98:0/2004814798 conn(0x7f221806c0f0 msgr2=0x7f221806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:59.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.523+0000 7f221c87c640 1 -- 172.21.15.98:0/2004814798 shutdown_connections 2023-11-28T17:49:59.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.523+0000 7f221c87c640 1 -- 172.21.15.98:0/2004814798 wait complete. 2023-11-28T17:49:59.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.525+0000 7f221c87c640 1 Processor -- start 2023-11-28T17:49:59.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.525+0000 7f221c87c640 1 -- start start 2023-11-28T17:49:59.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.526+0000 7f221c87c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22180c4380 0x7f22180ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.526+0000 7f221c87c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22180c4760 0x7f22181f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.526+0000 7f221c87c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f221815a0f0 0x7f22181f33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.526+0000 7f221c87c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2218163420 con 0x7f22180c4380 2023-11-28T17:49:59.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.526+0000 7f221c87c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2218156100 con 0x7f221815a0f0 2023-11-28T17:49:59.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.526+0000 7f221c87c640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22181f2990 con 0x7f22180c4760 2023-11-28T17:49:59.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.526+0000 7f221659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22180c4380 0x7f22180ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.526+0000 7f221659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22180c4380 0x7f22180ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59816/0 (socket says 172.21.15.98:59816) 2023-11-28T17:49:59.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.526+0000 7f221659c640 1 -- 172.21.15.98:0/155696680 learned_addr learned my addr 172.21.15.98:0/155696680 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:59.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.526+0000 7f2216d9d640 1 --2- 172.21.15.98:0/155696680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f221815a0f0 0x7f22181f33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.526+0000 7f221659c640 1 -- 172.21.15.98:0/155696680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22180c4760 msgr2=0x7f22181f2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.526+0000 7f221659c640 1 --2- 172.21.15.98:0/155696680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22180c4760 0x7f22181f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.526+0000 7f221659c640 1 -- 172.21.15.98:0/155696680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f221815a0f0 msgr2=0x7f22181f33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.527+0000 7f221659c640 1 --2- 172.21.15.98:0/155696680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f221815a0f0 0x7f22181f33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.527+0000 7f221659c640 1 -- 172.21.15.98:0/155696680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22080094d0 con 0x7f22180c4380 2023-11-28T17:49:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.527+0000 7f221659c640 1 --2- 172.21.15.98:0/155696680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22180c4380 0x7f22180ca180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2208000c00 tx=0x7f2208004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.527+0000 7f220f7fe640 1 -- 172.21.15.98:0/155696680 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22080307f0 con 0x7f22180c4380 2023-11-28T17:49:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.527+0000 7f220f7fe640 1 -- 172.21.15.98:0/155696680 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2208043400 con 0x7f22180c4380 2023-11-28T17:49:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.527+0000 7f221c87c640 1 -- 172.21.15.98:0/155696680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221814b560 con 0x7f22180c4380 2023-11-28T17:49:59.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.527+0000 7f220f7fe640 1 -- 172.21.15.98:0/155696680 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2208043790 con 0x7f22180c4380 2023-11-28T17:49:59.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.527+0000 7f221c87c640 1 -- 172.21.15.98:0/155696680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2218156680 con 0x7f22180c4380 2023-11-28T17:49:59.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.529+0000 7f221c87c640 1 -- 172.21.15.98:0/155696680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2218074f60 con 0x7f22180c4380 2023-11-28T17:49:59.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.529+0000 7f220f7fe640 1 -- 172.21.15.98:0/155696680 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f22080438f0 con 0x7f22180c4380 2023-11-28T17:49:59.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.530+0000 7f220f7fe640 1 --2- 172.21.15.98:0/155696680 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f21f806b2d0 0x7f21f806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.530+0000 7f220f7fe640 1 -- 172.21.15.98:0/155696680 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f220803e070 con 0x7f22180c4380 2023-11-28T17:49:59.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.531+0000 7f220dd9b640 1 --2- 172.21.15.98:0/155696680 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f21f806b2d0 0x7f21f806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.531+0000 7f220dd9b640 1 --2- 172.21.15.98:0/155696680 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f21f806b2d0 0x7f21f806d790 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2218140d20 tx=0x7f22000079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:59.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.535+0000 7f220f7fe640 1 -- 172.21.15.98:0/155696680 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2208077f10 con 0x7f22180c4380 2023-11-28T17:49:59.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.725+0000 7f221c87c640 1 -- 172.21.15.98:0/155696680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f22180cadc0 con 0x7f22180c4380 2023-11-28T17:49:59.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.726+0000 7f220f7fe640 1 -- 172.21.15.98:0/155696680 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f22080340a0 con 0x7f22180c4380 2023-11-28T17:49:59.727 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-11-28T17:49:59.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.728+0000 7f221c87c640 1 -- 172.21.15.98:0/155696680 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f21f806b2d0 msgr2=0x7f21f806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:59.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.729+0000 7f221c87c640 1 --2- 172.21.15.98:0/155696680 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f21f806b2d0 0x7f21f806d790 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2218140d20 tx=0x7f22000079b0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.729+0000 7f221c87c640 1 -- 172.21.15.98:0/155696680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22180c4380 msgr2=0x7f22180ca180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:59.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.729+0000 7f221c87c640 1 --2- 172.21.15.98:0/155696680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22180c4380 0x7f22180ca180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2208000c00 tx=0x7f2208004730 comp rx=0 tx=0).stop 2023-11-28T17:49:59.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.729+0000 7f221c87c640 1 -- 172.21.15.98:0/155696680 shutdown_connections 2023-11-28T17:49:59.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.729+0000 7f221c87c640 1 --2- 172.21.15.98:0/155696680 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f21f806b2d0 0x7f21f806d790 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.729+0000 7f221c87c640 1 --2- 172.21.15.98:0/155696680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f221815a0f0 0x7f22181f33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.729+0000 7f221c87c640 1 --2- 172.21.15.98:0/155696680 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f22180c4760 0x7f22181f2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.729+0000 7f221c87c640 1 --2- 172.21.15.98:0/155696680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22180c4380 0x7f22180ca180 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.729+0000 7f221c87c640 1 -- 172.21.15.98:0/155696680 >> 172.21.15.98:0/155696680 conn(0x7f221806c0f0 msgr2=0x7f22180cb6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:59.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.729+0000 7f221c87c640 1 -- 172.21.15.98:0/155696680 shutdown_connections 2023-11-28T17:49:59.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.729+0000 7f221c87c640 1 -- 172.21.15.98:0/155696680 wait complete. 2023-11-28T17:49:59.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:49:59.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.962+0000 7f658a82e640 1 Processor -- start 2023-11-28T17:49:59.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.962+0000 7f658a82e640 1 -- start start 2023-11-28T17:49:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.963+0000 7f658a82e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6584153f50 0x7f658415da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.963+0000 7f658a82e640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f65840c75b0 0x7f65840c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.963+0000 7f658a82e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658415dfd0 con 0x7f65840c69a0 2023-11-28T17:49:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.963+0000 7f658a82e640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6584154330 con 0x7f65840c75b0 2023-11-28T17:49:59.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.963+0000 7f658a82e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65840cb680 con 0x7f6584153f50 2023-11-28T17:49:59.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.964+0000 7f658902b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f65840c69a0 0x7f65840ca090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33968/0 (socket says 172.21.15.98:33968) 2023-11-28T17:49:59.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.964+0000 7f658902b640 1 -- 172.21.15.98:0/1183888957 learned_addr learned my addr 172.21.15.98:0/1183888957 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-11-28T17:49:59.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.964+0000 7f658982c640 1 --2- 172.21.15.98:0/1183888957 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6584153f50 0x7f658415da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.964+0000 7f658a02d640 1 --2- 172.21.15.98:0/1183888957 >> v2:172.21.15.113:3300/0 conn(0x7f65840c75b0 0x7f65840c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.964+0000 7f658882a640 1 -- 172.21.15.98:0/1183888957 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375264732 0 0) 0x7f658415dfd0 con 0x7f65840c69a0 2023-11-28T17:49:59.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.964+0000 7f658882a640 1 -- 172.21.15.98:0/1183888957 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6570003520 con 0x7f65840c69a0 2023-11-28T17:49:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658882a640 1 -- 172.21.15.98:0/1183888957 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3633876762 0 0) 0x7f6570003520 con 0x7f65840c69a0 2023-11-28T17:49:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658882a640 1 -- 172.21.15.98:0/1183888957 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6584153f50 msgr2=0x7f658415da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658882a640 1 --2- 172.21.15.98:0/1183888957 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6584153f50 0x7f658415da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658882a640 1 -- 172.21.15.98:0/1183888957 >> v2:172.21.15.113:3300/0 conn(0x7f65840c75b0 msgr2=0x7f65840c7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658882a640 1 --2- 172.21.15.98:0/1183888957 >> v2:172.21.15.113:3300/0 conn(0x7f65840c75b0 0x7f65840c7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658882a640 1 -- 172.21.15.98:0/1183888957 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65840cbbb0 con 0x7f65840c69a0 2023-11-28T17:49:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658882a640 1 -- 172.21.15.98:0/1183888957 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3699693888 0 0) 0x7f6574001670 con 0x7f65840c69a0 2023-11-28T17:49:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658882a640 1 -- 172.21.15.98:0/1183888957 >> v1:172.21.15.98:6789/0 conn(0x7f65840c69a0 legacy=0x7f65840ca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658882a640 1 --2- 172.21.15.98:0/1183888957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6570003970 0x7f6570003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658882a640 1 --2- 172.21.15.98:0/1183888957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65700047c0 0x7f6570004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658882a640 1 --2- 172.21.15.98:0/1183888957 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6584153f50 0x7f6570005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658882a640 1 -- 172.21.15.98:0/1183888957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65840cb680 con 0x7f65700047c0 2023-11-28T17:49:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658882a640 1 -- 172.21.15.98:0/1183888957 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657000b980 con 0x7f6570003970 2023-11-28T17:49:59.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658882a640 1 -- 172.21.15.98:0/1183888957 --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657000bac0 con 0x7f6584153f50 2023-11-28T17:49:59.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658a02d640 1 --2- 172.21.15.98:0/1183888957 >> v2:172.21.15.113:3300/0 conn(0x7f65840c75b0 0x7f65840c7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:49:59.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658902b640 1 --2- 172.21.15.98:0/1183888957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65700047c0 0x7f6570004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658982c640 1 --2- 172.21.15.98:0/1183888957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6570003970 0x7f6570003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.965+0000 7f658a02d640 1 --2- 172.21.15.98:0/1183888957 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6584153f50 0x7f6570005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.966+0000 7f658902b640 1 -- 172.21.15.98:0/1183888957 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6584153f50 msgr2=0x7f6570005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:59.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.966+0000 7f658902b640 1 --2- 172.21.15.98:0/1183888957 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6584153f50 0x7f6570005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.966+0000 7f658902b640 1 -- 172.21.15.98:0/1183888957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6570003970 msgr2=0x7f6570003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:49:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.966+0000 7f658902b640 1 --2- 172.21.15.98:0/1183888957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6570003970 0x7f6570003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.966+0000 7f658902b640 1 -- 172.21.15.98:0/1183888957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6570003520 con 0x7f65700047c0 2023-11-28T17:49:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.966+0000 7f658a02d640 1 --2- 172.21.15.98:0/1183888957 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f6584153f50 0x7f6570005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:59.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.966+0000 7f658982c640 1 --2- 172.21.15.98:0/1183888957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6570003970 0x7f6570003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:49:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.967+0000 7f658902b640 1 --2- 172.21.15.98:0/1183888957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65700047c0 0x7f6570004c40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f657400c240 tx=0x7f65700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c578f63d7afc0153 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.967+0000 7f658882a640 1 -- 172.21.15.98:0/1183888957 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6574005ad0 con 0x7f65700047c0 2023-11-28T17:49:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.967+0000 7f658882a640 1 -- 172.21.15.98:0/1183888957 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6574005c30 con 0x7f65700047c0 2023-11-28T17:49:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.967+0000 7f658882a640 1 -- 172.21.15.98:0/1183888957 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657402c8d0 con 0x7f65700047c0 2023-11-28T17:49:59.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.968+0000 7f658a82e640 1 -- 172.21.15.98:0/1183888957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65700047c0 msgr2=0x7f6570004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.968+0000 7f658a82e640 1 --2- 172.21.15.98:0/1183888957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65700047c0 0x7f6570004c40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f657400c240 tx=0x7f65700057f0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.968+0000 7f658a02d640 1 -- 172.21.15.98:0/1183888957 reap_dead start 2023-11-28T17:49:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.968+0000 7f658a82e640 1 -- 172.21.15.98:0/1183888957 shutdown_connections 2023-11-28T17:49:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.968+0000 7f658a82e640 1 -- 172.21.15.98:0/1183888957 >> 172.21.15.98:0/1183888957 conn(0x7f65840cf310 msgr2=0x7f65840cf720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:49:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.968+0000 7f658a82e640 1 -- 172.21.15.98:0/1183888957 shutdown_connections 2023-11-28T17:49:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.968+0000 7f658a82e640 1 -- 172.21.15.98:0/1183888957 wait complete. 2023-11-28T17:49:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.970+0000 7f658a82e640 1 Processor -- start 2023-11-28T17:49:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.970+0000 7f658a82e640 1 -- start start 2023-11-28T17:49:59.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.971+0000 7f658a82e640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65700047c0 0x7f65841553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.971+0000 7f658a82e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f658415a9a0 0x7f6584156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.971+0000 7f658a82e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6584154480 0x7f65841f0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.971+0000 7f658a82e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6584156a50 con 0x7f658415a9a0 2023-11-28T17:49:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.971+0000 7f658a82e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65841f08a0 con 0x7f6584154480 2023-11-28T17:49:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.971+0000 7f658a82e640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65841559e0 con 0x7f65700047c0 2023-11-28T17:49:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.971+0000 7f658902b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f658415a9a0 0x7f6584156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.971+0000 7f658902b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f658415a9a0 0x7f6584156e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59832/0 (socket says 172.21.15.98:59832) 2023-11-28T17:49:59.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.971+0000 7f658902b640 1 -- 172.21.15.98:0/2761484000 learned_addr learned my addr 172.21.15.98:0/2761484000 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-11-28T17:49:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.972+0000 7f658a02d640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6584154480 0x7f65841f0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.972+0000 7f658982c640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65700047c0 0x7f65841553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.972+0000 7f658902b640 1 -- 172.21.15.98:0/2761484000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65700047c0 msgr2=0x7f65841553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.972+0000 7f658902b640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65700047c0 0x7f65841553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.972+0000 7f658902b640 1 -- 172.21.15.98:0/2761484000 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6584154480 msgr2=0x7f65841f0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:49:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.972+0000 7f658902b640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6584154480 0x7f65841f0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:49:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.972+0000 7f658902b640 1 -- 172.21.15.98:0/2761484000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6584155cc0 con 0x7f658415a9a0 2023-11-28T17:49:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.972+0000 7f658982c640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65700047c0 0x7f65841553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:59.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.972+0000 7f658a02d640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6584154480 0x7f65841f0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:49:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.972+0000 7f658902b640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f658415a9a0 0x7f6584156e80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6574001eb0 tx=0x7f65840c2e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.972+0000 7f657a7fc640 1 -- 172.21.15.98:0/2761484000 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6574039030 con 0x7f658415a9a0 2023-11-28T17:49:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.972+0000 7f657a7fc640 1 -- 172.21.15.98:0/2761484000 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657402cd00 con 0x7f658415a9a0 2023-11-28T17:49:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.972+0000 7f658a82e640 1 -- 172.21.15.98:0/2761484000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6584155fa0 con 0x7f658415a9a0 2023-11-28T17:49:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.972+0000 7f657a7fc640 1 -- 172.21.15.98:0/2761484000 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657403c040 con 0x7f658415a9a0 2023-11-28T17:49:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.972+0000 7f658a82e640 1 -- 172.21.15.98:0/2761484000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65841f0530 con 0x7f658415a9a0 2023-11-28T17:49:59.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.973+0000 7f658a82e640 1 -- 172.21.15.98:0/2761484000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65841f06f0 con 0x7f658415a9a0 2023-11-28T17:49:59.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.974+0000 7f657a7fc640 1 -- 172.21.15.98:0/2761484000 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6574032070 con 0x7f658415a9a0 2023-11-28T17:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.976+0000 7f657a7fc640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f656806b2d0 0x7f656806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.976+0000 7f657a7fc640 1 -- 172.21.15.98:0/2761484000 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f65740aaff0 con 0x7f658415a9a0 2023-11-28T17:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.976+0000 7f658982c640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f656806b2d0 0x7f656806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.976+0000 7f658982c640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f656806b2d0 0x7f656806d790 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f65800098b0 tx=0x7f6580008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:49:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:49:59.979+0000 7f657a7fc640 1 -- 172.21.15.98:0/2761484000 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f65740dde90 con 0x7f658415a9a0 2023-11-28T17:50:00.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:00.184+0000 7f658a82e640 1 -- 172.21.15.98:0/2761484000 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6584074db0 con 0x7f658415a9a0 2023-11-28T17:50:00.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:00.185+0000 7f657a7fc640 1 -- 172.21.15.98:0/2761484000 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6574075df0 con 0x7f658415a9a0 2023-11-28T17:50:00.186 INFO:teuthology.orchestra.run.smithi098.stderr:listed 0 entries 2023-11-28T17:50:00.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:00.188+0000 7f658a82e640 1 -- 172.21.15.98:0/2761484000 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f656806b2d0 msgr2=0x7f656806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:50:00.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:00.188+0000 7f658a82e640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f656806b2d0 0x7f656806d790 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f65800098b0 tx=0x7f6580008040 comp rx=0 tx=0).stop 2023-11-28T17:50:00.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:00.188+0000 7f658a82e640 1 -- 172.21.15.98:0/2761484000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f658415a9a0 msgr2=0x7f6584156e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:50:00.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:00.188+0000 7f658a82e640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f658415a9a0 0x7f6584156e80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6574001eb0 tx=0x7f65840c2e60 comp rx=0 tx=0).stop 2023-11-28T17:50:00.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:00.188+0000 7f658a82e640 1 -- 172.21.15.98:0/2761484000 shutdown_connections 2023-11-28T17:50:00.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:00.188+0000 7f658a82e640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.98:6832/2274392793,v1:172.21.15.98:6833/2274392793] conn(0x7f656806b2d0 0x7f656806d790 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:00.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:00.188+0000 7f658a82e640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6584154480 0x7f65841f0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:00.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:00.188+0000 7f658a82e640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f658415a9a0 0x7f6584156e80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:00.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:00.188+0000 7f658a82e640 1 --2- 172.21.15.98:0/2761484000 >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f65700047c0 0x7f65841553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:00.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:00.188+0000 7f658a82e640 1 -- 172.21.15.98:0/2761484000 >> 172.21.15.98:0/2761484000 conn(0x7f65840cf310 msgr2=0x7f658406b560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:50:00.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:00.188+0000 7f658a82e640 1 -- 172.21.15.98:0/2761484000 shutdown_connections 2023-11-28T17:50:00.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:00.188+0000 7f658a82e640 1 -- 172.21.15.98:0/2761484000 wait complete. 2023-11-28T17:50:00.207 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2023-11-28T17:50:00.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:50:00.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-11-28T17:50:00.299 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:50:00.299 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-11-28T17:50:00.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:50:00.299 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T17:50:02.160 INFO:tasks.ceph.osd.0:Sent signal 15 2023-11-28T17:50:02.161 INFO:tasks.ceph.osd.1:Sent signal 15 2023-11-28T17:50:02.162 INFO:tasks.ceph.osd.2:Sent signal 15 2023-11-28T17:50:02.163 INFO:tasks.ceph.osd.3:Sent signal 15 2023-11-28T17:50:02.164 INFO:tasks.ceph.osd.4:Sent signal 15 2023-11-28T17:50:02.165 INFO:tasks.ceph.osd.5:Sent signal 15 2023-11-28T17:50:02.166 INFO:tasks.ceph.osd.6:Sent signal 15 2023-11-28T17:50:02.167 INFO:tasks.ceph.osd.7:Sent signal 15 2023-11-28T17:50:02.170 INFO:tasks.ceph.mds.a:Sent signal 15 2023-11-28T17:50:02.171 INFO:tasks.ceph.mds.c:Sent signal 15 2023-11-28T17:50:02.171 INFO:tasks.ceph.mds.b:Sent signal 15 2023-11-28T17:50:02.171 INFO:tasks.ceph.mds.d:Sent signal 15 2023-11-28T17:50:02.172 INFO:tasks.ceph.mds.f:Sent signal 15 2023-11-28T17:50:02.172 INFO:tasks.ceph.mon.a:Sent signal 15 2023-11-28T17:50:02.172 INFO:tasks.ceph.mon.b:Sent signal 15 2023-11-28T17:50:02.172 INFO:tasks.ceph.mon.c:Sent signal 15 2023-11-28T17:50:02.173 INFO:tasks.ceph.mgr.y:Sent signal 15 2023-11-28T17:50:02.173 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-11-28T17:50:02.174 INFO:tasks.ceph.osd.4.smithi113.stderr:2023-11-28T17:50:02.164+0000 7fd17ec28640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 88747) UID: 0 2023-11-28T17:50:02.174 INFO:tasks.ceph.osd.4.smithi113.stderr:2023-11-28T17:50:02.164+0000 7fd17ec28640 -1 osd.4 24 *** Got signal Terminated *** 2023-11-28T17:50:02.174 INFO:tasks.ceph.osd.4.smithi113.stderr:2023-11-28T17:50:02.164+0000 7fd17ec28640 -1 osd.4 24 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:50:02.175 INFO:tasks.ceph.osd.5.smithi113.stderr:2023-11-28T17:50:02.165+0000 7fb8605b7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 88748) UID: 0 2023-11-28T17:50:02.175 INFO:tasks.ceph.osd.5.smithi113.stderr:2023-11-28T17:50:02.165+0000 7fb8605b7640 -1 osd.5 24 *** Got signal Terminated *** 2023-11-28T17:50:02.175 INFO:tasks.ceph.osd.5.smithi113.stderr:2023-11-28T17:50:02.165+0000 7fb8605b7640 -1 osd.5 24 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:50:02.175 INFO:tasks.ceph.osd.0.smithi098.stderr:2023-11-28T17:50:02.161+0000 7ff521b93640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 88773) UID: 0 2023-11-28T17:50:02.175 INFO:tasks.ceph.osd.0.smithi098.stderr:2023-11-28T17:50:02.161+0000 7ff521b93640 -1 osd.0 24 *** Got signal Terminated *** 2023-11-28T17:50:02.176 INFO:tasks.ceph.osd.0.smithi098.stderr:2023-11-28T17:50:02.161+0000 7ff521b93640 -1 osd.0 24 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:50:02.176 INFO:tasks.ceph.osd.6.smithi113.stderr:2023-11-28T17:50:02.166+0000 7fb5ddcc1640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 88752) UID: 0 2023-11-28T17:50:02.176 INFO:tasks.ceph.osd.6.smithi113.stderr:2023-11-28T17:50:02.166+0000 7fb5ddcc1640 -1 osd.6 24 *** Got signal Terminated *** 2023-11-28T17:50:02.176 INFO:tasks.ceph.osd.6.smithi113.stderr:2023-11-28T17:50:02.166+0000 7fb5ddcc1640 -1 osd.6 24 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:50:02.177 INFO:tasks.ceph.osd.7.smithi113.stderr:2023-11-28T17:50:02.169+0000 7f5180b57640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 88754) UID: 0 2023-11-28T17:50:02.177 INFO:tasks.ceph.osd.7.smithi113.stderr:2023-11-28T17:50:02.169+0000 7f5180b57640 -1 osd.7 24 *** Got signal Terminated *** 2023-11-28T17:50:02.177 INFO:tasks.ceph.osd.7.smithi113.stderr:2023-11-28T17:50:02.169+0000 7f5180b57640 -1 osd.7 24 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:50:02.177 INFO:tasks.ceph.osd.1.smithi098.stderr:2023-11-28T17:50:02.162+0000 7f3f73a05640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 88772) UID: 0 2023-11-28T17:50:02.178 INFO:tasks.ceph.osd.1.smithi098.stderr:2023-11-28T17:50:02.162+0000 7f3f73a05640 -1 osd.1 24 *** Got signal Terminated *** 2023-11-28T17:50:02.178 INFO:tasks.ceph.osd.1.smithi098.stderr:2023-11-28T17:50:02.162+0000 7f3f73a05640 -1 osd.1 24 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:50:02.178 INFO:tasks.ceph.mds.b.smithi113.stderr:2023-11-28T17:50:02.171+0000 7f6ab14dd640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i b (PID: 92291) UID: 0 2023-11-28T17:50:02.178 INFO:tasks.ceph.mds.b.smithi113.stderr:2023-11-28T17:50:02.171+0000 7f6ab14dd640 -1 mds.b *** got signal Terminated *** 2023-11-28T17:50:02.178 INFO:tasks.ceph.osd.2.smithi098.stderr:2023-11-28T17:50:02.163+0000 7f3eca613640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 88779) UID: 0 2023-11-28T17:50:02.179 INFO:tasks.ceph.osd.2.smithi098.stderr:2023-11-28T17:50:02.163+0000 7f3eca613640 -1 osd.2 24 *** Got signal Terminated *** 2023-11-28T17:50:02.179 INFO:tasks.ceph.osd.2.smithi098.stderr:2023-11-28T17:50:02.163+0000 7f3eca613640 -1 osd.2 24 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:50:02.179 INFO:tasks.ceph.mds.d.smithi113.stderr:2023-11-28T17:50:02.172+0000 7f0dd6de1640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i d (PID: 92293) UID: 0 2023-11-28T17:50:02.179 INFO:tasks.ceph.mds.d.smithi113.stderr:2023-11-28T17:50:02.172+0000 7f0dd6de1640 -1 mds.d *** got signal Terminated *** 2023-11-28T17:50:02.180 INFO:tasks.ceph.osd.3.smithi098.stderr:2023-11-28T17:50:02.164+0000 7f9649875640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 88781) UID: 0 2023-11-28T17:50:02.180 INFO:tasks.ceph.osd.3.smithi098.stderr:2023-11-28T17:50:02.164+0000 7f9649875640 -1 osd.3 24 *** Got signal Terminated *** 2023-11-28T17:50:02.180 INFO:tasks.ceph.osd.3.smithi098.stderr:2023-11-28T17:50:02.164+0000 7f9649875640 -1 osd.3 24 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:50:02.180 INFO:tasks.ceph.mds.f.smithi113.stderr:2023-11-28T17:50:02.172+0000 7f18de66c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i f (PID: 92295) UID: 0 2023-11-28T17:50:02.181 INFO:tasks.ceph.mds.f.smithi113.stderr:2023-11-28T17:50:02.172+0000 7f18de66c640 -1 mds.f *** got signal Terminated *** 2023-11-28T17:50:02.181 INFO:tasks.ceph.mon.b.smithi113.stderr:2023-11-28T17:50:02.173+0000 7f53d8023640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 92748) UID: 0 2023-11-28T17:50:02.181 INFO:tasks.ceph.mds.a.smithi098.stderr:2023-11-28T17:50:02.171+0000 7f1b5dfe1640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i a (PID: 92456) UID: 0 2023-11-28T17:50:02.182 INFO:tasks.ceph.mds.a.smithi098.stderr:2023-11-28T17:50:02.171+0000 7f1b5dfe1640 -1 mds.a *** got signal Terminated *** 2023-11-28T17:50:02.182 INFO:tasks.ceph.mon.c.smithi113.stderr:2023-11-28T17:50:02.173+0000 7f07c0c6e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 92936) UID: 0 2023-11-28T17:50:02.182 INFO:tasks.ceph.mon.c.smithi113.stderr:2023-11-28T17:50:02.173+0000 7f07c0c6e640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-11-28T17:50:02.182 INFO:tasks.ceph.mds.c.smithi098.stderr:2023-11-28T17:50:02.172+0000 7fa2a71e7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i c (PID: 92459) UID: 0 2023-11-28T17:50:02.182 INFO:tasks.ceph.mds.c.smithi098.stderr:2023-11-28T17:50:02.172+0000 7fa2a71e7640 -1 mds.c *** got signal Terminated *** 2023-11-28T17:50:02.183 INFO:tasks.ceph.mon.a.smithi098.stderr:2023-11-28T17:50:02.173+0000 7fb098e23640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 104793) UID: 0 2023-11-28T17:50:02.183 INFO:tasks.ceph.mon.a.smithi098.stderr:2023-11-28T17:50:02.173+0000 7fb098e23640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-11-28T17:50:02.183 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:50:02.174+0000 7feeeaa49640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i y (PID: 88530) UID: 0 2023-11-28T17:50:02.183 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-11-28T17:50:02.174+0000 7feeeaa49640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-11-28T17:50:02.184 INFO:tasks.ceph.mon.b.smithi113.stderr:2023-11-28T17:50:02.173+0000 7f53d8023640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-11-28T17:50:02.376 INFO:tasks.ceph.mgr.x.smithi113.stderr:daemon-helper: command crashed with signal 15 2023-11-28T17:50:07.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:50:07.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.792+0000 7f2d875d5640 1 Processor -- start 2023-11-28T17:50:07.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.792+0000 7f2d875d5640 1 -- start start 2023-11-28T17:50:07.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.793+0000 7f2d875d5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d800cf6d0 0x7f2d800c7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:07.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.793+0000 7f2d875d5640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d8015a590 0x7f2d80153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:07.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.793+0000 7f2d875d5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015aa10 con 0x7f2d800cfab0 2023-11-28T17:50:07.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.793+0000 7f2d875d5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d800cf6d0 2023-11-28T17:50:07.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.793+0000 7f2d875d5640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d8015a590 2023-11-28T17:50:07.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.793+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d800cf6d0 msgr2=0x7f2d800c7600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:07.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.793+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d800cfab0 legacy=0x7f2d80159e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:07.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.793+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d8015a590 msgr2=0x7f2d80153cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:07.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.793+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d800cf6d0 0x7f2d800c7600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:07.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.793+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d8015a590 0x7f2d80153cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.994+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d800cfab0 legacy=0x7f2d80159e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.994+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d800cf6d0 msgr2=0x7f2d800c7600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.994+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d800cf6d0 0x7f2d800c7600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.994+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d8015a590 msgr2=0x7f2d80153cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:07.994+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d8015a590 0x7f2d80153cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:08.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:08.395+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d800cf6d0 msgr2=0x7f2d800c7600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:08.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:08.395+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d800cfab0 legacy=0x7f2d80159e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:08.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:08.395+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d8015a590 msgr2=0x7f2d80153cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:08.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:08.395+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d800cf6d0 0x7f2d800c7600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:08.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:08.395+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d8015a590 0x7f2d80153cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:09.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:09.196+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d800cf6d0 msgr2=0x7f2d800c7600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:09.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:09.196+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d800cfab0 legacy=0x7f2d80159e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:09.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:09.196+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d800cf6d0 0x7f2d800c7600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:09.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:09.196+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d8015a590 msgr2=0x7f2d80153cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:09.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:09.196+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d8015a590 0x7f2d80153cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:10.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:50:10.458 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:50:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:10.794+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d8015a590 msgr2=0x7f2d80153cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:10.794+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d8015a590 0x7f2d80153cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:10.794+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d800cf6d0 msgr2=0x7f2d800c7600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:10.794+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d800cf6d0 0x7f2d800c7600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:10.794+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d800cfab0 legacy=0x7f2d80159e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:10.794+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:10.794+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015d710 con 0x7f2d64003690 2023-11-28T17:50:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:10.794+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d640072e0 2023-11-28T17:50:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:10.794+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64003690 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:10.794+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:10.794+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:10.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:10.995+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64003690 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:10.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:10.995+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:10.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:10.995+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:11.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:11.396+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64003690 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:11.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:11.396+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:11.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:11.396+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:12.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:12.197+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64003690 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:12.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:12.197+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:12.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:12.197+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.795+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.795+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.795+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64003690 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.795+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.795+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:50:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.795+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015aa10 con 0x7f2d6400a330 2023-11-28T17:50:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.795+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015d710 con 0x7f2d64009eb0 2023-11-28T17:50:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.795+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d640072e0 2023-11-28T17:50:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.795+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.795+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400a330 legacy=0x7f2d6400d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.795+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.795+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.996+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400a330 legacy=0x7f2d6400d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.996+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.996+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:13.996+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:14.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:14.397+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400a330 legacy=0x7f2d6400d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:14.397+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:14.397+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:14.397+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:15.197+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400a330 legacy=0x7f2d6400d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:15.197+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:15.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:15.197+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:15.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:15.197+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.794+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.794+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.794+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.794+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400a330 legacy=0x7f2d6400d0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.794+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400e7c0 0x7f2d64010bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.794+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640111b0 0x7f2d640135d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.794+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d640111b0 2023-11-28T17:50:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.794+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d6400e7c0 2023-11-28T17:50:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.794+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400e7c0 msgr2=0x7f2d64010bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.794+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640111b0 msgr2=0x7f2d640135d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:16.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.794+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400e7c0 0x7f2d64010bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:16.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.794+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640111b0 0x7f2d640135d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:16.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.995+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640111b0 msgr2=0x7f2d640135d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:16.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.995+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640111b0 0x7f2d640135d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:16.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.995+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400e7c0 msgr2=0x7f2d64010bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:16.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:16.995+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400e7c0 0x7f2d64010bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:17.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:17.396+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640111b0 msgr2=0x7f2d640135d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:17.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:17.396+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640111b0 0x7f2d640135d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:17.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:17.396+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400e7c0 msgr2=0x7f2d64010bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:17.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:17.396+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400e7c0 0x7f2d64010bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:18.197+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640111b0 msgr2=0x7f2d640135d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:18.197+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640111b0 0x7f2d640135d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:18.197+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400e7c0 msgr2=0x7f2d64010bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:18.197+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400e7c0 0x7f2d64010bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:19.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.795+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400e7c0 msgr2=0x7f2d64010bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:19.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.795+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400e7c0 0x7f2d64010bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:19.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.795+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640111b0 msgr2=0x7f2d640135d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:19.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.795+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640111b0 0x7f2d640135d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:19.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.795+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:50:19.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.795+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:19.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.795+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64011630 0x7f2d6400eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:19.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.795+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6400aa60 con 0x7f2d64009eb0 2023-11-28T17:50:19.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.795+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d64011630 2023-11-28T17:50:19.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.795+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d640072e0 2023-11-28T17:50:19.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.795+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:19.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.795+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64011630 msgr2=0x7f2d6400eba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:19.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.795+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:19.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.795+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64011630 0x7f2d6400eba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:19.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.795+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:19.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.996+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:19.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.996+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64011630 msgr2=0x7f2d6400eba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:19.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.996+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:19.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.996+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64011630 0x7f2d6400eba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:19.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:19.996+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:20.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:20.397+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:20.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:20.397+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64011630 msgr2=0x7f2d6400eba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:20.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:20.397+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:20.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:20.397+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64011630 0x7f2d6400eba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:20.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:20.397+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:21.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:21.198+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:21.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:21.198+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64011630 msgr2=0x7f2d6400eba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:21.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:21.198+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:21.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:21.198+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64011630 0x7f2d6400eba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:21.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:21.198+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:22.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:22.795+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:22.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:22.795+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:22.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:22.795+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64011630 msgr2=0x7f2d6400eba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:22.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:22.795+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64011630 0x7f2d6400eba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:22.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:22.795+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:22.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:22.795+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64010210 0x7f2d64010620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:22.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:22.795+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015aa10 con 0x7f2d64010cf0 2023-11-28T17:50:22.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:22.795+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d64010210 2023-11-28T17:50:22.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:22.795+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d64010210 msgr2=0x7f2d64010620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:22.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:22.795+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64010210 0x7f2d64010620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:22.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:22.795+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64010cf0 legacy=0x7f2d6400f0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:22.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:22.996+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d64010210 msgr2=0x7f2d64010620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:22.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:22.996+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64010210 0x7f2d64010620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:22.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:22.996+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64010cf0 legacy=0x7f2d6400f0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:23.397+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64010cf0 legacy=0x7f2d6400f0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:23.397+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d64010210 msgr2=0x7f2d64010620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:23.397+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64010210 0x7f2d64010620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:24.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:24.198+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64010cf0 legacy=0x7f2d6400f0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:24.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:24.198+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d64010210 msgr2=0x7f2d64010620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:24.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:24.198+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64010210 0x7f2d64010620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:25.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.796+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d64010210 msgr2=0x7f2d64010620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:25.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.796+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64010210 0x7f2d64010620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:25.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.796+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64010cf0 legacy=0x7f2d6400f0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:25.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.796+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:25.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.796+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:50:25.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.796+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6400aa60 con 0x7f2d64010210 2023-11-28T17:50:25.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.796+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015aa10 con 0x7f2d64009eb0 2023-11-28T17:50:25.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.796+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d640072e0 2023-11-28T17:50:25.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.796+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:25.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.796+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64010210 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:25.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.796+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:25.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.796+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d6400f890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:25.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.997+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64010210 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.997+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.997+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:25.997+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d6400f890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:26.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:26.398+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64010210 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:26.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:26.398+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:26.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:26.398+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d6400f890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:26.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:26.398+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:27.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:27.199+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64010210 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:27.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:27.199+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:27.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:27.199+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d6400f890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:27.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:27.199+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:28.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.796+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:28.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.796+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640072e0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.796+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d6400f890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.796+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64010210 legacy=0x7f2d64005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.796+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 0x7f2d640105f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.796+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 0x7f2d6400e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.796+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d640135a0 2023-11-28T17:50:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.796+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d64010cf0 2023-11-28T17:50:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.796+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 msgr2=0x7f2d640105f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.796+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 msgr2=0x7f2d6400e160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.796+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 0x7f2d640105f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.796+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 0x7f2d6400e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:28.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.997+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 msgr2=0x7f2d640105f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:28.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.997+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 msgr2=0x7f2d6400e160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:28.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.997+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 0x7f2d6400e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:28.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:28.997+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 0x7f2d640105f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:29.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:29.398+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 msgr2=0x7f2d6400e160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:29.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:29.398+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 msgr2=0x7f2d640105f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:29.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:29.398+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 0x7f2d6400e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:29.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:29.398+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 0x7f2d640105f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:30.200+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 msgr2=0x7f2d6400e160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:30.200+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 msgr2=0x7f2d640105f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:30.200+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 0x7f2d6400e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:30.200+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 0x7f2d640105f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:31.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.797+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 msgr2=0x7f2d640105f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:31.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.797+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 0x7f2d640105f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:31.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.797+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 msgr2=0x7f2d6400e160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:31.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.797+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 0x7f2d6400e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:31.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.797+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:50:31.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.797+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640072e0 0x7f2d640105f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:31.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.797+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015d710 con 0x7f2d64010210 2023-11-28T17:50:31.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.797+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6400aa60 con 0x7f2d64009eb0 2023-11-28T17:50:31.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.797+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d640072e0 2023-11-28T17:50:31.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.797+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64010210 legacy=0x7f2d64005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:31.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.797+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640072e0 msgr2=0x7f2d640105f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:31.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.797+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d64006200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:31.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.797+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640072e0 0x7f2d640105f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:31.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.998+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64010210 legacy=0x7f2d64005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:31.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.998+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640072e0 msgr2=0x7f2d640105f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:32.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.998+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d64006200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:32.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:31.998+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640072e0 0x7f2d640105f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:32.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:32.399+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64010210 legacy=0x7f2d64005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:32.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:32.399+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640072e0 msgr2=0x7f2d640105f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:32.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:32.399+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d64006200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:32.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:32.399+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640072e0 0x7f2d640105f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:33.200+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64010210 legacy=0x7f2d64005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:33.200+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640072e0 msgr2=0x7f2d640105f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:33.200+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640072e0 0x7f2d640105f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:33.200+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d64006200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.798+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640072e0 msgr2=0x7f2d640105f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.798+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640072e0 0x7f2d640105f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.798+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d64009eb0 legacy=0x7f2d64006200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:34.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.798+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64010210 legacy=0x7f2d64005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.798+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d640097a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.798+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d640135a0 0x7f2d6400e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.798+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d640135a0 2023-11-28T17:50:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.798+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d64010cf0 2023-11-28T17:50:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.798+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.798+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d640135a0 msgr2=0x7f2d6400e160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.798+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.798+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d640135a0 0x7f2d6400e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.999+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.999+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d640135a0 msgr2=0x7f2d6400e160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.999+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d640135a0 0x7f2d6400e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:34.999+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:35.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:35.400+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:35.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:35.400+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d640135a0 msgr2=0x7f2d6400e160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:35.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:35.400+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d640135a0 0x7f2d6400e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:35.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:35.400+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:36.202+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:36.202+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:36.202+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d640135a0 msgr2=0x7f2d6400e160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:36.202+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d640135a0 0x7f2d6400e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.798+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.798+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.798+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d640135a0 msgr2=0x7f2d6400e160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.798+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d640135a0 0x7f2d6400e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.798+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d640097a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.798+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:50:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.798+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015aa10 con 0x7f2d64009e70 2023-11-28T17:50:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.798+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015d710 con 0x7f2d640072e0 2023-11-28T17:50:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.798+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d64010cf0 2023-11-28T17:50:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.798+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.798+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64009e70 legacy=0x7f2d6400d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.798+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.798+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:38.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.999+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64009e70 legacy=0x7f2d6400d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.999+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.999+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:38.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:37.999+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:38.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:38.400+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64009e70 legacy=0x7f2d6400d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:38.400+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:38.400+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:38.400+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:39.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:39.201+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64009e70 legacy=0x7f2d6400d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:39.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:39.201+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:39.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:39.201+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:39.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:39.201+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:40.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:50:40.613 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:50:40.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:40.799+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d640097a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:40.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:40.799+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d640097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:40.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:40.799+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:40.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:40.799+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64009e70 legacy=0x7f2d6400d0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:40.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:40.799+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 0x7f2d64006b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:40.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:40.799+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400d7b0 0x7f2d6400dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:40.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:40.799+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d640135a0 2023-11-28T17:50:40.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:40.799+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d6400d7b0 2023-11-28T17:50:40.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:40.799+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 msgr2=0x7f2d64006b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:40.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:40.799+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 0x7f2d64006b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:40.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:40.799+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400d7b0 msgr2=0x7f2d6400dbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:40.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:40.799+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400d7b0 0x7f2d6400dbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:41.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:41.000+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 msgr2=0x7f2d64006b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:41.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:41.000+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 0x7f2d64006b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:41.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:41.000+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400d7b0 msgr2=0x7f2d6400dbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:41.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:41.000+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400d7b0 0x7f2d6400dbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:41.401+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 msgr2=0x7f2d64006b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:41.401+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 0x7f2d64006b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:41.401+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400d7b0 msgr2=0x7f2d6400dbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:41.401+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400d7b0 0x7f2d6400dbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:42.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:42.203+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 msgr2=0x7f2d64006b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:42.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:42.203+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 0x7f2d64006b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:42.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:42.203+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400d7b0 msgr2=0x7f2d6400dbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:42.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:42.203+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400d7b0 0x7f2d6400dbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:43.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:43.799+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400d7b0 msgr2=0x7f2d6400dbc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:43.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:43.799+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400d7b0 0x7f2d6400dbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:43.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:43.799+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 msgr2=0x7f2d64006b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:43.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:43.799+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d640135a0 0x7f2d64006b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:43.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:43.799+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:50:43.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:43.799+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64017b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:43.799+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6400aa60 con 0x7f2d64009e70 2023-11-28T17:50:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:43.799+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015aa10 con 0x7f2d640072e0 2023-11-28T17:50:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:43.799+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400e100 2023-11-28T17:50:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:43.799+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:43.799+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64009e70 legacy=0x7f2d6400d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:43.799+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64017b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:43.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:43.799+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64017b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:44.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:44.000+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:44.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:44.000+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64009e70 legacy=0x7f2d6400d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:44.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:44.000+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64017b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:44.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:44.000+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64017b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:44.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:44.401+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:44.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:44.401+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64009e70 legacy=0x7f2d6400d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:44.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:44.401+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64017b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:44.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:44.401+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64017b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:45.203+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:45.203+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64009e70 legacy=0x7f2d6400d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:45.203+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64017b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:45.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:45.203+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64017b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:46.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:46.800+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64017b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:46.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:46.800+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64017b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:46.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:46.800+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:46.800+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64009e70 legacy=0x7f2d6400d0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:46.800+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:46.800+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400dcf0 0x7f2d64006b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:46.800+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400dcf0 2023-11-28T17:50:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:46.800+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d64010cf0 2023-11-28T17:50:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:46.800+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d6400d7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:46.800+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400d7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:46.800+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400dcf0 msgr2=0x7f2d64006b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:46.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:46.800+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400dcf0 0x7f2d64006b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:47.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:47.001+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d6400d7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:47.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:47.001+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400dcf0 msgr2=0x7f2d64006b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:47.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:47.001+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400d7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:47.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:47.001+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400dcf0 0x7f2d64006b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:47.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:47.402+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d6400d7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:47.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:47.402+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400dcf0 msgr2=0x7f2d64006b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:47.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:47.402+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400dcf0 0x7f2d64006b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:47.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:47.402+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400d7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:48.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:48.203+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d6400d7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:48.203+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400dcf0 msgr2=0x7f2d64006b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:48.203+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400dcf0 0x7f2d64006b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:48.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:48.203+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400d7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:49.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:49.800+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d6400d7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:49.800+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400d7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:49.800+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400dcf0 msgr2=0x7f2d64006b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:49.800+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400dcf0 0x7f2d64006b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:49.800+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 0x7f2d64018070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:49.800+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:50:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:49.800+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64017b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:49.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:49.800+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015d710 con 0x7f2d640072e0 2023-11-28T17:50:49.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:49.800+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400e100 2023-11-28T17:50:49.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:49.800+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d64007790 2023-11-28T17:50:49.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:49.800+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 msgr2=0x7f2d64018070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:49.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:49.801+0000 7f2d85b72640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64017b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:49.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:49.801+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 0x7f2d64018070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:49.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:49.801+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:49.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:49.801+0000 7f2d85b72640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64017b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:50.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:50.002+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 msgr2=0x7f2d64018070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:50.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:50.002+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:50.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:50.002+0000 7f2d85b72640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64017b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:50.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:50.002+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 0x7f2d64018070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:50.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:50.002+0000 7f2d85b72640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64017b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:50.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:50.403+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 msgr2=0x7f2d64018070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:50.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:50.403+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:50.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:50.403+0000 7f2d85b72640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64017b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:50.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:50.403+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 0x7f2d64018070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:50.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:50.403+0000 7f2d85b72640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64017b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:51.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:51.204+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 msgr2=0x7f2d64018070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:51.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:51.204+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:51.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:51.204+0000 7f2d85b72640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64017b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:51.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:51.204+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 0x7f2d64018070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:51.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:51.204+0000 7f2d85b72640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64017b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:52.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:52.800+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 msgr2=0x7f2d64018070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:52.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:52.800+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 0x7f2d64018070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:52.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:52.800+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64017b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:52.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:52.800+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64017b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:52.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:52.800+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:52.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:52.800+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:52.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:52.800+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6400aa60 con 0x7f2d64009e70 2023-11-28T17:50:52.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:52.800+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d64010cf0 2023-11-28T17:50:52.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:52.801+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64009e70 legacy=0x7f2d6400b050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:52.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:52.801+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d6400c770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:52.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:52.801+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400c770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:53.002+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64009e70 legacy=0x7f2d6400b050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:53.002+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d6400c770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:53.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:53.002+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400c770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:53.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:53.403+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64009e70 legacy=0x7f2d6400b050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:53.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:53.403+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d6400c770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:53.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:53.403+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400c770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:54.204+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64009e70 legacy=0x7f2d6400b050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:54.204+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d6400c770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:54.204+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400c770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:55.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:55.801+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d6400c770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:55.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:55.801+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400c770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:55.801+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64009e70 legacy=0x7f2d6400b050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:55.801+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:50:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:55.801+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 0x7f2d64018070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:55.801+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015d710 con 0x7f2d6400ccb0 2023-11-28T17:50:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:55.801+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6400aa60 con 0x7f2d640072e0 2023-11-28T17:50:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:55.801+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d64007790 2023-11-28T17:50:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:55.801+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:55.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:55.801+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:55.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:55.801+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 msgr2=0x7f2d64018070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:55.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:55.801+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 0x7f2d64018070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:56.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:56.002+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:56.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:56.002+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:56.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:56.002+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 msgr2=0x7f2d64018070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:56.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:56.002+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 0x7f2d64018070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:56.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:56.403+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:56.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:56.403+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:56.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:56.403+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 msgr2=0x7f2d64018070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:56.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:56.403+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 0x7f2d64018070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:57.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:57.204+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:50:57.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:57.205+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 msgr2=0x7f2d64018070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:50:57.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:57.205+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:50:57.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:57.205+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 0x7f2d64018070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:50:58.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:58.801+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 msgr2=0x7f2d64018070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:58.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:58.801+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64007790 0x7f2d64018070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:50:58.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:58.801+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:58.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:58.801+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:50:58.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:58.801+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:58.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:58.801+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64006a90 0x7f2d6400e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:50:58.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:58.801+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d64006a90 2023-11-28T17:50:58.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:58.801+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d64010cf0 2023-11-28T17:50:58.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:58.801+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d64006a90 msgr2=0x7f2d6400e100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:58.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:58.801+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:58.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:58.802+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64006a90 0x7f2d6400e100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:58.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:58.802+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:50:59.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:59.003+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d64006a90 msgr2=0x7f2d6400e100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:59.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:59.003+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:59.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:59.003+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64006a90 0x7f2d6400e100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:59.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:59.003+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:50:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:59.404+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d64006a90 msgr2=0x7f2d6400e100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:50:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:59.404+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64006a90 0x7f2d6400e100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:50:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:59.404+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:50:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:50:59.404+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:00.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:00.205+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d64006a90 msgr2=0x7f2d6400e100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:00.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:00.205+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64006a90 0x7f2d6400e100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:00.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:00.205+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:00.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:00.205+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:01.801+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:01.801+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64010cf0 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:01.801+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d64006a90 msgr2=0x7f2d6400e100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:01.801+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64006a90 0x7f2d6400e100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:01.801+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64017b30 0x7f2d6400b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:01.801+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:51:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:01.801+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015aa10 con 0x7f2d6400ccb0 2023-11-28T17:51:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:01.801+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015d710 con 0x7f2d640072e0 2023-11-28T17:51:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:01.801+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d64017b30 2023-11-28T17:51:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:01.802+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:01.802+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d64017b30 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:01.802+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:01.802+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64017b30 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:02.003+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:02.003+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d64017b30 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:02.003+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64017b30 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:02.003+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:02.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:02.404+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:02.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:02.404+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d64017b30 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:02.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:02.404+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64017b30 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:02.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:02.404+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:03.205+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:03.205+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d64017b30 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:03.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:03.205+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64017b30 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:03.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:03.205+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:04.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:04.802+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d64017b30 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:04.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:04.802+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d64017b30 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:04.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:04.802+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:04.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:04.802+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:04.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:04.802+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 0x7f2d6400d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:04.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:04.802+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017f10 0x7f2d6400e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:04.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:04.802+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d64010cf0 2023-11-28T17:51:04.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:04.802+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d64017f10 2023-11-28T17:51:04.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:04.802+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017f10 msgr2=0x7f2d6400e100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:04.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:04.802+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:04.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:04.802+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017f10 0x7f2d6400e100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:04.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:04.802+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:05.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:05.003+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:05.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:05.003+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017f10 msgr2=0x7f2d6400e100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:05.003+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:05.003+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017f10 0x7f2d6400e100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:05.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:05.404+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:05.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:05.404+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017f10 msgr2=0x7f2d6400e100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:05.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:05.404+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:05.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:05.404+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017f10 0x7f2d6400e100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:06.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:06.205+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017f10 msgr2=0x7f2d6400e100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:06.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:06.205+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:06.205+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017f10 0x7f2d6400e100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:06.205+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:07.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:07.802+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017f10 msgr2=0x7f2d6400e100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:07.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:07.802+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017f10 0x7f2d6400e100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:07.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:07.802+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:07.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:07.802+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64010cf0 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:07.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:07.802+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400b560 0x7f2d6400b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:07.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:07.802+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:51:07.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:07.802+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6400aa60 con 0x7f2d6400ccb0 2023-11-28T17:51:07.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:07.802+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015aa10 con 0x7f2d640072e0 2023-11-28T17:51:07.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:07.802+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400b560 2023-11-28T17:51:07.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:07.802+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400b560 msgr2=0x7f2d6400b940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:07.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:07.803+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400b560 0x7f2d6400b940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:07.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:07.803+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:07.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:07.803+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:08.004+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:08.004+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400b560 msgr2=0x7f2d6400b940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:08.004+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:08.004+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400b560 0x7f2d6400b940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:08.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:08.405+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400b560 msgr2=0x7f2d6400b940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:08.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:08.405+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:08.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:08.405+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:08.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:08.405+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400b560 0x7f2d6400b940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:09.206+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400b560 msgr2=0x7f2d6400b940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:09.206+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400b560 0x7f2d6400b940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:09.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:09.206+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:09.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:09.206+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:10.759 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:51:10.768 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:51:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:10.802+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400b560 msgr2=0x7f2d6400b940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:10.802+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400b560 0x7f2d6400b940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:10.802+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:10.802+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:10.802+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640097a0 0x7f2d6400b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:10.802+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:10.802+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400e100 2023-11-28T17:51:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:10.802+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d640097a0 2023-11-28T17:51:10.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:10.803+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:10.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:10.803+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:10.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:10.803+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640097a0 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:10.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:10.803+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640097a0 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:11.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:11.004+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:11.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:11.004+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:11.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:11.004+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640097a0 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:11.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:11.004+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640097a0 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:11.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:11.405+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:11.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:11.405+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:11.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:11.405+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640097a0 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:11.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:11.405+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640097a0 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:12.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:12.206+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:12.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:12.206+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:12.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:12.206+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640097a0 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:12.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:12.206+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640097a0 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:13.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:13.803+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640097a0 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:13.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:13.803+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640097a0 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:13.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:13.803+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:13.803+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:13.803+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017b30 0x7f2d6400be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:13.803+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:51:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:13.803+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:13.803+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015d710 con 0x7f2d6400ccb0 2023-11-28T17:51:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:13.803+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400e100 2023-11-28T17:51:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:13.803+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d64017b30 2023-11-28T17:51:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:13.803+0000 7f2d85b72640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:13.803+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:13.803+0000 7f2d85b72640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:13.803+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017b30 msgr2=0x7f2d6400be80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:13.803+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017b30 0x7f2d6400be80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:14.004+0000 7f2d85b72640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:14.004+0000 7f2d85b72640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:14.004+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:14.004+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017b30 msgr2=0x7f2d6400be80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:14.004+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017b30 0x7f2d6400be80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:14.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:14.405+0000 7f2d85b72640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:14.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:14.405+0000 7f2d85b72640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:14.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:14.405+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:14.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:14.405+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017b30 msgr2=0x7f2d6400be80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:14.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:14.405+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017b30 0x7f2d6400be80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:15.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:15.206+0000 7f2d85b72640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:15.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:15.206+0000 7f2d85b72640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:15.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:15.206+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:15.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:15.206+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017b30 msgr2=0x7f2d6400be80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:15.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:15.206+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017b30 0x7f2d6400be80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:16.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:16.803+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017b30 msgr2=0x7f2d6400be80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:16.803+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d64017b30 0x7f2d6400be80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:16.803+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:16.803+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:16.803+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400ccb0 legacy=0x7f2d6400d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:16.803+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:16.803+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6400aa60 con 0x7f2d640072e0 2023-11-28T17:51:16.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:16.803+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d64017f10 2023-11-28T17:51:16.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:16.804+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:16.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:16.804+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:16.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:16.804+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:17.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:17.005+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:17.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:17.005+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:17.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:17.005+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:17.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:17.406+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:17.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:17.406+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:17.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:17.406+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:18.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:18.207+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:18.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:18.207+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:18.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:18.207+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:19.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:19.803+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:19.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:19.803+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:19.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:19.803+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640072e0 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:19.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:19.803+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:19.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:19.803+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:51:19.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:19.803+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:19.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:19.804+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015d710 con 0x7f2d6400b020 2023-11-28T17:51:19.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:19.804+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400e100 2023-11-28T17:51:19.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:19.804+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d64017f10 2023-11-28T17:51:19.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:19.804+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b020 legacy=0x7f2d6400b400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:19.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:19.804+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400c3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:19.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:19.804+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:19.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:19.804+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:19.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:19.804+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:20.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:20.005+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:20.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:20.005+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b020 legacy=0x7f2d6400b400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:20.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:20.005+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400c3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:20.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:20.005+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:20.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:20.005+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:20.406+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:20.406+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b020 legacy=0x7f2d6400b400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:20.406+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:20.406+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400c3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:20.406+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:21.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:21.207+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:21.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:21.207+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b020 legacy=0x7f2d6400b400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:21.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:21.207+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400c3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:21.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:21.207+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:21.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:21.207+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:22.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:22.804+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 msgr2=0x7f2d6400d7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:22.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:22.804+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:22.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:22.804+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400c3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:22.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:22.804+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:22.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:22.804+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b020 legacy=0x7f2d6400b400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:22.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:22.804+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d6400c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:22.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:22.804+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6400aa60 con 0x7f2d64017b30 2023-11-28T17:51:22.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:22.804+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400bbd0 2023-11-28T17:51:22.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:22.804+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d6400c900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:22.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:22.804+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64017b30 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:22.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:22.804+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d6400c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:23.005+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64017b30 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:23.005+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d6400c900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:23.005+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d6400c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:23.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:23.406+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d6400c900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:23.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:23.406+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64017b30 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:23.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:23.406+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d6400c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:24.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:24.207+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64017b30 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:24.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:24.207+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d6400c900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:24.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:24.207+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d6400c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:25.804+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d6400c900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:25.804+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d6400c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:25.804+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64017b30 legacy=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:25.804+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:25.804+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:51:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:25.804+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015d710 con 0x7f2d6400bfe0 2023-11-28T17:51:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:25.804+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6400aa60 con 0x7f2d6400b020 2023-11-28T17:51:25.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:25.804+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400e100 2023-11-28T17:51:25.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:25.804+0000 7f2d85b72640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400c3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:25.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:25.804+0000 7f2d85b72640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:25.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:25.805+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400bfe0 legacy=0x7f2d6400ce40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:25.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:25.805+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b020 legacy=0x7f2d6400b400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:26.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:26.006+0000 7f2d85b72640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400c3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:26.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:26.006+0000 7f2d85b72640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:26.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:26.006+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400bfe0 legacy=0x7f2d6400ce40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:26.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:26.006+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b020 legacy=0x7f2d6400b400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:26.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:26.407+0000 7f2d85b72640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400c3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:26.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:26.407+0000 7f2d85b72640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:26.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:26.407+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400bfe0 legacy=0x7f2d6400ce40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:26.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:26.407+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b020 legacy=0x7f2d6400b400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:27.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:27.208+0000 7f2d85b72640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400c3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:27.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:27.208+0000 7f2d85b72640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:27.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:27.208+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400bfe0 legacy=0x7f2d6400ce40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:27.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:27.208+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b020 legacy=0x7f2d6400b400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:28.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:28.805+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400c3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:28.805+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:28.805+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b020 legacy=0x7f2d6400b400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:28.805+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400bfe0 legacy=0x7f2d6400ce40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:28.805+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:28.805+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d64010210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:28.805+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d64017f10 2023-11-28T17:51:28.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:28.805+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d6400bbd0 2023-11-28T17:51:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:28.805+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:28.805+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 msgr2=0x7f2d6400c900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:28.805+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d64010210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:28.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:28.805+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:29.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:29.006+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 msgr2=0x7f2d6400c900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:29.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:29.006+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:29.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:29.006+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:29.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:29.006+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d64010210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:29.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:29.407+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:29.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:29.407+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 msgr2=0x7f2d6400c900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:29.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:29.407+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d64010210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:29.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:29.407+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:30.208+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 msgr2=0x7f2d6400c900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:30.208+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:30.208+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d64010210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:30.208+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:31.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:31.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d64010210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:31.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 msgr2=0x7f2d6400c900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:31.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64017f10 0x7f2d6400c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:31.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:31.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:51:31.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d64010210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:31.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400bfe0 0x7f2d6400b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:31.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400e100 2023-11-28T17:51:31.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d6400bfe0 2023-11-28T17:51:31.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d64010750 con 0x7f2d6400bbd0 2023-11-28T17:51:31.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400c3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:31.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400bfe0 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:31.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:31.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400bfe0 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:31.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:31.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:31.804+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d64010210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:32.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:32.005+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400bfe0 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:32.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:32.005+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:32.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:32.005+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400c3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:32.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:32.005+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400bfe0 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:32.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:32.005+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:32.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:32.005+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d64010210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:32.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:32.406+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400bfe0 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:32.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:32.406+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400bfe0 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:32.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:32.406+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:32.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:32.406+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400c3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:32.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:32.406+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d64010210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:32.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:32.406+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:33.207+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400bfe0 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:33.207+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400bfe0 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:33.207+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:33.207+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400c3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:33.207+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:33.207+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d64010210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:34.804+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d64010210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:34.804+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d64010210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:34.804+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400bfe0 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:34.804+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d6400bfe0 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:34.804+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d6400c3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:34.804+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d6400c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:34.804+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015aa10 con 0x7f2d6400b560 2023-11-28T17:51:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:34.804+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015d710 con 0x7f2d6400d910 2023-11-28T17:51:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:34.804+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400b560 legacy=0x7f2d6400eba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:34.804+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400d910 legacy=0x7f2d6400f2b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:35.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:35.006+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400b560 legacy=0x7f2d6400eba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:35.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:35.006+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400d910 legacy=0x7f2d6400f2b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:35.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:35.407+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400b560 legacy=0x7f2d6400eba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:35.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:35.407+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400d910 legacy=0x7f2d6400f2b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:36.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:36.208+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400b560 legacy=0x7f2d6400eba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:36.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:36.208+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400d910 legacy=0x7f2d6400f2b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:37.804+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400d910 legacy=0x7f2d6400f2b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:37.804+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d6400b560 legacy=0x7f2d6400eba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:37.804+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:37.804+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:51:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:37.804+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6400aa60 con 0x7f2d64011590 2023-11-28T17:51:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:37.804+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015aa10 con 0x7f2d6400d910 2023-11-28T17:51:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:37.804+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400bbd0 2023-11-28T17:51:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:37.805+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400d910 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:37.805+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:37.805+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d640101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:37.805+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:38.006+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400d910 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:38.006+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:38.006+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d640101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:38.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:38.006+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:38.407+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400d910 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:38.407+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:38.407+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d640101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:38.407+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:39.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:39.208+0000 7f2d85b72640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400d910 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:39.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:39.208+0000 7f2d84b70640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:39.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:39.208+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d640101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:39.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:39.208+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:40.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:40.805+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d640101f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:40.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:40.805+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:40.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:40.805+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400d910 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:40.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:40.805+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:40.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:40.805+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:40.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:40.805+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:40.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:40.805+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400e100 2023-11-28T17:51:40.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:40.805+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d64013090 2023-11-28T17:51:40.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:40.805+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:40.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:40.805+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:40.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:40.805+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:40.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:40.805+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:40.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:51:40.926 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:51:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:41.006+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:41.006+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:41.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:41.006+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:41.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:41.006+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:41.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:41.407+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:44.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:41.407+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:44.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:41.407+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:44.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:41.407+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:44.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:42.208+0000 7f2d85371640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:44.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:42.208+0000 7f2d85371640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:44.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:42.208+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:44.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:42.208+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:44.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:43.806+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:44.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:43.806+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:44.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:43.806+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:44.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:43.806+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:44.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:43.806+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:44.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:43.806+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:44.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:43.806+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:51:44.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:43.806+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015d710 con 0x7f2d64011590 2023-11-28T17:51:44.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:43.806+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d64013090 2023-11-28T17:51:44.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:43.806+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d6400bbd0 2023-11-28T17:51:44.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:43.806+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:44.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:43.806+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d640101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:44.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:43.806+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:44.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:43.806+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:44.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:43.806+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:44.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:44.007+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:44.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:44.007+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:44.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:44.007+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:44.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:44.007+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d640101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:44.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:44.007+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:44.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:44.408+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:44.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:44.408+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:44.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:44.408+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:44.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:44.408+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d640101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:44.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:44.408+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:45.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:45.209+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:45.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:45.209+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:45.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:45.209+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d640101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:45.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:45.209+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:45.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:45.209+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:46.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:46.806+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d640101f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:46.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:46.806+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:46.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:46.806+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:46.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:46.806+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:46.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:46.806+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:46.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:46.806+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400b560 0x7f2d64004180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:46.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:46.806+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6400aa60 con 0x7f2d6400e100 2023-11-28T17:51:46.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:46.806+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400b560 2023-11-28T17:51:46.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:46.806+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400b560 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:46.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:46.806+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400b560 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:46.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:46.806+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400e100 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:47.007+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400b560 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:47.007+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400b560 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:47.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:47.007+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400e100 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:47.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:47.408+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400b560 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:47.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:47.408+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400b560 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:47.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:47.408+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400e100 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:48.209+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400b560 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:48.209+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400b560 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:48.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:48.209+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400e100 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:49.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:49.806+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400b560 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:49.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:49.806+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400b560 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:49.806+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400e100 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:49.806+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:49.806+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:49.806+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:51:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:49.806+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015d710 con 0x7f2d64011590 2023-11-28T17:51:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:49.806+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d64013090 2023-11-28T17:51:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:49.806+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d6400bbd0 2023-11-28T17:51:49.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:49.806+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:49.806+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d640101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:49.806+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:49.806+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:49.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:49.806+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:50.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:50.007+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:50.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:50.007+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:50.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:50.007+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d640101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:50.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:50.007+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:50.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:50.007+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:50.408+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:50.408+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:50.408+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d640101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:50.408+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:50.408+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:51.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:51.210+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:51.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:51.210+0000 7f2d84b70640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:51.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:51.210+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d640101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:51.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:51.210+0000 7f2d84b70640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:51.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:51.210+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:52.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:52.807+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 msgr2=0x7f2d640101f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:52.807+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400bbd0 0x7f2d640101f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:52.807+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:52.807+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:52.807+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:52.807+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:52.807+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6400aa60 con 0x7f2d6400b560 2023-11-28T17:51:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:52.807+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400e100 2023-11-28T17:51:52.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:52.807+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:52.807+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b560 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:52.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:52.807+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:53.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:53.008+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:53.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:53.008+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:53.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:53.008+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b560 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:53.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:53.409+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:53.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:53.409+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:53.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:53.409+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b560 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:54.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:54.210+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:54.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:54.210+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:54.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:54.210+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b560 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:55.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:55.807+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:55.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:55.807+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:55.807+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b560 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:55.807+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:55.807+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:55.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:55.807+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:51:55.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:55.807+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015d710 con 0x7f2d64011590 2023-11-28T17:51:55.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:55.807+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400e100 2023-11-28T17:51:55.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:55.807+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d64013090 2023-11-28T17:51:55.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:55.807+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:55.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:55.807+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:55.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:55.807+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:55.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:55.807+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:55.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:55.807+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:56.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:56.008+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:56.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:56.008+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:56.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:56.008+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:56.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:56.008+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:56.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:56.008+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:56.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:56.409+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:56.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:56.409+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:56.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:56.409+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:56.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:56.409+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:56.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:56.409+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:51:57.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:57.210+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:51:57.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:57.210+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:57.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:57.210+0000 7f2d85b72640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:51:57.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:57.210+0000 7f2d85371640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:51:57.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:57.210+0000 7f2d85371640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:51:58.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:58.808+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:58.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:58.808+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:58.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:58.808+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:58.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:58.808+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:51:58.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:58.808+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:51:58.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:58.808+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640101f0 0x7f2d6400bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:51:58.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:58.808+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6400aa60 con 0x7f2d6400b560 2023-11-28T17:51:58.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:58.808+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d640101f0 2023-11-28T17:51:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:58.808+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b560 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:58.808+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640101f0 msgr2=0x7f2d6400bbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:58.808+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640101f0 0x7f2d6400bbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:51:59.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:59.009+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b560 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:59.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:59.009+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640101f0 msgr2=0x7f2d6400bbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:59.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:59.009+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640101f0 0x7f2d6400bbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:51:59.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:59.410+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b560 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:51:59.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:59.410+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640101f0 msgr2=0x7f2d6400bbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:51:59.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:51:59.410+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640101f0 0x7f2d6400bbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:00.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:00.212+0000 7f2d85371640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b560 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:00.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:00.212+0000 7f2d84b70640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640101f0 msgr2=0x7f2d6400bbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:00.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:00.212+0000 7f2d84b70640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640101f0 0x7f2d6400bbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:01.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:01.808+0000 7f2d76ffd640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640101f0 msgr2=0x7f2d6400bbd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:01.808+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d640101f0 0x7f2d6400bbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:01.808+0000 7f2d76ffd640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d6400b560 legacy=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:01.808+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:01.808+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:01.808+0000 7f2d85b72640 1 -- reap_dead start 2023-11-28T17:52:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:01.808+0000 7f2d76ffd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8015d710 con 0x7f2d64011590 2023-11-28T17:52:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:01.808+0000 7f2d76ffd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400e100 2023-11-28T17:52:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:01.808+0000 7f2d76ffd640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8015d430 con 0x7f2d64013090 2023-11-28T17:52:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:01.808+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:01.808+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:01.808+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:01.809+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:01.809+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:02.009+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:02.009+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:02.009+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:02.010+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:02.010+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:02.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:02.410+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:02.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:02.410+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:02.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:02.410+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:02.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:02.411+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:02.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:02.411+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:03.211+0000 7f2d85371640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:03.211+0000 7f2d84b70640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:03.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:03.211+0000 7f2d84b70640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:03.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:03.212+0000 7f2d85b72640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:03.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:03.212+0000 7f2d85b72640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:04.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:04.808+0000 7f2d76ffd640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 msgr2=0x7f2d6400b020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:04.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:04.808+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f2d64013090 0x7f2d6400b020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:04.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:04.808+0000 7f2d76ffd640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 msgr2=0x7f2d64004180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:04.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:04.808+0000 7f2d76ffd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d6400e100 0x7f2d64004180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:04.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:04.808+0000 7f2d76ffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f2d64011590 legacy=0x7f2d64011970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:04.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:04.808+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400b560 0x7f2d64003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:04.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:04.808+0000 7f2d76ffd640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6400aa60 con 0x7f2d640101f0 2023-11-28T17:52:04.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:04.808+0000 7f2d76ffd640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8015acf0 con 0x7f2d6400b560 2023-11-28T17:52:04.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:04.809+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400b560 msgr2=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:04.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:04.809+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400b560 0x7f2d64003a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:04.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:04.809+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640101f0 legacy=0x7f2d6400eba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:05.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:05.010+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400b560 msgr2=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:05.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:05.010+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640101f0 legacy=0x7f2d6400eba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:05.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:05.010+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400b560 0x7f2d64003a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:05.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:05.411+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400b560 msgr2=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:05.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:05.411+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640101f0 legacy=0x7f2d6400eba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:05.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:05.411+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400b560 0x7f2d64003a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:06.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:06.212+0000 7f2d85371640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400b560 msgr2=0x7f2d64003a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:06.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:06.212+0000 7f2d84b70640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f2d640101f0 legacy=0x7f2d6400eba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:06.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:06.212+0000 7f2d85371640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f2d6400b560 0x7f2d64003a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:52:07.685 ERROR:tasks.mon_thrash.mon_thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mon_thrash.py", line 252, in do_thrash self._do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mon_thrash.py", line 377, in _do_thrash status = self.mds_cluster.status() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/filesystem.py", line 396, in status return FSStatus(self.mon_manager, epoch) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/filesystem.py", line 88, in __init__ self.map = json.loads(self.get_ceph_cmd_stdout(*cmd)) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' 2023-11-28T17:52:07.696 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2023-11-28T17:52:07.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2023-11-28T17:52:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:07.922+0000 7f9fe6ae0640 1 Processor -- start 2023-11-28T17:52:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:07.922+0000 7f9fe6ae0640 1 -- start start 2023-11-28T17:52:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:07.924+0000 7f9fe6ae0640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fe0157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:07.924+0000 7f9fe6ae0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 0x7f9fe00c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:07.924+0000 7f9fe6ae0640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 0x7f9fe00c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:07.924+0000 7f9fe6ae0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fe015bc10 2023-11-28T17:52:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:07.924+0000 7f9fe6ae0640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fe0151d80 2023-11-28T17:52:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:07.924+0000 7f9fe6ae0640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe006e4d0 con 0x7f9fe00c6140 2023-11-28T17:52:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:07.924+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 msgr2=0x7f9fe00c5c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:07.924+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 0x7f9fe00c5c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:07.924+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 msgr2=0x7f9fe00c4910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:07.924+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fe0157b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:07.924+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 0x7f9fe00c4910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:07.924+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fe0157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:08.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:08.125+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fe0157b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:08.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:08.125+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 msgr2=0x7f9fe00c5c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:08.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:08.125+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 msgr2=0x7f9fe00c4910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:08.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:08.125+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fe0157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:08.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:08.125+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 0x7f9fe00c5c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:08.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:08.125+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 0x7f9fe00c4910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:08.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:08.526+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 msgr2=0x7f9fe00c5c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:08.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:08.526+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fe0157b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:08.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:08.526+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 0x7f9fe00c5c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:08.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:08.526+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 msgr2=0x7f9fe00c4910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:08.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:08.526+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fe0157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:08.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:08.526+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 0x7f9fe00c4910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:09.327+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 msgr2=0x7f9fe00c5c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:09.327+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 0x7f9fe00c5c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:09.327+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 msgr2=0x7f9fe00c4910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:09.327+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fe0157b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:09.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:09.327+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 0x7f9fe00c4910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:09.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:09.327+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fe0157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:10.924+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 msgr2=0x7f9fe00c4910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:10.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:10.924+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 0x7f9fe00c4910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:10.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:10.924+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fe0157b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:10.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:10.924+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fe0157b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:10.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:10.924+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 msgr2=0x7f9fe00c5c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:10.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:10.924+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 0x7f9fe00c5c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:10.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:10.924+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe006e7b0 con 0x7f9fc40073f0 2023-11-28T17:52:10.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:10.924+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc40036b0 2023-11-28T17:52:10.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:10.925+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc40073f0 legacy=0x7f9fc40098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:10.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:10.925+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc40036b0 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:11.099 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:52:11.108 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:52:11.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:11.126+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc40036b0 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:11.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:11.126+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc40073f0 legacy=0x7f9fc40098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:11.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:11.527+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc40073f0 legacy=0x7f9fc40098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:11.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:11.527+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc40036b0 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:12.328+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc40073f0 legacy=0x7f9fc40098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:12.328+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc40036b0 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:13.925+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc40036b0 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:13.925+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc40073f0 legacy=0x7f9fc40098b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:13.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:13.925+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:52:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:13.925+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:13.925+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 0x7f9fc400c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:13.925+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 0x7f9fc400b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:13.925+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fe015bc10 2023-11-28T17:52:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:13.925+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fe0151d80 2023-11-28T17:52:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:13.925+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe006e4d0 con 0x7f9fe00c6140 2023-11-28T17:52:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:13.925+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:13.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:13.925+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 msgr2=0x7f9fc400b110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:13.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:13.925+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 0x7f9fc400b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:13.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:13.925+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:13.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:13.925+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 msgr2=0x7f9fc400c880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:13.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:13.925+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 0x7f9fc400c880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:14.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:14.126+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:14.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:14.126+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 msgr2=0x7f9fc400b110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:14.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:14.126+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 0x7f9fc400b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:14.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:14.126+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:14.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:14.127+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 msgr2=0x7f9fc400c880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:14.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:14.127+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 0x7f9fc400c880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:14.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:14.527+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:14.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:14.527+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 msgr2=0x7f9fc400b110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:14.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:14.527+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 0x7f9fc400b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:14.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:14.527+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:14.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:14.528+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 msgr2=0x7f9fc400c880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:14.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:14.528+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 0x7f9fc400c880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:15.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:15.328+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:15.328+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 msgr2=0x7f9fc400b110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:15.328+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 0x7f9fc400b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:15.328+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:15.329+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 msgr2=0x7f9fc400c880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:15.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:15.329+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 0x7f9fc400c880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:16.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:16.925+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 msgr2=0x7f9fc400b110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:16.925+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fe00c6140 0x7f9fc400b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:16.925+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:16.925+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:16.925+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 msgr2=0x7f9fc400c880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:16.925+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fe015bc10 0x7f9fc400c880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:16.925+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400d360 con 0x7f9fc400b650 2023-11-28T17:52:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:16.925+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe006e7b0 con 0x7f9fc400ba30 2023-11-28T17:52:16.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:16.926+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:16.926+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400ba30 legacy=0x7f9fc40098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:17.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:17.127+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:17.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:17.127+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400ba30 legacy=0x7f9fc40098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:17.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:17.528+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:17.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:17.528+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400ba30 legacy=0x7f9fc40098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:18.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:18.329+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:18.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:18.329+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400ba30 legacy=0x7f9fc40098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:19.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:19.926+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400ba30 legacy=0x7f9fc40098b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:19.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:19.926+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:19.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:19.926+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:52:19.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:19.926+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:19.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:19.926+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc400b650 2023-11-28T17:52:19.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:19.926+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400d360 con 0x7f9fc40078a0 2023-11-28T17:52:19.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:19.926+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fe0151d80 2023-11-28T17:52:19.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:19.926+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:19.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:19.926+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc40078a0 legacy=0x7f9fc4009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:19.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:19.926+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:19.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:19.926+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:20.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:20.127+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:20.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:20.127+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc40078a0 legacy=0x7f9fc4009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:20.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:20.127+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:20.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:20.127+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:20.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:20.528+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:20.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:20.528+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc40078a0 legacy=0x7f9fc4009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:20.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:20.528+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:20.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:20.528+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:21.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:21.329+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:21.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:21.329+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc40078a0 legacy=0x7f9fc4009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:21.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:21.329+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:21.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:21.329+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:22.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:22.926+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:22.926+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:22.926+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc40078a0 legacy=0x7f9fc4009fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:22.926+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:22.926+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b110 0x7f9fc400c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:22.926+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400a6f0 0x7f9fc4014690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:22.926+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc400a6f0 2023-11-28T17:52:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:22.926+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc400b110 2023-11-28T17:52:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:22.927+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b110 msgr2=0x7f9fc400c880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:22.927+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400a6f0 msgr2=0x7f9fc4014690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:22.927+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b110 0x7f9fc400c880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:22.927+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400a6f0 0x7f9fc4014690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:23.128+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400a6f0 msgr2=0x7f9fc4014690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:23.128+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b110 msgr2=0x7f9fc400c880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:23.128+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400a6f0 0x7f9fc4014690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:23.128+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b110 0x7f9fc400c880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:23.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:23.529+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400a6f0 msgr2=0x7f9fc4014690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:23.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:23.529+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400a6f0 0x7f9fc4014690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:23.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:23.529+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b110 msgr2=0x7f9fc400c880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:23.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:23.529+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b110 0x7f9fc400c880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:24.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:24.330+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400a6f0 msgr2=0x7f9fc4014690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:24.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:24.330+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400a6f0 0x7f9fc4014690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:24.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:24.330+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b110 msgr2=0x7f9fc400c880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:24.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:24.330+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b110 0x7f9fc400c880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:25.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:25.927+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b110 msgr2=0x7f9fc400c880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:25.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:25.927+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b110 0x7f9fc400c880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:25.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:25.927+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400a6f0 msgr2=0x7f9fc4014690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:25.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:25.927+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400a6f0 0x7f9fc4014690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:25.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:25.927+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:52:25.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:25.927+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:25.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:25.927+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe006e7b0 con 0x7f9fc400b650 2023-11-28T17:52:25.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:25.927+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc40078a0 2023-11-28T17:52:25.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:25.927+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fe0151d80 2023-11-28T17:52:25.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:25.927+0000 7f9fe507d640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc40078a0 legacy=0x7f9fc4009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:25.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:25.927+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:25.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:25.927+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:25.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:25.927+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:26.128+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:26.128+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:26.128+0000 7f9fe507d640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc40078a0 legacy=0x7f9fc4009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:26.128+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:26.529+0000 7f9fe507d640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc40078a0 legacy=0x7f9fc4009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:26.529+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:26.529+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:26.529+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:27.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:27.330+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:27.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:27.330+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:27.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:27.330+0000 7f9fe507d640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc40078a0 legacy=0x7f9fc4009fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:27.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:27.330+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:28.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:28.927+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:28.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:28.927+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:28.927+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc40078a0 legacy=0x7f9fc4009fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:28.927+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:28.927+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400aad0 0x7f9fc4014690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:28.927+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 0x7f9fc4018fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:28.927+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc400aad0 2023-11-28T17:52:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:28.927+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc400c880 2023-11-28T17:52:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:28.927+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400aad0 msgr2=0x7f9fc4014690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:28.927+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400aad0 0x7f9fc4014690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:28.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:28.927+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 msgr2=0x7f9fc4018fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:28.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:28.928+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 0x7f9fc4018fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:29.128+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400aad0 msgr2=0x7f9fc4014690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:29.128+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400aad0 0x7f9fc4014690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:29.129+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 msgr2=0x7f9fc4018fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:29.129+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 0x7f9fc4018fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:29.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:29.529+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400aad0 msgr2=0x7f9fc4014690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:29.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:29.529+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400aad0 0x7f9fc4014690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:29.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:29.530+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 msgr2=0x7f9fc4018fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:29.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:29.530+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 0x7f9fc4018fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:30.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:30.330+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400aad0 msgr2=0x7f9fc4014690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:30.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:30.330+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400aad0 0x7f9fc4014690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:30.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:30.331+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 msgr2=0x7f9fc4018fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:30.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:30.331+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 0x7f9fc4018fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:31.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:31.927+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 msgr2=0x7f9fc4018fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:31.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:31.927+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 0x7f9fc4018fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:31.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:31.927+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400aad0 msgr2=0x7f9fc4014690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:31.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:31.927+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc400aad0 0x7f9fc4014690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:31.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:31.927+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:52:31.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:31.927+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 0x7f9fc4018fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:31.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:31.927+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:31.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:31.927+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400d360 con 0x7f9fc400b650 2023-11-28T17:52:31.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:31.927+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fe0151d80 2023-11-28T17:52:31.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:31.927+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc400c880 2023-11-28T17:52:31.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:31.928+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:31.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:31.928+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 msgr2=0x7f9fc4018fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:31.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:31.928+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:31.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:31.928+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:31.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:31.928+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 0x7f9fc4018fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:32.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:32.129+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:32.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:32.129+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 msgr2=0x7f9fc4018fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:32.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:32.129+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:32.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:32.129+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:32.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:32.129+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 0x7f9fc4018fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:32.530+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:32.530+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 msgr2=0x7f9fc4018fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:32.530+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:32.530+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 0x7f9fc4018fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:32.530+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:33.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:33.331+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:33.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:33.331+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 msgr2=0x7f9fc4018fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:33.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:33.331+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:33.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:33.331+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:33.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:33.331+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 0x7f9fc4018fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:34.928+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 msgr2=0x7f9fc4018fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:34.928+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c880 0x7f9fc4018fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:34.928+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 msgr2=0x7f9fc400c340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:34.928+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fe0151d80 0x7f9fc400c340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:34.928+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:34.928+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:34.928+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe006e7b0 con 0x7f9fc4013740 2023-11-28T17:52:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:34.928+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4009d60 2023-11-28T17:52:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:34.928+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 msgr2=0x7f9fc400a140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:34.928+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4013740 legacy=0x7f9fc4013b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:34.928+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:35.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:35.129+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 msgr2=0x7f9fc400a140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:35.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:35.129+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:35.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:35.129+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4013740 legacy=0x7f9fc4013b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:35.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:35.530+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 msgr2=0x7f9fc400a140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:35.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:35.530+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:35.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:35.530+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4013740 legacy=0x7f9fc4013b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:36.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:36.331+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 msgr2=0x7f9fc400a140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:36.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:36.331+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:36.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:36.331+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4013740 legacy=0x7f9fc4013b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:37.928+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 msgr2=0x7f9fc400a140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:37.928+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:37.928+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4013740 legacy=0x7f9fc4013b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:37.928+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:52:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:37.928+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:37.928+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400d360 con 0x7f9fc400b650 2023-11-28T17:52:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:37.928+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe006e7b0 con 0x7f9fc4013740 2023-11-28T17:52:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:37.928+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4009d60 2023-11-28T17:52:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:37.928+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 msgr2=0x7f9fc400a140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:37.928+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:37.929+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4013740 legacy=0x7f9fc4013b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:37.929+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:38.130+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 msgr2=0x7f9fc400a140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:38.130+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:38.130+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:38.130+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4013740 legacy=0x7f9fc4013b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:38.531+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 msgr2=0x7f9fc400a140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:38.531+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:38.531+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:38.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:38.531+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4013740 legacy=0x7f9fc4013b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:39.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:39.332+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:39.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:39.332+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 msgr2=0x7f9fc400a140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:39.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:39.332+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:39.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:39.332+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4013740 legacy=0x7f9fc4013b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:40.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:40.929+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 msgr2=0x7f9fc400a140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:40.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:40.929+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:40.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:40.929+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4013740 legacy=0x7f9fc4013b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:40.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:40.929+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:40.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:40.929+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:40.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:40.929+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4014770 0x7f9fc400c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:40.929+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4018fe0 2023-11-28T17:52:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:40.929+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc4014770 2023-11-28T17:52:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:40.929+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4014770 msgr2=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:40.929+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:40.929+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4014770 0x7f9fc400c320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:40.929+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:41.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:41.130+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:41.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:41.130+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4014770 msgr2=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:41.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:41.130+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4014770 0x7f9fc400c320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:41.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:41.130+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:41.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:52:41.260 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:52:41.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:41.531+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:41.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:41.531+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4014770 msgr2=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:41.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:41.531+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4014770 0x7f9fc400c320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:41.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:41.531+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:42.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:42.332+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:42.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:42.332+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4014770 msgr2=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:42.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:42.332+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4014770 0x7f9fc400c320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:42.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:42.332+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:43.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:43.929+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4014770 msgr2=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:43.929+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4014770 0x7f9fc400c320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:43.929+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:43.929+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:43.929+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:52:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:43.929+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:43.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:43.929+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:43.929+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc400b650 2023-11-28T17:52:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:43.929+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4009d60 2023-11-28T17:52:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:43.929+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc4018fe0 2023-11-28T17:52:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:43.929+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:43.929+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 msgr2=0x7f9fc400a140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:43.929+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:43.929+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:43.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:43.929+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:44.129+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:44.129+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 msgr2=0x7f9fc400a140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:44.129+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:44.129+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:44.129+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:44.530+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 msgr2=0x7f9fc400a140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:44.530+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:44.530+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:44.530+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:44.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:44.530+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:45.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:45.331+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 msgr2=0x7f9fc400a140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:45.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:45.331+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:45.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:45.331+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:45.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:45.331+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:45.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:45.331+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:46.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:46.929+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:46.929+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:46.929+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 msgr2=0x7f9fc400a140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:46.929+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4009d60 0x7f9fc400a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:46.929+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:46.929+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4013740 0x7f9fc4013b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:46.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:46.929+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400d360 con 0x7f9fc4014770 2023-11-28T17:52:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:46.929+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4013740 2023-11-28T17:52:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:46.929+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4014770 legacy=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:46.929+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4013740 msgr2=0x7f9fc4013b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:46.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:46.929+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4013740 0x7f9fc4013b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:47.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:47.130+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4014770 legacy=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:47.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:47.130+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4013740 msgr2=0x7f9fc4013b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:47.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:47.130+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4013740 0x7f9fc4013b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:47.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:47.531+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4014770 legacy=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:47.531+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4013740 msgr2=0x7f9fc4013b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:47.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:47.531+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4013740 0x7f9fc4013b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:48.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:48.332+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4013740 msgr2=0x7f9fc4013b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:48.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:48.332+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4014770 legacy=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:48.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:48.332+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4013740 0x7f9fc4013b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:49.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:49.929+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4013740 msgr2=0x7f9fc4013b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:49.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:49.929+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc4013740 0x7f9fc4013b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:49.929+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4014770 legacy=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:49.929+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:52:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:49.929+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:49.929+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc400b650 2023-11-28T17:52:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:49.929+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400d360 con 0x7f9fc4014770 2023-11-28T17:52:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:49.929+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc40088b0 2023-11-28T17:52:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:49.929+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4006260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:49.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:49.929+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:49.929+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4014770 legacy=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:49.929+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:50.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:50.130+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4006260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:50.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:50.130+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:50.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:50.130+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4014770 legacy=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:50.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:50.130+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:50.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:50.531+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4006260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:50.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:50.531+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:50.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:50.531+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4014770 legacy=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:50.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:50.531+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:51.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:51.332+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4006260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:51.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:51.332+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:51.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:51.332+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4014770 legacy=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:51.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:51.332+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:52.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:52.930+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:52.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:52.930+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:52.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:52.930+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4014770 legacy=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:52.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:52.930+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4006260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:52.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:52.930+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:52.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:52.930+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:52.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:52.930+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4018fe0 2023-11-28T17:52:52.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:52.930+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc4013b50 2023-11-28T17:52:52.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:52.930+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:52.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:52.930+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:52.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:52.930+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:52.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:52.930+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:53.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:53.131+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:53.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:53.131+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:53.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:53.131+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:53.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:53.131+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:53.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:53.532+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:53.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:53.532+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:53.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:53.532+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:53.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:53.532+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:54.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:54.333+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:54.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:54.333+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:54.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:54.333+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:54.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:54.333+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:55.929+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:55.929+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:55.929+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:55.929+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:55.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:55.929+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:55.929+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:55.929+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:52:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:55.929+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe006e7b0 con 0x7f9fc400b650 2023-11-28T17:52:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:55.929+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4018fe0 2023-11-28T17:52:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:55.929+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc40088b0 2023-11-28T17:52:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:55.930+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:55.930+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:55.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:55.930+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:55.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:55.930+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:55.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:55.930+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:56.131+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:56.131+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:56.131+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:56.131+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:56.131+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:56.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:56.532+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:56.532+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:56.532+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:56.532+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:56.532+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:52:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:57.333+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:52:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:57.333+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:52:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:57.333+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:52:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:57.333+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:57.333+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:52:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:58.930+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:58.930+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:58.930+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:58.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:58.930+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:52:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:58.930+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:52:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:58.930+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4014770 0x7f9fc40091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:52:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:58.930+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc4013b50 2023-11-28T17:52:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:58.930+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4014770 2023-11-28T17:52:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:58.930+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4013b50 legacy=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:58.930+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4014770 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:58.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:58.930+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4014770 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:52:59.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:59.131+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4014770 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:59.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:59.131+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4013b50 legacy=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:59.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:59.131+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4014770 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:52:59.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:59.532+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4014770 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:52:59.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:59.532+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4013b50 legacy=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:52:59.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:52:59.532+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4014770 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:00.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:00.334+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4014770 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:00.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:00.334+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4013b50 legacy=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:00.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:00.334+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4014770 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:01.931+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4014770 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:01.931+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4014770 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:01.931+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc4013b50 legacy=0x7f9fc400c320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:01.931+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:01.931+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:01.931+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:53:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:01.931+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe006e7b0 con 0x7f9fc400b650 2023-11-28T17:53:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:01.931+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4018fe0 2023-11-28T17:53:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:01.931+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc40088b0 2023-11-28T17:53:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:01.931+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:01.931+0000 7f9fe507d640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:01.931+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:01.931+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:01.931+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:02.132+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:02.132+0000 7f9fe507d640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:02.132+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:02.132+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:02.132+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:02.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:02.533+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:02.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:02.533+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:02.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:02.533+0000 7f9fe507d640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:02.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:02.533+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:02.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:02.533+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:03.335 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:03.334+0000 7f9fe507d640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:03.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:03.334+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:03.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:03.334+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:03.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:03.334+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:03.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:03.334+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:04.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:04.931+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:04.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:04.931+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:04.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:04.931+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:04.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:04.931+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:04.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:04.931+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:04.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:04.931+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:04.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:04.931+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc4014770 2023-11-28T17:53:04.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:04.931+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4013b50 2023-11-28T17:53:04.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:04.931+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4014770 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:04.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:04.931+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:04.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:04.931+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:05.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:05.132+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4014770 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:05.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:05.132+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:05.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:05.132+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:05.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:05.533+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4014770 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:05.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:05.533+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:05.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:05.533+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:06.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:06.334+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4014770 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:06.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:06.334+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:06.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:06.334+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:07.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:07.931+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:07.931+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:07.931+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4014770 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:07.931+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:07.931+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:53:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:07.931+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:07.931+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe006e7b0 con 0x7f9fc400c320 2023-11-28T17:53:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:07.931+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4013b50 2023-11-28T17:53:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:07.931+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc40088b0 2023-11-28T17:53:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:07.931+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:07.931+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:07.931+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc401d4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:07.931+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:07.931+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:08.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:08.133+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:08.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:08.133+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:08.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:08.133+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:08.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:08.133+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:08.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:08.133+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc401d4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:08.534+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc401d4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:08.534+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:08.534+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:08.534+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:08.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:08.534+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:09.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:09.336+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc401d4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:09.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:09.336+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:09.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:09.336+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:09.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:09.336+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:09.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:09.336+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:10.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:10.932+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:10.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:10.932+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:10.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:10.932+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc40091d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:10.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:10.932+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc40091d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:10.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:10.932+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc401d4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:10.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:10.932+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:10.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:10.932+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc400b650 2023-11-28T17:53:10.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:10.932+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4018fe0 2023-11-28T17:53:10.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:10.932+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:10.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:10.932+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:10.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:10.932+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:11.133+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:11.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:11.133+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:11.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:11.133+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:11.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:53:11.394 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:53:11.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:11.534+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:11.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:11.534+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:11.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:11.534+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:12.335+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:12.335+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:12.335+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:13.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:13.932+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:13.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:13.932+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:13.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:13.932+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:13.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:13.933+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:53:13.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:13.933+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc4014230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:13.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:13.933+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe006e7b0 con 0x7f9fc400c320 2023-11-28T17:53:13.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:13.933+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc400b650 2023-11-28T17:53:13.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:13.933+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4013b50 2023-11-28T17:53:13.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:13.933+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:13.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:13.933+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:13.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:13.933+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:13.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:13.933+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:14.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:14.134+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:14.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:14.134+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:14.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:14.134+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:14.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:14.134+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:14.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:14.535+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:14.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:14.535+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:14.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:14.535+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:14.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:14.535+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:15.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:15.336+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:15.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:15.336+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:15.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:15.336+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:15.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:15.336+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:16.933+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc4014230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:16.933+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc4014230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:16.933+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:16.933+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:16.933+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc400c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:16.933+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:16.933+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4018fe0 2023-11-28T17:53:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:16.933+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc40088b0 2023-11-28T17:53:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:16.933+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:16.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:16.933+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:16.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:16.933+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:16.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:16.934+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:17.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:17.134+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:17.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:17.134+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:17.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:17.135+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:17.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:17.135+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:17.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:17.535+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:17.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:17.535+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:17.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:17.536+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:17.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:17.536+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:18.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:18.336+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:18.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:18.336+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:18.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:18.337+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:18.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:18.337+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:19.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:19.933+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:19.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:19.933+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:19.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:19.933+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:19.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:19.933+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:19.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:19.933+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:19.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:19.933+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:53:19.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:19.933+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400d360 con 0x7f9fc400c320 2023-11-28T17:53:19.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:19.933+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe006e7b0 con 0x7f9fc400b650 2023-11-28T17:53:19.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:19.933+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc40088b0 2023-11-28T17:53:19.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:19.934+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:19.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:19.934+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:19.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:19.934+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:19.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:19.934+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:20.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:20.135+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:20.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:20.135+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:20.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:20.135+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:20.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:20.135+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:20.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:20.536+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:20.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:20.536+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:20.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:20.536+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:20.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:20.536+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:21.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:21.337+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:21.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:21.337+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:21.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:21.337+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:21.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:21.337+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:22.934+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 msgr2=0x7f9fc4008c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:22.934+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc40088b0 0x7f9fc4008c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:22.934+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:22.934+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:22.934+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc400c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:22.934+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc4014200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:22.934+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4018fe0 2023-11-28T17:53:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:22.934+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc4013b50 2023-11-28T17:53:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:22.934+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:22.934+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:22.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:22.934+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc4014200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:22.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:22.934+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc4014200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:23.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:23.135+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:23.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:23.135+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:23.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:23.135+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc4014200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:23.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:23.135+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc4014200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:23.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:23.536+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:23.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:23.536+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:23.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:23.536+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc4014200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:23.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:23.536+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc4014200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:24.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:24.337+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:24.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:24.337+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:24.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:24.337+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc4014200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:24.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:24.337+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc4014200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:25.934+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc4014200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:25.934+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc4014200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:25.934+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:25.934+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc4018fe0 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:25.934+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:25.934+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:53:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:25.934+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:25.934+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc400b650 2023-11-28T17:53:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:25.934+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc40091d0 2023-11-28T17:53:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:25.934+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc4013b50 2023-11-28T17:53:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:25.934+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:25.934+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:25.935+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401d4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:25.935+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:25.935+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:26.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:26.136+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401d4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:26.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:26.136+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:26.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:26.136+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:26.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:26.136+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:26.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:26.136+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:26.537+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:26.537+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401d4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:26.537+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:26.537+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:26.537+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:27.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:27.338+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401d4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:27.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:27.338+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:27.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:27.338+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:27.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:27.338+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:27.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:27.338+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:28.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:28.934+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:28.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:28.934+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:28.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:28.934+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401d4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:28.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:28.934+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:28.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:28.934+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:28.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:28.934+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:28.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:28.934+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400d360 con 0x7f9fc40088b0 2023-11-28T17:53:28.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:28.934+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc400c320 2023-11-28T17:53:28.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:28.935+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 msgr2=0x7f9fc4014200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:28.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:28.935+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc40088b0 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:28.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:28.935+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:29.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:29.136+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc40088b0 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:29.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:29.136+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 msgr2=0x7f9fc4014200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:29.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:29.136+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:29.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:29.537+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc40088b0 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:29.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:29.537+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 msgr2=0x7f9fc4014200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:29.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:29.537+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:30.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:30.338+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc40088b0 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:30.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:30.338+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 msgr2=0x7f9fc4014200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:30.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:30.338+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:31.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:31.935+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 msgr2=0x7f9fc4014200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:31.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:31.935+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:31.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:31.935+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc40088b0 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:31.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:31.935+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:53:31.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:31.935+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:31.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:31.935+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:31.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:31.935+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc4008c90 2023-11-28T17:53:31.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:31.935+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc40091d0 2023-11-28T17:53:31.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:31.935+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc4013b50 2023-11-28T17:53:31.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:31.935+0000 7f9fe507d640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:31.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:31.935+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:31.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:31.935+0000 7f9fe507d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:31.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:31.935+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:31.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:31.935+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:32.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:32.136+0000 7f9fe507d640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:32.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:32.136+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:32.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:32.136+0000 7f9fe507d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:32.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:32.136+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:32.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:32.136+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:32.537+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:32.537+0000 7f9fe507d640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:32.537+0000 7f9fe507d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:32.537+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:32.537+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:33.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:33.338+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:33.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:33.338+0000 7f9fe507d640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:33.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:33.338+0000 7f9fe507d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:33.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:33.338+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:33.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:33.338+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:34.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:34.935+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:34.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:34.935+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:34.935+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:34.935+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:34.935+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:34.935+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:34.935+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400d360 con 0x7f9fc400b650 2023-11-28T17:53:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:34.935+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc400c320 2023-11-28T17:53:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:34.935+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:34.935+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:34.935+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:35.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:35.136+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:35.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:35.136+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:35.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:35.136+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:35.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:35.537+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:35.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:35.537+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:35.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:35.537+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:36.338+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:36.338+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:36.338+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:37.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:37.935+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:37.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:37.935+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:37.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:37.935+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:37.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:37.935+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:37.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:37.935+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:53:37.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:37.935+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:37.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:37.935+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc4008c90 2023-11-28T17:53:37.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:37.935+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc40091d0 2023-11-28T17:53:37.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:37.935+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc400c320 2023-11-28T17:53:37.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:37.936+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:37.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:37.936+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:37.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:37.936+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:37.936+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:37.936+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:38.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:38.137+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:38.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:38.137+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:38.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:38.137+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:38.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:38.137+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:38.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:38.137+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:38.538+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:38.538+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:38.538+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:38.538+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:38.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:38.538+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:39.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:39.339+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:39.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:39.339+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:39.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:39.339+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:39.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:39.339+0000 7f9fe507d640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:39.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:39.339+0000 7f9fe507d640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:40.936+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:40.936+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400c320 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:40.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:40.936+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:40.936+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:40.936+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:40.936+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:40.936+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400d360 con 0x7f9fc400b650 2023-11-28T17:53:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:40.936+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4013b50 2023-11-28T17:53:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:40.936+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:40.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:40.936+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:40.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:40.936+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:41.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:41.137+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:41.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:41.137+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:41.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:41.137+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:41.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:53:41.532 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:53:41.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:41.538+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:41.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:41.538+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:41.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:41.538+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:42.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:42.339+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:42.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:42.339+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:42.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:42.339+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:43.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:43.936+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:43.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:43.936+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:43.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:43.936+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400b650 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:43.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:43.936+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:43.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:43.936+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:43.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:43.936+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:53:43.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:43.936+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc4008c90 2023-11-28T17:53:43.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:43.936+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc40091d0 2023-11-28T17:53:43.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:43.936+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc4013b50 2023-11-28T17:53:43.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:43.936+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:43.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:43.936+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:43.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:43.936+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:43.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:43.937+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:43.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:43.937+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:44.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:44.137+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:44.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:44.138+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:44.138+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:44.138+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:44.138+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:44.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:44.538+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:44.539+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:44.539+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:44.539+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:44.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:44.539+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:45.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:45.339+0000 7f9fe507d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:45.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:45.340+0000 7f9fdffff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:45.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:45.340+0000 7f9fdffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:45.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:45.340+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:45.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:45.340+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:46.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:46.937+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:46.937+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:46.937+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:46.937+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:46.937+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:46.937+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc4014160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:46.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:46.937+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400d360 con 0x7f9fc400c320 2023-11-28T17:53:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:46.937+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc400b650 2023-11-28T17:53:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:46.937+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:46.937+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:46.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:46.937+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:47.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:47.138+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:47.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:47.138+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:47.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:47.138+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:47.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:47.539+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:47.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:47.539+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:47.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:47.539+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:48.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:48.340+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:48.340+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:48.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:48.340+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:49.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:49.937+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:49.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:49.937+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:49.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:49.937+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:49.937+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:49.937+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:53:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:49.937+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc4014160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:49.937+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc400c320 2023-11-28T17:53:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:49.937+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc40091d0 2023-11-28T17:53:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:49.937+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc400b650 2023-11-28T17:53:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:49.937+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:49.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:49.937+0000 7f9fe507d640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:49.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:49.937+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:49.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:49.937+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:49.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:49.937+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:50.138+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:50.138+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:50.138+0000 7f9fe507d640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:50.138+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:50.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:50.138+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:50.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:50.539+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:50.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:50.539+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:50.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:50.539+0000 7f9fe507d640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:50.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:50.539+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:50.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:50.539+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:51.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:51.340+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:51.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:51.340+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:51.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:51.340+0000 7f9fe507d640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:51.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:51.340+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:51.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:51.340+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:52.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:52.937+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:52.937+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:52.937+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:52.937+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:52.937+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:52.937+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:52.937+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400d360 con 0x7f9fc4008c90 2023-11-28T17:53:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:52.937+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4013b50 2023-11-28T17:53:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:52.937+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:52.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:52.937+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:52.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:52.937+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:53.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:53.138+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:53.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:53.138+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:53.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:53.138+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:53.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:53.539+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:53.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:53.539+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:53.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:53.539+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:54.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:54.340+0000 7f9fe487c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:54.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:54.340+0000 7f9fdffff640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:54.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:54.340+0000 7f9fdffff640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:55.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:55.938+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:55.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:55.938+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4013b50 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:55.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:55.938+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4008c90 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:55.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:55.938+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:55.938+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:53:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:55.938+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 0x7f9fc4014160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:55.938+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc400c320 2023-11-28T17:53:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:55.938+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc40091d0 2023-11-28T17:53:55.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:55.938+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc4008c90 2023-11-28T17:53:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:55.938+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:55.938+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:55.938+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:55.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:55.938+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:55.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:55.938+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:56.139+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:56.139+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:56.139+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:56.139+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:56.139+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:56.540+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:56.540+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:56.540+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:56.540+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:56.540+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:53:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:57.340+0000 7f9fe487c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:53:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:57.340+0000 7f9fe487c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:57.340+0000 7f9fe507d640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:53:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:57.340+0000 7f9fe507d640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:53:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:57.340+0000 7f9fdffff640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:53:58.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:58.937+0000 7f9fde7fc640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:58.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:58.937+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:58.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:58.937+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:58.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:58.937+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:53:58.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:58.937+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:53:58.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:58.937+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc400c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:53:58.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:58.937+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400d360 con 0x7f9fc4013b50 2023-11-28T17:53:58.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:58.937+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc400b650 2023-11-28T17:53:58.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:58.937+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:58.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:58.937+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4013b50 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:58.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:58.937+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:53:59.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:59.138+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4013b50 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:59.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:59.138+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:59.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:59.138+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:53:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:59.539+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4013b50 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:53:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:59.539+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:53:59.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:53:59.539+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:00.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:00.340+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4013b50 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:00.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:00.340+0000 7f9fe487c640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:00.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:00.340+0000 7f9fe487c640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:01.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:01.937+0000 7f9fde7fc640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:01.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:01.937+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b650 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:01.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:01.937+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc4013b50 legacy=0x7f9fc4009d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:01.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:01.937+0000 7f9fe507d640 1 -- reap_dead start 2023-11-28T17:54:01.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:01.937+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:01.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:01.937+0000 7f9fde7fc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400c160 con 0x7f9fc401b9e0 2023-11-28T17:54:01.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:01.937+0000 7f9fde7fc640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc400d360 con 0x7f9fc400c320 2023-11-28T17:54:01.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:01.937+0000 7f9fde7fc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc40091d0 2023-11-28T17:54:01.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:01.937+0000 7f9fe507d640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:01.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:01.937+0000 7f9fe507d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:01.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:01.937+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:01.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:01.938+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc401b9e0 legacy=0x7f9fc401ddd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:02.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:02.138+0000 7f9fe507d640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:02.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:02.138+0000 7f9fe507d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:02.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:02.139+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:02.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:02.139+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc401b9e0 legacy=0x7f9fc401ddd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:02.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:02.539+0000 7f9fe507d640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:02.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:02.539+0000 7f9fe507d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:02.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:02.540+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc401b9e0 legacy=0x7f9fc401ddd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:02.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:02.540+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:03.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:03.340+0000 7f9fe507d640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:03.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:03.340+0000 7f9fe507d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:03.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:03.341+0000 7f9fdffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc401b9e0 legacy=0x7f9fc401ddd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:03.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:03.341+0000 7f9fe487c640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:04.938+0000 7f9fde7fc640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 msgr2=0x7f9fc401b4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:04.938+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9fc40091d0 0x7f9fc401b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:04.938+0000 7f9fde7fc640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f9fc400c320 legacy=0x7f9fc4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:04.938+0000 7f9fde7fc640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9fc401b9e0 legacy=0x7f9fc401ddd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:04.938+0000 7f9fde7fc640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 0x7f9fc400c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:04.938+0000 7f9fde7fc640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b5c0 0x7f9fc4014160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:04.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:04.938+0000 7f9fde7fc640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe015b150 con 0x7f9fc4008c90 2023-11-28T17:54:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:04.938+0000 7f9fde7fc640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe006e1f0 con 0x7f9fc400b5c0 2023-11-28T17:54:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:04.938+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:04.938+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b5c0 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:04.938+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b5c0 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:04.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:04.938+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:05.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:05.139+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:05.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:05.139+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b5c0 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:05.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:05.139+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b5c0 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:05.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:05.139+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:05.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:05.540+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:05.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:05.540+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b5c0 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:05.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:05.540+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b5c0 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:05.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:05.540+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:06.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:06.341+0000 7f9fe487c640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 msgr2=0x7f9fc400c700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:06.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:06.341+0000 7f9fdffff640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b5c0 msgr2=0x7f9fc4014160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:06.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:06.341+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f9fc400b5c0 0x7f9fc4014160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:06.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:06.341+0000 7f9fe487c640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f9fc4008c90 0x7f9fc400c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:07.822 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:54:07.823 ERROR:teuthology.run_tasks:Manager failed: mon_thrash Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mon_thrash.py", line 420, in task manager.wait_for_mon_quorum_size(len(mons)) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3156, in wait_for_mon_quorum_size if len(self.get_mon_quorum()) == size: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3141, in get_mon_quorum out = self.raw_cluster_cmd('quorum_status') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status' 2023-11-28T17:54:07.831 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-11-28T17:54:07.842 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-11-28T17:54:07.843 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-11-28T17:54:07.843 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:54:07.905 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054299,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2023-11-28T17:54:07.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:54:07.906 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-11-28T17:54:07.957 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T17:54:07.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:54:07.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:54:08.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:54:08.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2023-11-28T17:54:08.165 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-28T17:54:08.175 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-28T17:54:08.176 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:54:08.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.366+0000 7f4c9a983640 1 Processor -- start 2023-11-28T17:54:08.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.366+0000 7f4c9a983640 1 -- start start 2023-11-28T17:54:08.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.367+0000 7f4c9a983640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c94150ac0 0x7f4c940716e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:08.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.367+0000 7f4c9a983640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c94157300 0x7f4c94071c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:08.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.367+0000 7f4c9a983640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94150ea0 con 0x7f4c9406b1e0 2023-11-28T17:54:08.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.367+0000 7f4c9a983640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c9415ad90 con 0x7f4c94157300 2023-11-28T17:54:08.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.367+0000 7f4c9a983640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c94150ac0 2023-11-28T17:54:08.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.367+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c9406b1e0 legacy=0x7f4c94156bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:08.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.367+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c94157300 msgr2=0x7f4c94071c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:08.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.367+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c94150ac0 msgr2=0x7f4c940716e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:08.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.367+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c94157300 0x7f4c94071c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:08.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.367+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c94150ac0 0x7f4c940716e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.568+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c94157300 msgr2=0x7f4c94071c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.568+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c9406b1e0 legacy=0x7f4c94156bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.568+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c94150ac0 msgr2=0x7f4c940716e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.568+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c94157300 0x7f4c94071c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:08.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.568+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c94150ac0 0x7f4c940716e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:08.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.969+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c94157300 msgr2=0x7f4c94071c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:08.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.969+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c94157300 0x7f4c94071c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:08.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.969+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c9406b1e0 legacy=0x7f4c94156bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:08.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.969+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c94150ac0 msgr2=0x7f4c940716e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:08.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:08.969+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c94150ac0 0x7f4c940716e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:09.770+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c94157300 msgr2=0x7f4c94071c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:09.770+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c94157300 0x7f4c94071c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:09.770+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c94150ac0 msgr2=0x7f4c940716e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:09.770+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c9406b1e0 legacy=0x7f4c94156bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:09.770+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c94150ac0 0x7f4c940716e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:11.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.367+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c94150ac0 msgr2=0x7f4c940716e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.367+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c94150ac0 0x7f4c940716e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.367+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c94157300 msgr2=0x7f4c94071c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.367+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c94157300 0x7f4c94071c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.367+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c9406b1e0 legacy=0x7f4c94156bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.367+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78007400 0x7f4c780098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.367+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94159f30 con 0x7f4c780036c0 2023-11-28T17:54:11.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.367+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c78007400 2023-11-28T17:54:11.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.368+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c780036c0 legacy=0x7f4c78005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:11.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.368+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78007400 msgr2=0x7f4c780098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:11.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.368+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78007400 0x7f4c780098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:11.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.569+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c780036c0 legacy=0x7f4c78005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:11.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.569+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78007400 msgr2=0x7f4c780098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:11.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.569+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78007400 0x7f4c780098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:11.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:54:11.709 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:54:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.970+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c780036c0 legacy=0x7f4c78005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.970+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78007400 msgr2=0x7f4c780098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:11.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:11.970+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78007400 0x7f4c780098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:12.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:12.772+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c780036c0 legacy=0x7f4c78005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:12.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:12.772+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78007400 msgr2=0x7f4c780098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:12.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:12.772+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78007400 0x7f4c780098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.368+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78007400 msgr2=0x7f4c780098c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.368+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78007400 0x7f4c780098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.368+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c780036c0 legacy=0x7f4c78005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.368+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:54:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.368+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800aa70 0x7f4c780098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.368+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94150ea0 con 0x7f4c7800a690 2023-11-28T17:54:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.368+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94159f30 con 0x7f4c9406b1e0 2023-11-28T17:54:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.368+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800aa70 2023-11-28T17:54:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.368+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800aa70 msgr2=0x7f4c780098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.368+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c9406b1e0 legacy=0x7f4c78009f80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.368+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800a690 legacy=0x7f4c78003b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:14.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.368+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800aa70 0x7f4c780098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.569+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800a690 legacy=0x7f4c78003b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.569+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800aa70 msgr2=0x7f4c780098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:14.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.569+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c9406b1e0 legacy=0x7f4c78009f80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:14.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.569+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800aa70 0x7f4c780098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:14.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.970+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800a690 legacy=0x7f4c78003b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:14.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.970+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800aa70 msgr2=0x7f4c780098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:14.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.970+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800aa70 0x7f4c780098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:14.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:14.970+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c9406b1e0 legacy=0x7f4c78009f80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:15.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:15.771+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800aa70 msgr2=0x7f4c780098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:15.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:15.771+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800a690 legacy=0x7f4c78003b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:15.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:15.771+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800aa70 0x7f4c780098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:15.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:15.771+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c9406b1e0 legacy=0x7f4c78009f80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:17.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.369+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800aa70 msgr2=0x7f4c780098c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.369+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800aa70 0x7f4c780098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.369+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c9406b1e0 legacy=0x7f4c78009f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.369+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800a690 legacy=0x7f4c78003b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.369+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800e2c0 0x7f4c7800e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.369+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800ed70 0x7f4c7800f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.369+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800ed70 2023-11-28T17:54:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.369+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c7800e2c0 2023-11-28T17:54:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.369+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800ed70 msgr2=0x7f4c7800f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.369+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800e2c0 msgr2=0x7f4c7800e700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.369+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800ed70 0x7f4c7800f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.369+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800e2c0 0x7f4c7800e700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.570+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800ed70 msgr2=0x7f4c7800f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.570+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800ed70 0x7f4c7800f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.570+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800e2c0 msgr2=0x7f4c7800e700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.570+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800e2c0 0x7f4c7800e700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:17.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.971+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800ed70 msgr2=0x7f4c7800f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:17.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.971+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800ed70 0x7f4c7800f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:17.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.971+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800e2c0 msgr2=0x7f4c7800e700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:17.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:17.971+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800e2c0 0x7f4c7800e700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:18.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:18.772+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800ed70 msgr2=0x7f4c7800f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:18.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:18.772+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800ed70 0x7f4c7800f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:18.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:18.772+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800e2c0 msgr2=0x7f4c7800e700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:18.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:18.772+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800e2c0 0x7f4c7800e700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:20.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.369+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800e2c0 msgr2=0x7f4c7800e700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:20.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.369+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800e2c0 0x7f4c7800e700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:20.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.369+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800ed70 msgr2=0x7f4c7800f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:20.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.369+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800ed70 0x7f4c7800f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:20.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.369+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:54:20.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.369+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:20.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.369+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:20.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.369+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c7800f150 2023-11-28T17:54:20.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.369+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c780098a0 2023-11-28T17:54:20.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.369+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c78009f40 2023-11-28T17:54:20.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.369+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c7800f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:20.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.369+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:20.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.369+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:20.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.369+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 msgr2=0x7f4c7800a360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:20.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.369+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.570+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c7800f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.570+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.570+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.570+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 msgr2=0x7f4c7800a360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:20.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.570+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:20.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.971+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c7800f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:20.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.971+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:20.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.971+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:20.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.971+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 msgr2=0x7f4c7800a360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:20.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:20.971+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:21.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:21.772+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c7800f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:21.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:21.772+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 msgr2=0x7f4c7800a360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:21.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:21.772+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:21.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:21.772+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:21.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:21.772+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:23.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.370+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 msgr2=0x7f4c7800a360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:23.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.370+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:23.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.370+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:23.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.370+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:23.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.370+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c7800f9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:23.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.370+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780124f0 0x7f4c780148e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:23.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.370+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94150ea0 con 0x7f4c7800a8a0 2023-11-28T17:54:23.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.370+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c780124f0 2023-11-28T17:54:23.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.370+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800a8a0 legacy=0x7f4c78005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:23.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.370+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780124f0 msgr2=0x7f4c780148e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:23.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.370+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780124f0 0x7f4c780148e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:23.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.571+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780124f0 msgr2=0x7f4c780148e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:23.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.571+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800a8a0 legacy=0x7f4c78005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:23.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.571+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780124f0 0x7f4c780148e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.972+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780124f0 msgr2=0x7f4c780148e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.972+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800a8a0 legacy=0x7f4c78005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:23.972+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780124f0 0x7f4c780148e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:24.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:24.773+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780124f0 msgr2=0x7f4c780148e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:24.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:24.773+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800a8a0 legacy=0x7f4c78005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:24.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:24.773+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780124f0 0x7f4c780148e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:26.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.370+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780124f0 msgr2=0x7f4c780148e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:26.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.371+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780124f0 0x7f4c780148e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:26.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.371+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800a8a0 legacy=0x7f4c78005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:26.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.371+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:54:26.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.371+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:26.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.371+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:26.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.371+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c7800a8a0 2023-11-28T17:54:26.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.371+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c780098a0 2023-11-28T17:54:26.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.371+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c78009f40 2023-11-28T17:54:26.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.371+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:26.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.371+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 msgr2=0x7f4c7800a360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:26.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.371+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:26.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.371+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800a8a0 legacy=0x7f4c78005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:26.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.371+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:26.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.572+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:26.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.572+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:26.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.572+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 msgr2=0x7f4c7800a360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:26.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.572+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800a8a0 legacy=0x7f4c78005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:26.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.572+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:26.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.973+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:26.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.973+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:26.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.973+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 msgr2=0x7f4c7800a360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:26.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.973+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800a8a0 legacy=0x7f4c78005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:26.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:26.973+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:27.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:27.774+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:27.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:27.774+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:27.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:27.774+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 msgr2=0x7f4c7800a360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:27.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:27.774+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800a8a0 legacy=0x7f4c78005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:27.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:27.775+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:29.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.371+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 msgr2=0x7f4c7800a360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:29.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.371+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:29.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.371+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:29.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.371+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:29.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.371+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800a8a0 legacy=0x7f4c78005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:29.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.371+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:29.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.371+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94150ea0 con 0x7f4c7800b0e0 2023-11-28T17:54:29.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.371+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800f150 2023-11-28T17:54:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.372+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b0e0 legacy=0x7f4c78014500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.372+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 msgr2=0x7f4c7800f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:29.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.372+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:29.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.573+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b0e0 legacy=0x7f4c78014500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:29.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.573+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 msgr2=0x7f4c7800f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:29.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.573+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:29.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.974+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b0e0 legacy=0x7f4c78014500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:29.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.974+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 msgr2=0x7f4c7800f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:29.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:29.974+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:30.775+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b0e0 legacy=0x7f4c78014500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:30.775+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 msgr2=0x7f4c7800f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:30.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:30.775+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:32.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.372+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 msgr2=0x7f4c7800f9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:32.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.372+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.372+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b0e0 legacy=0x7f4c78014500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.372+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:54:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.372+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.372+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.372+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c7800b4f0 2023-11-28T17:54:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.372+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c780098a0 2023-11-28T17:54:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.372+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c78009f40 2023-11-28T17:54:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.372+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b4f0 legacy=0x7f4c7800b8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.372+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.372+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 msgr2=0x7f4c7800a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.372+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:32.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.372+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:32.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.573+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b4f0 legacy=0x7f4c7800b8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:32.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.573+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:32.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.573+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:32.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.573+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 msgr2=0x7f4c7800a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:32.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.573+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.974+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b4f0 legacy=0x7f4c7800b8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.974+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.974+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 msgr2=0x7f4c7800a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.974+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:32.974+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:33.775+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b4f0 legacy=0x7f4c7800b8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:33.775+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:33.775+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:33.775+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 msgr2=0x7f4c7800a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:33.775+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.372+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 msgr2=0x7f4c7800a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.372+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78009f40 0x7f4c7800a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.372+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.372+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780098a0 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.372+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b4f0 legacy=0x7f4c7800b8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.372+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800a880 0x7f4c7800f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.372+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94150ea0 con 0x7f4c7800f150 2023-11-28T17:54:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.372+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800a880 2023-11-28T17:54:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.373+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c78005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.373+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800a880 msgr2=0x7f4c7800f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.373+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800a880 0x7f4c7800f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:35.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.574+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c78005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:35.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.574+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800a880 msgr2=0x7f4c7800f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:35.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.574+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800a880 0x7f4c7800f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:35.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.975+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c78005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:35.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.975+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800a880 msgr2=0x7f4c7800f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:35.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:35.975+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800a880 0x7f4c7800f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:36.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:36.776+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c78005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:36.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:36.776+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800a880 msgr2=0x7f4c7800f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:36.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:36.776+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800a880 0x7f4c7800f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.373+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800a880 msgr2=0x7f4c7800f9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.373+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800a880 0x7f4c7800f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.373+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c78005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.373+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:54:38.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.373+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:38.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.373+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c7800b4f0 2023-11-28T17:54:38.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.373+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94150ea0 con 0x7f4c7800f9b0 2023-11-28T17:54:38.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.373+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800f150 2023-11-28T17:54:38.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.373+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b4f0 legacy=0x7f4c7800b8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:38.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.373+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:38.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.373+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:38.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.373+0000 7f4c9a182640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78014500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:38.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.574+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b4f0 legacy=0x7f4c7800b8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:38.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.574+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:38.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.574+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:38.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.574+0000 7f4c9a182640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78014500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:38.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.975+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b4f0 legacy=0x7f4c7800b8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:38.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.975+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:38.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.975+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:38.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:38.975+0000 7f4c9a182640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78014500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:39.776+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b4f0 legacy=0x7f4c7800b8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:39.776+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:39.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:39.776+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:39.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:39.776+0000 7f4c9a182640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78014500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:41.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.373+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:41.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.373+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:41.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.373+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78014500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:41.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.373+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b4f0 legacy=0x7f4c7800b8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:41.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.373+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78009f40 0x7f4c7800a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:41.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.373+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800a860 0x7f4c78008880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:41.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.373+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800a860 2023-11-28T17:54:41.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.373+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c78009f40 2023-11-28T17:54:41.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.374+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78009f40 msgr2=0x7f4c7800a320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:41.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.374+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800a860 msgr2=0x7f4c78008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:41.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.374+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78009f40 0x7f4c7800a320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:41.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.374+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800a860 0x7f4c78008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.575+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800a860 msgr2=0x7f4c78008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.575+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78009f40 msgr2=0x7f4c7800a320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.575+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800a860 0x7f4c78008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.575+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78009f40 0x7f4c7800a320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:41.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:54:41.887 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:54:41.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.976+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78009f40 msgr2=0x7f4c7800a320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:41.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.976+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800a860 msgr2=0x7f4c78008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:41.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.976+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78009f40 0x7f4c7800a320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:41.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:41.976+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800a860 0x7f4c78008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:42.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:42.777+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800a860 msgr2=0x7f4c78008880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:42.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:42.777+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78009f40 msgr2=0x7f4c7800a320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:42.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:42.777+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800a860 0x7f4c78008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:42.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:42.777+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78009f40 0x7f4c7800a320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:44.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.374+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78009f40 msgr2=0x7f4c7800a320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.374+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78009f40 0x7f4c7800a320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.374+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800a860 msgr2=0x7f4c78008880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.374+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800a860 0x7f4c78008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.374+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:54:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.374+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78008dc0 0x7f4c780091a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.374+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94159f30 con 0x7f4c7800b4f0 2023-11-28T17:54:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.374+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c7800f9b0 2023-11-28T17:54:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.374+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c78008dc0 2023-11-28T17:54:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.374+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78008dc0 msgr2=0x7f4c780091a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.374+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78014500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.374+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78008dc0 0x7f4c780091a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.374+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b4f0 legacy=0x7f4c7800b8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:44.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.575+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78008dc0 msgr2=0x7f4c780091a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:44.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.575+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78014500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:44.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.575+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b4f0 legacy=0x7f4c7800b8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:44.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.575+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78008dc0 0x7f4c780091a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:44.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.976+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78008dc0 msgr2=0x7f4c780091a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:44.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.976+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b4f0 legacy=0x7f4c7800b8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:44.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.976+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78014500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:44.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:44.976+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78008dc0 0x7f4c780091a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:45.777+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78008dc0 msgr2=0x7f4c780091a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:45.777+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b4f0 legacy=0x7f4c7800b8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:45.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:45.777+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78014500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:45.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:45.777+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78008dc0 0x7f4c780091a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:47.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.374+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78008dc0 msgr2=0x7f4c780091a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:47.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.374+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78008dc0 0x7f4c780091a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:47.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.374+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78014500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:47.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.374+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800b4f0 legacy=0x7f4c7800b8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:47.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.374+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:47.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.374+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c78008860 0x7f4c780096e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:47.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.374+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c78008860 2023-11-28T17:54:47.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.374+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c7800f150 2023-11-28T17:54:47.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.375+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c78008860 msgr2=0x7f4c780096e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:47.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.375+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:47.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.375+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c78008860 0x7f4c780096e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:47.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.375+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:47.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.576+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c78008860 msgr2=0x7f4c780096e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:47.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.576+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c78008860 0x7f4c780096e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:47.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.576+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:47.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.576+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.977+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c78008860 msgr2=0x7f4c780096e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.977+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c78008860 0x7f4c780096e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.977+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:47.977+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:48.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:48.778+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c78008860 msgr2=0x7f4c780096e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:48.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:48.778+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c78008860 0x7f4c780096e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:48.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:48.778+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:48.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:48.778+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:50.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.375+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:50.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.375+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:50.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.375+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c78008860 msgr2=0x7f4c780096e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:50.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.375+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c78008860 0x7f4c780096e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:50.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.375+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:50.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.375+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:54:50.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.375+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f9b0 0x7f4c78009f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:50.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.375+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94150ea0 con 0x7f4c7800a480 2023-11-28T17:54:50.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.375+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800f9b0 2023-11-28T17:54:50.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.375+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c7800f150 2023-11-28T17:54:50.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.375+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:50.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.375+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800a480 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:50.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.375+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f9b0 msgr2=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:50.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.375+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:50.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.375+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f9b0 0x7f4c78009f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.576+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.576+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800a480 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.576+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f9b0 msgr2=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.576+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:50.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.576+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f9b0 0x7f4c78009f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.977+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.977+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800a480 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.977+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f9b0 msgr2=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.977+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:50.977+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f9b0 0x7f4c78009f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:51.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:51.778+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800a480 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:51.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:51.778+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:51.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:51.778+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f9b0 msgr2=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:51.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:51.778+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:51.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:51.778+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f9b0 0x7f4c78009f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:53.376 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.375+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:53.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.375+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:53.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.375+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f9b0 msgr2=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:53.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.375+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f9b0 0x7f4c78009f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:53.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.375+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800a480 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:53.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.375+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78013500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:53.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.375+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94159f30 con 0x7f4c78009830 2023-11-28T17:54:53.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.375+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800dc10 2023-11-28T17:54:53.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.376+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78013500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:53.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.376+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78009830 legacy=0x7f4c78006b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:53.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.376+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78013500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:53.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.577+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78013500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:53.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.577+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78013500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:53.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.577+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78009830 legacy=0x7f4c78006b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:53.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.978+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78013500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:53.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.978+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78013500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:53.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:53.978+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78009830 legacy=0x7f4c78006b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:54.779+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78013500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:54.779+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78013500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:54.779+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78009830 legacy=0x7f4c78006b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:56.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.376+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78013500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:56.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.376+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78013500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:56.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.376+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78009830 legacy=0x7f4c78006b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:56.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.376+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:54:56.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.376+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:56.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.376+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94150ea0 con 0x7f4c7800f9b0 2023-11-28T17:54:56.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.376+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94159f30 con 0x7f4c78013a40 2023-11-28T17:54:56.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.376+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800f150 2023-11-28T17:54:56.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.376+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78013a40 legacy=0x7f4c78013e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:56.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.376+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:56.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.376+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:56.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.376+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:56.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.577+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78013a40 legacy=0x7f4c78013e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:56.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.577+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:56.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.577+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:56.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.577+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.978+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78013a40 legacy=0x7f4c78013e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.978+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.978+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:56.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:56.978+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:57.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:57.779+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78013a40 legacy=0x7f4c78013e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:54:57.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:57.779+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:54:57.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:57.779+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:54:57.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:57.780+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:54:59.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.377+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:59.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.377+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:54:59.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.377+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78013a40 legacy=0x7f4c78013e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.377+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:54:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.377+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 0x7f4c780124b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.377+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78012e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:54:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.377+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c780129f0 2023-11-28T17:54:59.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.377+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c780145d0 2023-11-28T17:54:59.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.377+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 msgr2=0x7f4c78012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:59.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.377+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:59.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.377+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 msgr2=0x7f4c780124b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:59.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.377+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 0x7f4c780124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:54:59.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.578+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 msgr2=0x7f4c78012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:59.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.578+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:59.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.578+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 msgr2=0x7f4c780124b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:59.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.578+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 0x7f4c780124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:54:59.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.979+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 msgr2=0x7f4c78012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:54:59.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.979+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:54:59.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.979+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 msgr2=0x7f4c780124b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:54:59.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:54:59.979+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 0x7f4c780124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:00.780+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 msgr2=0x7f4c78012e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:00.780+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:00.780+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 msgr2=0x7f4c780124b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:00.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:00.780+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 0x7f4c780124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.377+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 msgr2=0x7f4c780124b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.377+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 0x7f4c780124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.377+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 msgr2=0x7f4c78012e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.377+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78012e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.377+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.377+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:55:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.377+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 0x7f4c780124b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.377+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c7800f9b0 2023-11-28T17:55:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.377+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800f150 2023-11-28T17:55:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.377+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c780145d0 2023-11-28T17:55:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.377+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.377+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.377+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 msgr2=0x7f4c780124b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:02.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.377+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:02.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.377+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 0x7f4c780124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:02.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.578+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:02.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.578+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:02.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.578+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:02.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.578+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 msgr2=0x7f4c780124b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:02.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.578+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 0x7f4c780124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.979+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.979+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.979+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.979+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 msgr2=0x7f4c780124b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:02.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:02.979+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 0x7f4c780124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:03.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:03.780+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:03.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:03.780+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:03.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:03.780+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:03.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:03.780+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 msgr2=0x7f4c780124b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:03.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:03.780+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 0x7f4c780124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:05.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.378+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 msgr2=0x7f4c780124b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.378+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c780145d0 0x7f4c780124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.378+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.378+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.378+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.378+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78006b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.378+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94150ea0 con 0x7f4c78012dd0 2023-11-28T17:55:05.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.378+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c780129f0 2023-11-28T17:55:05.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.378+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012dd0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:05.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.378+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 msgr2=0x7f4c78006b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:05.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.378+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78006b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.579+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 msgr2=0x7f4c78006b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.579+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012dd0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.579+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78006b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:05.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.980+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 msgr2=0x7f4c78006b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:05.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.980+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78006b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:05.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:05.980+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012dd0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:06.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:06.781+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 msgr2=0x7f4c78006b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:06.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:06.781+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78006b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:06.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:06.781+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012dd0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:08.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.378+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 msgr2=0x7f4c78006b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:08.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.378+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78006b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:08.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.378+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012dd0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:08.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.378+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78006b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.378+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:55:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.378+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.378+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c7800f9b0 2023-11-28T17:55:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.378+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c780129f0 2023-11-28T17:55:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.378+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c7800f150 2023-11-28T17:55:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.378+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 msgr2=0x7f4c78006b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.378+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.378+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78006b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.378+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.378+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.579+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 msgr2=0x7f4c78006b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.579+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78006b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.579+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.579+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.579+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:08.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.980+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 msgr2=0x7f4c78006b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:08.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.980+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78006b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:08.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.980+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:08.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.980+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:08.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:08.980+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:09.781+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 msgr2=0x7f4c78006b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:09.781+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78006b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:09.781+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:09.781+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:09.781+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:11.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.378+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:11.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.378+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:11.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.378+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 msgr2=0x7f4c78006b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:11.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.378+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780129f0 0x7f4c78006b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:11.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.378+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:11.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.378+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:11.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.378+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94150ea0 con 0x7f4c78012dd0 2023-11-28T17:55:11.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.378+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c78012480 2023-11-28T17:55:11.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.378+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 msgr2=0x7f4c780145d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:11.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.378+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012dd0 legacy=0x7f4c780131b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:11.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.378+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:11.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.579+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 msgr2=0x7f4c780145d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:11.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.579+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012dd0 legacy=0x7f4c780131b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:11.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.579+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.980+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 msgr2=0x7f4c780145d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.980+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012dd0 legacy=0x7f4c780131b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:11.980+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:12.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:55:12.034 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:55:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:12.781+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 msgr2=0x7f4c780145d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:12.781+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012dd0 legacy=0x7f4c780131b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:12.781+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.378+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 msgr2=0x7f4c780145d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.378+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.378+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012dd0 legacy=0x7f4c780131b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:14.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.378+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:55:14.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.378+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:14.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.378+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:14.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.378+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c7800f9b0 2023-11-28T17:55:14.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.378+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800f150 2023-11-28T17:55:14.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.378+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c78012480 2023-11-28T17:55:14.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.378+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 msgr2=0x7f4c780145d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:14.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.378+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:14.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.378+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:14.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.378+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:14.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.378+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.579+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.579+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 msgr2=0x7f4c780145d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.579+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.579+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.579+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.980+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.980+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 msgr2=0x7f4c780145d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.980+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.980+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:14.980+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:15.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:15.781+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:15.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:15.781+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:15.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:15.781+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 msgr2=0x7f4c780145d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:15.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:15.781+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:15.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:15.781+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.379+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 msgr2=0x7f4c780145d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.379+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.379+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.379+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.379+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.379+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780138a0 0x7f4c78013cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.379+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94150ea0 con 0x7f4c780141f0 2023-11-28T17:55:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.379+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c780138a0 2023-11-28T17:55:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.379+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c780141f0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.379+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780138a0 msgr2=0x7f4c78013cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.379+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780138a0 0x7f4c78013cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.580+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780138a0 msgr2=0x7f4c78013cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.580+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c780141f0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.580+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780138a0 0x7f4c78013cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:17.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.981+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780138a0 msgr2=0x7f4c78013cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:17.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.981+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780138a0 0x7f4c78013cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:17.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:17.981+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c780141f0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:18.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:18.782+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780138a0 msgr2=0x7f4c78013cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:18.782+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780138a0 0x7f4c78013cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:18.782+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c780141f0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:20.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.379+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c780138a0 msgr2=0x7f4c78013cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:20.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.379+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c780138a0 0x7f4c78013cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:20.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.379+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c780141f0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:20.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.379+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:20.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.379+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:55:20.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.379+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:20.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.379+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c780141f0 2023-11-28T17:55:20.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.379+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800f150 2023-11-28T17:55:20.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.379+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c78012480 2023-11-28T17:55:20.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.379+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:20.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.379+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c780141f0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:20.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.379+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 msgr2=0x7f4c780145d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:20.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.379+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:20.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.379+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.580+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.580+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.580+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 msgr2=0x7f4c780145d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.580+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c780141f0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:20.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.580+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:20.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.981+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:20.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.981+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:20.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.981+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 msgr2=0x7f4c780145d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:20.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.981+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c780141f0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:20.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:20.981+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:21.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:21.783+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:21.783+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:21.783+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 msgr2=0x7f4c780145d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:21.783+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c780141f0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:21.783+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:23.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.380+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 msgr2=0x7f4c780145d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:23.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.380+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78012480 0x7f4c780145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:23.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.380+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:23.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.380+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:23.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.380+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c780141f0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:23.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.380+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78006ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:23.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.380+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94150ea0 con 0x7f4c7800f9b0 2023-11-28T17:55:23.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.380+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800dc10 2023-11-28T17:55:23.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.380+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78006ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:23.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.380+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78006ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:23.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.380+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:23.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.581+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78006ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:23.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.581+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78006ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:23.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.581+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:23.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.982+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:23.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.982+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78006ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:23.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:23.982+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78006ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:24.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:24.783+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78006ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:24.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:24.783+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:24.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:24.783+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78006ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:26.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.379+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78006ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:26.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.379+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78006ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:26.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.380+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:26.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.380+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:55:26.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.380+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78006ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:26.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.380+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:26.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.380+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c7800f9b0 2023-11-28T17:55:26.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.380+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800dc10 2023-11-28T17:55:26.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.380+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c7800f150 2023-11-28T17:55:26.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.380+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78006ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:26.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.380+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:26.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.380+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78006ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:26.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.380+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:26.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.380+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:26.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.581+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78006ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:26.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.581+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:26.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.581+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78006ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:26.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.581+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:26.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.581+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:26.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.982+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78006ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:26.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.982+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:26.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.982+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78006ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:26.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.982+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:26.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:26.982+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:27.783+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78006ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:27.783+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78006ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:27.783+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:27.783+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:27.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:27.783+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:29.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.381+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.381+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.381+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78006ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.381+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78006ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.381+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.381+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78003aa0 0x7f4c78005e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.381+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94150ea0 con 0x7f4c78012480 2023-11-28T17:55:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.381+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c78003aa0 2023-11-28T17:55:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.381+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012480 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:29.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.381+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78003aa0 msgr2=0x7f4c78005e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:29.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.381+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78003aa0 0x7f4c78005e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:29.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.582+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78003aa0 msgr2=0x7f4c78005e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:29.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.582+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012480 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:29.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.582+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78003aa0 0x7f4c78005e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:29.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.983+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78003aa0 msgr2=0x7f4c78005e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:29.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.983+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012480 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:29.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:29.983+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78003aa0 0x7f4c78005e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:30.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:30.784+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78003aa0 msgr2=0x7f4c78005e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:30.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:30.784+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012480 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:30.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:30.784+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78003aa0 0x7f4c78005e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:32.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.380+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78003aa0 msgr2=0x7f4c78005e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:32.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.380+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78003aa0 0x7f4c78005e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:32.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.381+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012480 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:32.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.381+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:55:32.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.381+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:32.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.381+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c780063d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:32.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.381+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c7800f9b0 2023-11-28T17:55:32.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.381+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800dc10 2023-11-28T17:55:32.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.381+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c7800f150 2023-11-28T17:55:32.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.381+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c780063d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:32.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.381+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:32.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.381+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:32.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.381+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c780063d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:32.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.381+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.582+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.582+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c780063d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.582+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c780063d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.582+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.582+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.983+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.983+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c780063d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.983+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c780063d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.983+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:32.983+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:33.784+0000 7f4c9a182640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:33.784+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c780063d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:33.784+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c780063d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:33.784+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:33.784+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.382+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.382+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.382+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c780063d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.382+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c780063d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.382+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.382+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 0x7f4c78006cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.382+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94150ea0 con 0x7f4c78012480 2023-11-28T17:55:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.382+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c78006910 2023-11-28T17:55:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.382+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012480 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.382+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 msgr2=0x7f4c78006cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:35.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.382+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 0x7f4c78006cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:35.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.583+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 msgr2=0x7f4c78006cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.583+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012480 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:35.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.583+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 0x7f4c78006cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:35.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.984+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 msgr2=0x7f4c78006cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:35.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.984+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012480 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:35.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:35.984+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 0x7f4c78006cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:36.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:36.785+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 msgr2=0x7f4c78006cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:36.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:36.785+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012480 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:36.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:36.785+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 0x7f4c78006cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:38.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 msgr2=0x7f4c78006cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:38.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 0x7f4c78006cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:38.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c78012480 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:38.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:55:38.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78005ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:38.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:38.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 0x7f4c78006cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800dc10 2023-11-28T17:55:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c7800f150 2023-11-28T17:55:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c78012480 con 0x7f4c78006910 2023-11-28T17:55:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:38.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:38.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 msgr2=0x7f4c78006cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:38.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.382+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 0x7f4c78006cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:38.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.583+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:38.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.583+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:38.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.583+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:38.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.583+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 msgr2=0x7f4c78006cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:38.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.583+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:38.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.583+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 0x7f4c78006cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:38.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.984+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:38.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.984+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:38.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.984+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:38.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.984+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:38.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.984+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 msgr2=0x7f4c78006cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:38.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:38.984+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 0x7f4c78006cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:39.786+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:39.786+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:39.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:39.786+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:39.786+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 msgr2=0x7f4c78006cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:39.786+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:39.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:39.786+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 0x7f4c78006cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:41.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:41.383+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 msgr2=0x7f4c78006cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:41.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:41.383+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006910 0x7f4c78006cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:41.383+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:41.383+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:41.383+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 msgr2=0x7f4c78005ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:41.383+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800dc10 0x7f4c78005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:41.383+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c78005ff0 2023-11-28T17:55:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:41.383+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94150ea0 con 0x7f4c7800f9b0 2023-11-28T17:55:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:41.384+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c78005ff0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:41.384+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:41.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:41.585+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c78005ff0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:41.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:41.585+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:41.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:41.986+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c78005ff0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:41.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:41.986+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:42.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:55:42.183 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:55:42.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:42.787+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c78005ff0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:42.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:42.787+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:44.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.383+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f9b0 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:44.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.383+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c78005ff0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.384+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:55:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.384+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c780063d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.384+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800e6a0 0x7f4c78006910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.384+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94159f30 con 0x7f4c7800f990 2023-11-28T17:55:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.384+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800e6a0 2023-11-28T17:55:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.384+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c7800f150 2023-11-28T17:55:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.384+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c780063d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:44.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.384+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800e6a0 msgr2=0x7f4c78006910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.384+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f990 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.384+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c780063d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.384+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800e6a0 0x7f4c78006910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:44.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.585+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800e6a0 msgr2=0x7f4c78006910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:44.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.585+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800e6a0 0x7f4c78006910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:44.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.585+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c780063d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:44.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.585+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f990 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:44.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.585+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c780063d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.986+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800e6a0 msgr2=0x7f4c78006910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.986+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800e6a0 0x7f4c78006910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.986+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c780063d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.986+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f990 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:44.986+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c780063d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:45.787+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800e6a0 msgr2=0x7f4c78006910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:45.787+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800e6a0 0x7f4c78006910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:45.787+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f990 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:45.787+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c780063d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:45.788+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c780063d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.384+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c780063d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.384+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c7800f150 0x7f4c780063d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.384+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800e6a0 msgr2=0x7f4c78006910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.384+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800e6a0 0x7f4c78006910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.384+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f990 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.384+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006e50 0x7f4c78005ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.384+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c78005ff0 2023-11-28T17:55:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.384+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c78006e50 2023-11-28T17:55:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.385+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c78005ff0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.385+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006e50 msgr2=0x7f4c78005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.385+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006e50 0x7f4c78005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:47.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.586+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c78005ff0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.586+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006e50 msgr2=0x7f4c78005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:47.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.586+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006e50 0x7f4c78005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:47.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.987+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c78005ff0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:47.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.987+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006e50 msgr2=0x7f4c78005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:47.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:47.987+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006e50 0x7f4c78005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:48.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:48.788+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c78005ff0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:48.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:48.788+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006e50 msgr2=0x7f4c78005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:48.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:48.788+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006e50 0x7f4c78005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.384+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006e50 msgr2=0x7f4c78005ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.384+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78006e50 0x7f4c78005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:50.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.384+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c78005ff0 legacy=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.385+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:55:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.385+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.385+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c7800d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.385+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94159f30 con 0x7f4c7800f990 2023-11-28T17:55:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.385+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800f150 2023-11-28T17:55:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.385+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c78011450 2023-11-28T17:55:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.385+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.385+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f990 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.385+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.385+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 msgr2=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.385+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c7800d500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:50.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.586+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:50.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.586+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:50.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.586+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 msgr2=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:50.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.586+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f990 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:50.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.586+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c7800d500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.987+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.987+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 msgr2=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.987+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f990 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.987+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c7800d500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:50.987+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:51.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:51.788+0000 7f4c9a182640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 msgr2=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:51.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:51.788+0000 7f4c9a182640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c7800d500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:51.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:51.788+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f990 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:51.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:51.788+0000 7f4c99180640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:51.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:51.788+0000 7f4c99180640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:53.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.385+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 msgr2=0x7f4c7800d500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.385+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c7800d500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.385+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.385+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f150 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.385+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f990 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:53.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.385+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.385+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c7800da40 2023-11-28T17:55:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.385+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c78011f40 2023-11-28T17:55:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.385+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800da40 legacy=0x7f4c78011830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.385+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 msgr2=0x7f4c78016d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.386+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:53.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.586+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800da40 legacy=0x7f4c78011830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:53.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.587+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 msgr2=0x7f4c78016d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:53.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.587+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:53.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.987+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800da40 legacy=0x7f4c78011830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.988+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 msgr2=0x7f4c78016d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:53.988+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:54.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:54.788+0000 7f4c99981640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800da40 legacy=0x7f4c78011830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:54.789+0000 7f4c99180640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 msgr2=0x7f4c78016d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:54.789+0000 7f4c99180640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:56.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.385+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 msgr2=0x7f4c78016d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:56.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.385+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:56.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.385+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800da40 legacy=0x7f4c78011830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:56.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.385+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:56.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.385+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:55:56.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.385+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:56.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.386+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94159f30 con 0x7f4c7800f150 2023-11-28T17:55:56.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.386+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800f960 2023-11-28T17:55:56.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.386+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c78011f40 2023-11-28T17:55:56.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.386+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 msgr2=0x7f4c78016d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:56.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.386+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:56.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.386+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:56.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.386+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:56.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.386+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:56.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.587+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:56.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.587+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:56.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.587+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:56.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.587+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 msgr2=0x7f4c78016d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:56.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.587+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:56.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.988+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:56.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.988+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:56.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.988+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 msgr2=0x7f4c78016d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:56.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.988+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:56.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:56.988+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:55:57.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:57.789+0000 7f4c99981640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 msgr2=0x7f4c78016d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:55:57.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:57.789+0000 7f4c9a182640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:55:57.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:57.789+0000 7f4c99981640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:57.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:57.789+0000 7f4c9a182640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:55:57.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:57.789+0000 7f4c99180640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:55:59.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.386+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 msgr2=0x7f4c78016d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:59.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.386+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:59.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.386+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:59.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.386+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:55:59.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.386+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:55:59.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.386+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c78011830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:55:59.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.386+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c7800d500 2023-11-28T17:55:59.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.386+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c78011450 2023-11-28T17:55:59.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.386+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800d500 legacy=0x7f4c7800d910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:59.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.386+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 msgr2=0x7f4c78011830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:59.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.387+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c78011830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:55:59.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.587+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800d500 legacy=0x7f4c7800d910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:59.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.588+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 msgr2=0x7f4c78011830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:59.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.588+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c78011830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:55:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.988+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800d500 legacy=0x7f4c7800d910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:55:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.989+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 msgr2=0x7f4c78011830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:55:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:55:59.989+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c78011830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:56:00.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:00.789+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800d500 legacy=0x7f4c7800d910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:56:00.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:00.790+0000 7f4c99981640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 msgr2=0x7f4c78011830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:56:00.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:00.790+0000 7f4c99981640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c78011830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:56:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.386+0000 7f4c8b7fe640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 msgr2=0x7f4c78011830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:56:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.386+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c78011830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:56:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.386+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800d500 legacy=0x7f4c7800d910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:56:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.386+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 0x7f4c7800e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:56:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.386+0000 7f4c9a182640 1 -- reap_dead start 2023-11-28T17:56:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.386+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:56:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.386+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94159f30 con 0x7f4c7800d500 2023-11-28T17:56:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.386+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c7800f960 2023-11-28T17:56:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.386+0000 7f4c8b7fe640 1 -- --> v2:172.21.15.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7800f820 con 0x7f4c78011f40 2023-11-28T17:56:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.387+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800d500 legacy=0x7f4c7800d910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:56:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.387+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:56:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.387+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:56:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.387+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 msgr2=0x7f4c78016d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:56:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.387+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:56:02.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.588+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800d500 legacy=0x7f4c7800d910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:56:02.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.588+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 msgr2=0x7f4c78016d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:56:02.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.588+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:56:02.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.588+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:56:02.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.588+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:56:02.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.989+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800d500 legacy=0x7f4c7800d910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:56:02.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.989+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 msgr2=0x7f4c78016d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:56:02.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.989+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:56:02.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.989+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:56:02.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:02.989+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:56:03.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:03.790+0000 7f4c99180640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800d500 legacy=0x7f4c7800d910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2023-11-28T17:56:03.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:03.790+0000 7f4c9a182640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 msgr2=0x7f4c78016d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3300/0 2023-11-28T17:56:03.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:03.790+0000 7f4c99981640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2023-11-28T17:56:03.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:03.790+0000 7f4c99981640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:56:03.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:03.790+0000 7f4c9a182640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:56:05.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.387+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 msgr2=0x7f4c78016d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:56:05.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.387+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.113:3300/0 conn(0x7f4c78011f40 0x7f4c78016d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:56:05.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.387+0000 7f4c8b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 msgr2=0x7f4c7800e6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:56:05.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.387+0000 7f4c8b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4c7800f960 0x7f4c7800e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:56:05.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.387+0000 7f4c8b7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4c7800d500 legacy=0x7f4c7800d910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:56:05.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.387+0000 7f4c8b7fe640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c78011870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:56:05.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.387+0000 7f4c8b7fe640 1 -- --> v1:172.21.15.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c780102b0 con 0x7f4c7800f150 2023-11-28T17:56:05.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.387+0000 7f4c8b7fe640 1 -- --> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c941567a0 con 0x7f4c78011450 2023-11-28T17:56:05.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.387+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:56:05.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.387+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 msgr2=0x7f4c78011870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:56:05.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.387+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c78011870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:56:05.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.588+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:56:05.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.588+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 msgr2=0x7f4c78011870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:56:05.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.588+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c78011870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:56:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.989+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:56:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.989+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 msgr2=0x7f4c78011870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:56:05.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:05.989+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c78011870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:56:06.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:06.790+0000 7f4c99981640 1 -- >> v1:172.21.15.113:6789/0 conn(0x7f4c7800f150 legacy=0x7f4c78009f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.113:6789/0 2023-11-28T17:56:06.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:06.790+0000 7f4c99180640 1 -- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 msgr2=0x7f4c78011870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.113:3301/0 2023-11-28T17:56:06.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-11-28T17:56:06.791+0000 7f4c99180640 1 --2- >> [v2:172.21.15.113:3301/0,v1:172.21.15.113:6790/0] conn(0x7f4c78011450 0x7f4c78011870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:56:08.271 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:56:08.272 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mon_thrash.py", line 420, in task manager.wait_for_mon_quorum_size(len(mons)) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3156, in wait_for_mon_quorum_size if len(self.get_mon_quorum()) == size: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3141, in get_mon_quorum out = self.raw_cluster_cmd('quorum_status') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-28T17:56:08.289 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-28T17:56:08.289 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T17:56:08.289 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.292 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T17:56:08.293 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T17:56:08.293 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.293 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T17:56:08.293 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-11-28T17:56:08.293 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:56:08.294 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mon_thrash.py", line 420, in task manager.wait_for_mon_quorum_size(len(mons)) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3156, in wait_for_mon_quorum_size if len(self.get_mon_quorum()) == size: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3141, in get_mon_quorum out = self.raw_cluster_cmd('quorum_status') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e' 2023-11-28T17:56:08.309 INFO:tasks.ceph.mds.e:Stopped 2023-11-28T17:56:08.309 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T17:56:08.309 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.310 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T17:56:08.310 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T17:56:08.310 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.310 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T17:56:08.310 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-11-28T17:56:08.310 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.311 INFO:tasks.ceph.mds.f:Stopped 2023-11-28T17:56:08.313 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-28T17:56:08.313 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-28T17:56:08.313 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.314 INFO:tasks.ceph.osd.0:Stopped 2023-11-28T17:56:08.314 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-28T17:56:08.314 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.314 INFO:tasks.ceph.osd.1:Stopped 2023-11-28T17:56:08.314 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-28T17:56:08.314 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.315 INFO:tasks.ceph.osd.2:Stopped 2023-11-28T17:56:08.315 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-11-28T17:56:08.315 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.315 INFO:tasks.ceph.osd.3:Stopped 2023-11-28T17:56:08.315 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-11-28T17:56:08.315 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.316 INFO:tasks.ceph.osd.4:Stopped 2023-11-28T17:56:08.316 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-11-28T17:56:08.316 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.316 INFO:tasks.ceph.osd.5:Stopped 2023-11-28T17:56:08.316 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-11-28T17:56:08.316 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.317 INFO:tasks.ceph.osd.6:Stopped 2023-11-28T17:56:08.317 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-11-28T17:56:08.317 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.317 INFO:tasks.ceph.osd.7:Stopped 2023-11-28T17:56:08.317 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-28T17:56:08.317 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-11-28T17:56:08.317 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.318 INFO:tasks.ceph.mgr.y:Stopped 2023-11-28T17:56:08.318 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-28T17:56:08.318 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:56:08.318 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mon_thrash.py", line 420, in task manager.wait_for_mon_quorum_size(len(mons)) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3156, in wait_for_mon_quorum_size if len(self.get_mon_quorum()) == size: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3141, in get_mon_quorum out = self.raw_cluster_cmd('quorum_status') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi113 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' 2023-11-28T17:56:08.335 INFO:tasks.ceph.mgr.x:Stopped 2023-11-28T17:56:08.335 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-28T17:56:08.336 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-28T17:56:08.336 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.336 INFO:tasks.ceph.mon.a:Stopped 2023-11-28T17:56:08.336 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-28T17:56:08.336 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.338 INFO:tasks.ceph.mon.b:Stopped 2023-11-28T17:56:08.338 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-28T17:56:08.338 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:56:08.339 INFO:tasks.ceph.mon.c:Stopped 2023-11-28T17:56:08.339 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-28T17:56:08.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-11-28T17:56:08.475 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi098.front.sepia.ceph.com 2023-11-28T17:56:08.475 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-28T17:56:08.619 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi098.front.sepia.ceph.com 2023-11-28T17:56:08.619 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-28T17:56:08.777 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi098.front.sepia.ceph.com 2023-11-28T17:56:08.778 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-28T17:56:08.929 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi098.front.sepia.ceph.com 2023-11-28T17:56:08.929 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-11-28T17:56:09.084 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi113.front.sepia.ceph.com 2023-11-28T17:56:09.084 DEBUG:teuthology.orchestra.run.smithi113:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-11-28T17:56:09.291 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi113.front.sepia.ceph.com 2023-11-28T17:56:09.292 DEBUG:teuthology.orchestra.run.smithi113:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-11-28T17:56:09.470 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi113.front.sepia.ceph.com 2023-11-28T17:56:09.470 DEBUG:teuthology.orchestra.run.smithi113:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-11-28T17:56:09.638 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi113.front.sepia.ceph.com 2023-11-28T17:56:09.638 DEBUG:teuthology.orchestra.run.smithi113:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-11-28T17:56:09.829 INFO:tasks.ceph:Archiving mon data... 2023-11-28T17:56:09.831 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398/data/mon.a.tgz 2023-11-28T17:56:09.831 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2023-11-28T17:56:09.891 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.BJ9s114TPB 2023-11-28T17:56:09.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.BJ9s114TPB 2023-11-28T17:56:10.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.BJ9s114TPB 2023-11-28T17:56:10.261 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.BJ9s114TPB is 447KB 2023-11-28T17:56:10.369 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.BJ9s114TPB 2023-11-28T17:56:10.425 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398/data/mon.b.tgz 2023-11-28T17:56:10.425 DEBUG:teuthology.orchestra.run.smithi113:> mktemp 2023-11-28T17:56:10.485 INFO:teuthology.orchestra.run.smithi113.stdout:/tmp/tmp.NnGfyzz79N 2023-11-28T17:56:10.486 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.NnGfyzz79N 2023-11-28T17:56:10.749 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0666 /tmp/tmp.NnGfyzz79N 2023-11-28T17:56:10.895 DEBUG:teuthology.orchestra.remote:smithi113:/tmp/tmp.NnGfyzz79N is 469KB 2023-11-28T17:56:11.004 DEBUG:teuthology.orchestra.run.smithi113:> rm -fr /tmp/tmp.NnGfyzz79N 2023-11-28T17:56:11.062 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398/data/mon.c.tgz 2023-11-28T17:56:11.062 DEBUG:teuthology.orchestra.run.smithi113:> mktemp 2023-11-28T17:56:11.117 INFO:teuthology.orchestra.run.smithi113.stdout:/tmp/tmp.QBijPtPVxU 2023-11-28T17:56:11.117 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.QBijPtPVxU 2023-11-28T17:56:11.307 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0666 /tmp/tmp.QBijPtPVxU 2023-11-28T17:56:11.458 DEBUG:teuthology.orchestra.remote:smithi113:/tmp/tmp.QBijPtPVxU is 471KB 2023-11-28T17:56:11.569 DEBUG:teuthology.orchestra.run.smithi113:> rm -fr /tmp/tmp.QBijPtPVxU 2023-11-28T17:56:11.627 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-28T17:56:11.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-28T17:56:11.636 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-28T17:56:11.732 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-28T17:56:11.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-28T17:56:11.739 DEBUG:teuthology.orchestra.run.smithi113:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-28T17:56:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-28T17:56:11.835 INFO:teuthology.orchestra.run.smithi113.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-28T17:56:11.848 INFO:tasks.ceph:Archiving crash dumps... 2023-11-28T17:56:11.850 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398/remote/smithi098/crash 2023-11-28T17:56:11.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-28T17:56:11.985 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398/remote/smithi113/crash 2023-11-28T17:56:11.986 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-28T17:56:12.086 INFO:tasks.ceph:Shutting down logrotate 2023-11-28T17:56:12.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:56:12.096 DEBUG:teuthology.orchestra.run.smithi113:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:56:12.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-28T17:56:12.342 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-28T17:56:12.446 INFO:tasks.ceph:Compressing logs... 2023-11-28T17:56:12.446 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-28T17:56:12.456 DEBUG:teuthology.orchestra.run.smithi113:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-28T17:56:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94370.log 2023-11-28T17:56:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93984.log 2023-11-28T17:56:12.541 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93975.log 2023-11-28T17:56:12.541 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.94370.log.gz 2023-11-28T17:56:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106011.log 2023-11-28T17:56:12.543 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.93984.log.gz 2023-11-28T17:56:12.544 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102756.log 2023-11-28T17:56:12.544 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.93975.log.gz 2023-11-28T17:56:12.545 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.106011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103320.log 2023-11-28T17:56:12.546 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102756.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-28T17:56:12.546 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.102756.log.gz 2023-11-28T17:56:12.546 INFO:teuthology.orchestra.run.smithi098.stderr: 95.8% -- replaced with /var/log/ceph/ceph-client.admin.106011.log.gz 2023-11-28T17:56:12.547 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104340.log 2023-11-28T17:56:12.547 INFO:teuthology.orchestra.run.smithi098.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.103320.log.gz 2023-11-28T17:56:12.548 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102425.log 2023-11-28T17:56:12.548 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-11-28T17:56:12.549 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104340.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.104340.log.gz 2023-11-28T17:56:12.549 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104424.log 2023-11-28T17:56:12.550 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88497.log 2023-11-28T17:56:12.551 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94557.log 2023-11-28T17:56:12.551 INFO:teuthology.orchestra.run.smithi098.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.102425.log.gz 2023-11-28T17:56:12.552 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88657.log 2023-11-28T17:56:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104424.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.104424.log.gz 2023-11-28T17:56:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94236.log 2023-11-28T17:56:12.553 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph-client.admin.88497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88537.log 2023-11-28T17:56:12.553 INFO:teuthology.orchestra.run.smithi113.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.88497.log.gz 2023-11-28T17:56:12.553 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph-client.admin.88657.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-28T17:56:12.554 INFO:teuthology.orchestra.run.smithi113.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88657.log.gz 2023-11-28T17:56:12.555 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph-client.admin.88537.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-11-28T17:56:12.555 INFO:teuthology.orchestra.run.smithi113.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88537.log.gz 2023-11-28T17:56:12.555 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94557.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.94557.log.gz 2023-11-28T17:56:12.556 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94768.log 2023-11-28T17:56:12.556 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92788.log 2023-11-28T17:56:12.556 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.94236.log.gz 2023-11-28T17:56:12.557 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-11-28T17:56:12.557 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph-mds.b.log: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-28T17:56:12.557 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88377.log 2023-11-28T17:56:12.558 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94768.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.94768.log.gz 2023-11-28T17:56:12.558 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105909.log 2023-11-28T17:56:12.558 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.92788.log: gzip -5 --verbose -- 89.3% /var/log/ceph/ceph-client.admin.101591.log 2023-11-28T17:56:12.558 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92788.log.gz 2023-11-28T17:56:12.559 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96967.log 2023-11-28T17:56:12.559 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-11-28T17:56:12.560 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph-client.admin.88377.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88377.log.gz 2023-11-28T17:56:12.560 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-11-28T17:56:12.561 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101591.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.101591.log.gz 2023-11-28T17:56:12.561 INFO:teuthology.orchestra.run.smithi098.stderr: 95.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.105909.log.gz 2023-11-28T17:56:12.561 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.103506.log 2023-11-28T17:56:12.561 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88917.log 2023-11-28T17:56:12.561 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.96967.log.gz 2023-11-28T17:56:12.562 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103506.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.103506.log.gz 2023-11-28T17:56:12.562 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92358.log 2023-11-28T17:56:12.563 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2023-11-28T17:56:12.563 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-28T17:56:12.564 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.88917.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.88917.log.gz 2023-11-28T17:56:12.564 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97953.log 2023-11-28T17:56:12.564 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.92358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97685.log 2023-11-28T17:56:12.565 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.92358.log.gz 2023-11-28T17:56:12.565 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88457.log 2023-11-28T17:56:12.566 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.97953.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.97953.log.gz 2023-11-28T17:56:12.566 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96557.log 2023-11-28T17:56:12.567 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.97685.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.82516.log 2023-11-28T17:56:12.567 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.97685.log.gz 2023-11-28T17:56:12.568 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.88457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88617.log 2023-11-28T17:56:12.569 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96557.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.96557.log.gz 2023-11-28T17:56:12.569 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96290.log 2023-11-28T17:56:12.569 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.tmp-client.admin.82516.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.82516.log.gz 2023-11-28T17:56:12.570 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97468.log 2023-11-28T17:56:12.570 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96827.log 2023-11-28T17:56:12.571 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96290.log.gz 2023-11-28T17:56:12.572 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.97468.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97468.log.gz 2023-11-28T17:56:12.572 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93269.log 2023-11-28T17:56:12.573 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph-client.admin.88617.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.88457.log.gz 2023-11-28T17:56:12.573 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-28T17:56:12.574 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96827.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.96827.log.gz 2023-11-28T17:56:12.574 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94193.log 2023-11-28T17:56:12.574 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93269.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.93269.log.gz 2023-11-28T17:56:12.575 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102299.log 2023-11-28T17:56:12.575 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- 87.9% /var/log/ceph/ceph-mon.b.log 2023-11-28T17:56:12.576 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94193.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.94193.log.gz 2023-11-28T17:56:12.576 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98507.log 2023-11-28T17:56:12.577 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105353.log 2023-11-28T17:56:12.577 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.102299.log.gz 2023-11-28T17:56:12.577 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.88617.log.gz 2023-11-28T17:56:12.577 INFO:teuthology.orchestra.run.smithi113.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-28T17:56:12.578 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.98507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102089.log 2023-11-28T17:56:12.578 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.98507.log.gz 2023-11-28T17:56:12.579 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105306.log 2023-11-28T17:56:12.579 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.105353.log.gz 2023-11-28T17:56:12.580 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88417.log 2023-11-28T17:56:12.580 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.88417.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88417.log.gz 2023-11-28T17:56:12.581 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94641.log 2023-11-28T17:56:12.581 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.102089.log.gz 2023-11-28T17:56:12.581 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102612.log 2023-11-28T17:56:12.582 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.105306.log.gz 2023-11-28T17:56:12.582 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-11-28T17:56:12.583 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103758.log 2023-11-28T17:56:12.583 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.94641.log.gz 2023-11-28T17:56:12.584 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102612.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.102612.log.gz 2023-11-28T17:56:12.584 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95218.log 2023-11-28T17:56:12.585 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103758.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.103758.log.gz 2023-11-28T17:56:12.585 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.101963.log 2023-11-28T17:56:12.586 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.95218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105600.log 2023-11-28T17:56:12.586 INFO:teuthology.orchestra.run.smithi098.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.95218.log.gz 2023-11-28T17:56:12.587 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103986.log 2023-11-28T17:56:12.587 INFO:teuthology.orchestra.run.smithi098.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.101963.log.gz 2023-11-28T17:56:12.588 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105600.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105600.log.gz 2023-11-28T17:56:12.588 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104676.log 2023-11-28T17:56:12.589 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103278.log 2023-11-28T17:56:12.589 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103986.log.gz 2023-11-28T17:56:12.590 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph-mds.d.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-28T17:56:12.590 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88577.log 2023-11-28T17:56:12.591 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104466.log 2023-11-28T17:56:12.591 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.104676.log.gz 2023-11-28T17:56:12.591 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103278.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.103278.log.gz -5 2023-11-28T17:56:12.592 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph-client.admin.103008.log 2023-11-28T17:56:12.592 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97426.log 2023-11-28T17:56:12.593 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.104466.log.gz 2023-11-28T17:56:12.593 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-11-28T17:56:12.594 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98549.log 2023-11-28T17:56:12.594 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.103008.log.gz 2023-11-28T17:56:12.595 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.97426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102341.log 2023-11-28T17:56:12.595 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97426.log.gz 2023-11-28T17:56:12.596 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.98549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105786.log 2023-11-28T17:56:12.596 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.98549.log.gz 2023-11-28T17:56:12.597 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103800.log 2023-11-28T17:56:12.597 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.102341.log.gz 2023-11-28T17:56:12.598 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97132.log 2023-11-28T17:56:12.598 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105786.log.gz 2023-11-28T17:56:12.599 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105702.log 2023-11-28T17:56:12.599 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/ceph-client.admin.88577.log: /var/log/ceph/ceph-mon.c.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.88577.log.gz 2023-11-28T17:56:12.600 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.103800.log.gz 2023-11-28T17:56:12.600 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.97132.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.97132.log.gz 2023-11-28T17:56:12.601 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104962.log 2023-11-28T17:56:12.601 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105702.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105702.log.gzgzip 2023-11-28T17:56:12.602 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.104238.log 2023-11-28T17:56:12.603 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104962.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104962.log.gz 2023-11-28T17:56:12.603 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94726.log 2023-11-28T17:56:12.604 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104238.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.104238.log.gz 2023-11-28T17:56:12.604 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102570.log 2023-11-28T17:56:12.605 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96248.log 2023-11-28T17:56:12.605 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.94726.log.gz 2023-11-28T17:56:12.606 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93136.log 2023-11-28T17:56:12.607 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.102570.log.gz 2023-11-28T17:56:12.607 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97048.log 2023-11-28T17:56:12.607 INFO:teuthology.orchestra.run.smithi098.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96248.log.gz 2023-11-28T17:56:12.609 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93136.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.93136.log.gz 2023-11-28T17:56:12.609 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96374.log 2023-11-28T17:56:12.610 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.97048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101735.log 2023-11-28T17:56:12.610 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.97048.log.gz 2023-11-28T17:56:12.611 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99220.log 2023-11-28T17:56:12.612 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.96374.log.gz 2023-11-28T17:56:12.613 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93655.log 2023-11-28T17:56:12.613 INFO:teuthology.orchestra.run.smithi098.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.101735.log.gz 2023-11-28T17:56:12.614 INFO:teuthology.orchestra.run.smithi113.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-11-28T17:56:12.614 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.99220.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.99220.log.gz 2023-11-28T17:56:12.614 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102131.log 2023-11-28T17:56:12.615 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99119.log 2023-11-28T17:56:12.615 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.93655.log.gz 2023-11-28T17:56:12.615 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98233.log 2023-11-28T17:56:12.616 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.102131.log.gz 2023-11-28T17:56:12.616 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.99119.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.99119.log.gz -5 2023-11-28T17:56:12.616 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph-client.admin.96514.log 2023-11-28T17:56:12.618 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.98233.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98233.log.gz 2023-11-28T17:56:12.618 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101255.log 2023-11-28T17:56:12.619 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92926.log 2023-11-28T17:56:12.619 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96514.log.gz 2023-11-28T17:56:12.620 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93010.log 2023-11-28T17:56:12.620 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.101255.log.gz 2023-11-28T17:56:12.621 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.92926.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.92926.log.gz 2023-11-28T17:56:12.621 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102924.log 2023-11-28T17:56:12.622 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101423.log 2023-11-28T17:56:12.622 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.93010.log.gz 2023-11-28T17:56:12.623 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93395.log 2023-11-28T17:56:12.623 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.102924.log.gz 2023-11-28T17:56:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101423.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.101423.log.gz 2023-11-28T17:56:12.624 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103422.log 2023-11-28T17:56:12.625 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105264.log 2023-11-28T17:56:12.626 INFO:teuthology.orchestra.run.smithi098.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.93395.log.gz 2023-11-28T17:56:12.626 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103632.log 2023-11-28T17:56:12.627 INFO:teuthology.orchestra.run.smithi098.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.103422.log.gz 2023-11-28T17:56:12.628 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105264.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.105264.log.gz -5 2023-11-28T17:56:12.628 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph-client.admin.95086.log 2023-11-28T17:56:12.629 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103632.log: 89.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.103632.log.gz -5 2023-11-28T17:56:12.629 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph-client.admin.96164.log 2023-11-28T17:56:12.630 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.95086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93647.log 2023-11-28T17:56:12.631 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96164.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.95086.log.gz 2023-11-28T17:56:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105212.log 2023-11-28T17:56:12.632 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96164.log.gz 2023-11-28T17:56:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93647.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.93647.log.gz 2023-11-28T17:56:12.632 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105432.log 2023-11-28T17:56:12.633 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101339.log 2023-11-28T17:56:12.634 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105212.log.gz 2023-11-28T17:56:12.638 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105432.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-28T17:56:12.638 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.105432.log.gz 2023-11-28T17:56:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101339.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.101339.log.gz 2023-11-28T17:56:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101507.log 2023-11-28T17:56:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93479.log 2023-11-28T17:56:12.639 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94599.log 2023-11-28T17:56:12.639 INFO:teuthology.orchestra.run.smithi098.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.101507.log.gz 2023-11-28T17:56:12.639 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-28T17:56:12.640 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99178.log 2023-11-28T17:56:12.640 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.93479.log.gz 2023-11-28T17:56:12.640 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94599.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.94599.log.gz 2023-11-28T17:56:12.640 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102672.log 2023-11-28T17:56:12.641 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.99178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104550.log 2023-11-28T17:56:12.642 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.99178.log.gz 2023-11-28T17:56:12.642 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92475.log 2023-11-28T17:56:12.642 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.102672.log.gz 2023-11-28T17:56:12.643 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104634.log 2023-11-28T17:56:12.644 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104550.log.gz 2023-11-28T17:56:12.644 INFO:teuthology.orchestra.run.smithi113.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2023-11-28T17:56:12.645 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.92475.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.92475.log.gz 2023-11-28T17:56:12.645 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92741.log 2023-11-28T17:56:12.646 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93094.log 2023-11-28T17:56:12.646 INFO:teuthology.orchestra.run.smithi098.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.104634.log.gz 2023-11-28T17:56:12.647 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.92741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93658.log 2023-11-28T17:56:12.647 INFO:teuthology.orchestra.run.smithi098.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.92741.log.gz 2023-11-28T17:56:12.648 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93094.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.93094.log.gz 2023-11-28T17:56:12.648 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104592.log 2023-11-28T17:56:12.649 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104382.log 2023-11-28T17:56:12.649 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.93658.log.gz 2023-11-28T17:56:12.650 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92882.log 2023-11-28T17:56:12.650 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104592.log.gz 2023-11-28T17:56:12.652 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104382.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.104382.log.gz 2023-11-28T17:56:12.652 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96122.log 2023-11-28T17:56:12.652 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.92882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103716.log 2023-11-28T17:56:12.653 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92882.log.gz 2023-11-28T17:56:12.654 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103176.log 2023-11-28T17:56:12.654 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96122.log.gz 2023-11-28T17:56:12.655 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103716.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103716.log.gz 2023-11-28T17:56:12.655 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94936.log 2023-11-28T17:56:12.656 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96870.log 2023-11-28T17:56:12.656 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.103176.log.gz 2023-11-28T17:56:12.657 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93996.log 2023-11-28T17:56:12.657 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.94936.log.gz 2023-11-28T17:56:12.658 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103464.log 2023-11-28T17:56:12.658 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.96870.log.gz 2023-11-28T17:56:12.659 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92573.log 2023-11-28T17:56:12.660 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.93996.log.gz 2023-11-28T17:56:12.660 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92685.log 2023-11-28T17:56:12.661 INFO:teuthology.orchestra.run.smithi098.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.103464.log.gz 2023-11-28T17:56:12.661 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.92573.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92573.log.gz 2023-11-28T17:56:12.662 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105660.log 2023-11-28T17:56:12.663 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.92685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96080.log 2023-11-28T17:56:12.664 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.92685.log.gz 2023-11-28T17:56:12.664 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94978.log 2023-11-28T17:56:12.665 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105660.log.gz 2023-11-28T17:56:12.666 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96080.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96080.log.gz 2023-11-28T17:56:12.666 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104797.log 2023-11-28T17:56:12.668 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97384.log 2023-11-28T17:56:12.668 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.94978.log.gz 2023-11-28T17:56:12.668 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103134.log 2023-11-28T17:56:12.668 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.104797.log.gz 2023-11-28T17:56:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.97384.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.97384.log.gz 2023-11-28T17:56:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96725.log 2023-11-28T17:56:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99731.log 2023-11-28T17:56:12.671 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.103134.log.gz 2023-11-28T17:56:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97300.log 2023-11-28T17:56:12.671 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96725.log.gz 2023-11-28T17:56:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.99731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93353.log 2023-11-28T17:56:12.672 INFO:teuthology.orchestra.run.smithi098.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.99731.log.gz 2023-11-28T17:56:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.97300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104196.log 2023-11-28T17:56:12.673 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.97300.log.gz 2023-11-28T17:56:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103842.log 2023-11-28T17:56:12.675 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.93353.log.gz 2023-11-28T17:56:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104196.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.104196.log.gz 2023-11-28T17:56:12.676 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99773.log 2023-11-28T17:56:12.677 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96332.log 2023-11-28T17:56:12.677 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.103842.log.gz 2023-11-28T17:56:12.678 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.99773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101879.log 2023-11-28T17:56:12.678 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.99773.log.gz 2023-11-28T17:56:12.679 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105744.log 2023-11-28T17:56:12.679 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96332.log.gz 2023-11-28T17:56:12.680 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102527.log 2023-11-28T17:56:12.680 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.101879.log.gz 2023-11-28T17:56:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105496.log 2023-11-28T17:56:12.682 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105744.log.gz 2023-11-28T17:56:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102527.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.102527.log.gz 2023-11-28T17:56:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102215.log 2023-11-28T17:56:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101549.log 2023-11-28T17:56:12.684 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.105496.log.gz 2023-11-28T17:56:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93980.log 2023-11-28T17:56:12.685 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.102215.log.gz 2023-11-28T17:56:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101549.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101549.log.gz 2023-11-28T17:56:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96599.log 2023-11-28T17:56:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92830.log 2023-11-28T17:56:12.687 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.93980.log.gz 2023-11-28T17:56:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96599.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-28T17:56:12.688 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.96599.log.gz 2023-11-28T17:56:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.92830.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92830.log.gz 2023-11-28T17:56:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93995.log 2023-11-28T17:56:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.95806.log 2023-11-28T17:56:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93178.log 2023-11-28T17:56:12.692 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.93995.log.gz 2023-11-28T17:56:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.0.95806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94810.log 2023-11-28T17:56:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93178.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.93178.log.gz 2023-11-28T17:56:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96785.log 2023-11-28T17:56:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94810.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.94810.log.gz 2023-11-28T17:56:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97510.log 2023-11-28T17:56:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96785.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.96785.log.gz 2023-11-28T17:56:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97174.log 2023-11-28T17:56:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.97510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99815.log 2023-11-28T17:56:12.700 INFO:teuthology.orchestra.run.smithi098.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.97510.log.gz 2023-11-28T17:56:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.97174.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.97174.log.gz 2023-11-28T17:56:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102840.log 2023-11-28T17:56:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.99815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103944.log 2023-11-28T17:56:12.704 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4%/var/log/ceph/ceph-client.admin.102840.log: -- replaced with /var/log/ceph/ceph-client.admin.99815.log.gz 2023-11-28T17:56:12.705 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.102840.log.gz 2023-11-28T17:56:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102798.log 2023-11-28T17:56:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103944.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.103944.log.gz 2023-11-28T17:56:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-11-28T17:56:12.708 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101213.log 2023-11-28T17:56:12.708 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.102798.log.gz 2023-11-28T17:56:12.709 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93997.log 2023-11-28T17:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102383.log 2023-11-28T17:56:12.711 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.101213.log.gz 2023-11-28T17:56:12.713 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95020.log 2023-11-28T17:56:12.713 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.93997.log.gz 2023-11-28T17:56:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96683.log 2023-11-28T17:56:12.714 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.102383.log.gz 2023-11-28T17:56:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.95020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88532.log 2023-11-28T17:56:12.716 INFO:teuthology.orchestra.run.smithi098.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.95020.log.gz 2023-11-28T17:56:12.717 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97258.log 2023-11-28T17:56:12.718 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96683.log.gz 2023-11-28T17:56:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.88532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106192.log 2023-11-28T17:56:12.719 INFO:teuthology.orchestra.run.smithi098.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.88532.log.gz 2023-11-28T17:56:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.97258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95996.log 2023-11-28T17:56:12.720 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.97258.log.gz 2023-11-28T17:56:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.106192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93653.log 2023-11-28T17:56:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.95996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102005.log 2023-11-28T17:56:12.723 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95996.log.gz 2023-11-28T17:56:12.724 INFO:teuthology.orchestra.run.smithi098.stderr: 95.7% -- replaced with /var/log/ceph/ceph-client.admin.106192.log.gz 2023-11-28T17:56:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97342.log 2023-11-28T17:56:12.725 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.93653.log.gz 2023-11-28T17:56:12.727 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102005.log: gzip 89.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.102005.log.gz --verbose 2023-11-28T17:56:12.727 INFO:teuthology.orchestra.run.smithi098.stderr: -- /var/log/ceph/ceph-client.admin.93648.log 2023-11-28T17:56:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.97342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102485.log 2023-11-28T17:56:12.728 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.97342.log.gz 2023-11-28T17:56:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93227.log 2023-11-28T17:56:12.729 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.93648.log.gz 2023-11-28T17:56:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101693.log 2023-11-28T17:56:12.731 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.102485.log.gz 2023-11-28T17:56:12.732 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92968.log 2023-11-28T17:56:12.732 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.93227.log.gz 2023-11-28T17:56:12.734 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105558.log 2023-11-28T17:56:12.734 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.101693.log.gz 2023-11-28T17:56:12.735 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.92968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94683.log 2023-11-28T17:56:12.736 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92968.log.gz 2023-11-28T17:56:12.737 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101777.log 2023-11-28T17:56:12.737 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.105558.log.gz 2023-11-28T17:56:12.738 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102257.log 2023-11-28T17:56:12.738 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.94683.log.gz 2023-11-28T17:56:12.739 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101819.log 2023-11-28T17:56:12.740 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.101777.log.gz 2023-11-28T17:56:12.741 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95284.log 2023-11-28T17:56:12.741 INFO:teuthology.orchestra.run.smithi098.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.102257.log.gz 2023-11-28T17:56:12.743 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102882.log 2023-11-28T17:56:12.743 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.101819.log.gz 2023-11-28T17:56:12.744 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.95284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104920.log 2023-11-28T17:56:12.745 INFO:teuthology.orchestra.run.smithi098.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.95284.log.gz 2023-11-28T17:56:12.745 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102882.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2023-11-28T17:56:12.746 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.102882.log.gz 2023-11-28T17:56:12.747 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104920.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-11-28T17:56:12.747 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104920.log.gz 2023-11-28T17:56:12.748 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105148.log 2023-11-28T17:56:12.750 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97090.log 2023-11-28T17:56:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105148.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.105148.log.gz 2023-11-28T17:56:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.97090.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.97090.log.gz 2023-11-28T17:56:12.755 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98308.log 2023-11-28T17:56:12.756 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101381.log 2023-11-28T17:56:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.98308.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.98308.log.gz 2023-11-28T17:56:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101381.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.101381.log.gz 2023-11-28T17:56:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88790.log 2023-11-28T17:56:12.761 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103548.log 2023-11-28T17:56:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.88790.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.88790.log.gz 2023-11-28T17:56:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103548.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.103548.log.gz 2023-11-28T17:56:12.764 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105487.log 2023-11-28T17:56:12.766 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93660.log 2023-11-28T17:56:12.767 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105487.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.105487.log.gz 2023-11-28T17:56:12.769 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93660.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.93660.log.gz 2023-11-28T17:56:12.769 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102173.log 2023-11-28T17:56:12.771 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94852.log 2023-11-28T17:56:12.773 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102173.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.102173.log.gz 2023-11-28T17:56:12.774 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94278.log 2023-11-28T17:56:12.775 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94852.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.94852.log.gz 2023-11-28T17:56:12.775 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103674.log 2023-11-28T17:56:12.776 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94278.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.94278.log.gz 2023-11-28T17:56:12.777 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94320.log 2023-11-28T17:56:12.778 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103674.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.103674.log.gz 2023-11-28T17:56:12.779 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104154.log 2023-11-28T17:56:12.780 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94320.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.94320.log.gz 2023-11-28T17:56:12.781 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93992.log 2023-11-28T17:56:12.782 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104154.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.104154.log.gz 2023-11-28T17:56:12.783 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-28T17:56:12.785 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93992.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.93992.log.gz 2023-11-28T17:56:12.786 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105370.log 2023-11-28T17:56:12.788 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.105370.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105370.log.gz 2023-11-28T17:56:12.789 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104112.log 2023-11-28T17:56:12.792 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101465.log 2023-11-28T17:56:12.793 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.104112.log.gz 2023-11-28T17:56:12.794 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93991.log 2023-11-28T17:56:12.797 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101465.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.101465.log.gz 2023-11-28T17:56:12.800 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96641.log 2023-11-28T17:56:12.803 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96641.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.93991.log.gz 2023-11-28T17:56:12.804 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105022.log 2023-11-28T17:56:12.805 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96641.log.gz 2023-11-28T17:56:12.807 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105106.log 2023-11-28T17:56:12.810 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105106.log: /var/log/ceph/ceph-client.admin.105022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104508.log 2023-11-28T17:56:12.812 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105022.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103380.log 2023-11-28T17:56:12.813 INFO:teuthology.orchestra.run.smithi098.stderr: 2023-11-28T17:56:12.814 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104508.log: 89.0% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105106.log.gz 2023-11-28T17:56:12.816 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.104508.log.gz/var/log/ceph/ceph-client.admin.103380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97634.log 2023-11-28T17:56:12.817 INFO:teuthology.orchestra.run.smithi098.stderr: 2023-11-28T17:56:12.818 INFO:teuthology.orchestra.run.smithi098.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.103380.log.gz 2023-11-28T17:56:12.821 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.97634.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-28T17:56:12.825 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.97634.log.gz 2023-11-28T17:56:12.825 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97216.log 2023-11-28T17:56:12.829 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-28T17:56:12.831 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.97216.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-28T17:56:12.834 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.2.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97216.log.gz 2023-11-28T17:56:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102966.log 2023-11-28T17:56:12.843 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-11-28T17:56:12.843 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102966.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.102966.log.gz 2023-11-28T17:56:12.848 INFO:teuthology.orchestra.run.smithi113.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-11-28T17:56:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98147.log 2023-11-28T17:56:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.98147.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.98147.log.gz 2023-11-28T17:56:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101297.log 2023-11-28T17:56:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93437.log 2023-11-28T17:56:12.863 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101297.log: 89.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95152.log 2023-11-28T17:56:12.864 INFO:teuthology.orchestra.run.smithi098.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz/var/log/ceph/ceph-client.admin.93437.log: 2023-11-28T17:56:12.866 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92631.log 2023-11-28T17:56:12.867 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.95152.log: -- replaced with /var/log/ceph/ceph-client.admin.101297.log.gz 2023-11-28T17:56:12.868 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.93437.log.gz 2023-11-28T17:56:12.868 INFO:teuthology.orchestra.run.smithi098.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.95152.log.gz 2023-11-28T17:56:12.870 INFO:teuthology.orchestra.run.smithi113.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-11-28T17:56:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103236.log 2023-11-28T17:56:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.92631.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.92631.log.gz 2023-11-28T17:56:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96206.log 2023-11-28T17:56:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103236.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104028.log 2023-11-28T17:56:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96206.log: -- replaced with /var/log/ceph/ceph-client.admin.103236.log.gz 2023-11-28T17:56:12.883 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96206.log.gz 2023-11-28T17:56:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104878.log 2023-11-28T17:56:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104028.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104028.log.gz 2023-11-28T17:56:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94515.log 2023-11-28T17:56:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104878.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.104878.log.gz 2023-11-28T17:56:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102714.log 2023-11-28T17:56:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94515.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.94515.log.gz 2023-11-28T17:56:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103884.log 2023-11-28T17:56:12.899 INFO:teuthology.orchestra.run.smithi098.stderr: 90.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103050.log 2023-11-28T17:56:12.899 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.102714.log.gz 2023-11-28T17:56:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.103884.log: /var/log/ceph/ceph-client.admin.103050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94473.log 2023-11-28T17:56:12.905 INFO:teuthology.orchestra.run.smithi098.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.103884.log.gz 2023-11-28T17:56:12.907 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.103050.log.gz 2023-11-28T17:56:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101921.log 2023-11-28T17:56:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94473.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.94473.log.gz 2023-11-28T17:56:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105064.log 2023-11-28T17:56:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101921.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.101921.log.gz 2023-11-28T17:56:12.918 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93052.log 2023-11-28T17:56:12.919 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105064.log.gz 2023-11-28T17:56:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96038.log 2023-11-28T17:56:12.923 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93052.log.gz 2023-11-28T17:56:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105828.log 2023-11-28T17:56:12.928 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96038.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96038.log.gz 2023-11-28T17:56:12.932 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.105828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94894.log 2023-11-28T17:56:12.933 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.105828.log.gz 2023-11-28T17:56:12.940 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.94894.log: 88.8% -5 --verbose -- /var/log/ceph/ceph-client.admin.96432.log 2023-11-28T17:56:12.941 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.94894.log.gz 2023-11-28T17:56:12.942 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-11-28T17:56:12.943 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104718.log 2023-11-28T17:56:12.945 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.96432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103590.log 2023-11-28T17:56:12.947 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101633.log 2023-11-28T17:56:12.947 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.96432.log.gz 2023-11-28T17:56:12.948 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104718.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104718.log.gz 2023-11-28T17:56:12.950 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94430.log 2023-11-28T17:56:12.954 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.101633.log: /var/log/ceph/ceph-client.admin.103590.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99262.log 2023-11-28T17:56:12.955 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.103590.log.gz 2023-11-28T17:56:12.956 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.94430.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.94430.log.gz 2023-11-28T17:56:12.956 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.101633.log.gz 2023-11-28T17:56:12.958 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104280.log 2023-11-28T17:56:12.962 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.99262.log: /var/log/ceph/ceph-client.admin.104280.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.99262.log.gz 2023-11-28T17:56:12.963 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.104280.log.gz 2023-11-28T17:56:12.964 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104070.log 2023-11-28T17:56:12.967 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.104070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103092.log 2023-11-28T17:56:12.969 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.104070.log.gz -5 2023-11-28T17:56:12.970 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph-mds.a.log 2023-11-28T17:56:12.974 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.103092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93311.log 94.0% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2023-11-28T17:56:12.974 INFO:teuthology.orchestra.run.smithi098.stderr: 2023-11-28T17:56:12.976 INFO:teuthology.orchestra.run.smithi098.stderr: 92.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93661.log 2023-11-28T17:56:12.976 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.103092.log.gz 2023-11-28T17:56:12.979 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93661.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.93661.log.gz 2023-11-28T17:56:12.979 INFO:teuthology.orchestra.run.smithi098.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-11-28T17:56:12.980 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95386.log 2023-11-28T17:56:12.981 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93311.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.93311.log.gz 2023-11-28T17:56:12.983 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.95386.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.95386.log.gz 2023-11-28T17:56:12.984 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102047.log 2023-11-28T17:56:12.985 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93650.log 2023-11-28T17:56:12.987 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.102047.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.102047.log.gz 2023-11-28T17:56:12.988 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.93650.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.93650.log.gz 2023-11-28T17:56:12.992 INFO:teuthology.orchestra.run.smithi098.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-28T17:56:13.459 INFO:teuthology.orchestra.run.smithi113.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-11-28T17:56:13.517 INFO:teuthology.orchestra.run.smithi113.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-11-28T17:56:13.595 INFO:teuthology.orchestra.run.smithi098.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-28T17:56:13.715 INFO:teuthology.orchestra.run.smithi098.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-28T17:56:13.781 INFO:teuthology.orchestra.run.smithi113.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-11-28T17:56:13.922 INFO:teuthology.orchestra.run.smithi113.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-11-28T17:56:13.948 INFO:teuthology.orchestra.run.smithi098.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-28T17:56:14.158 INFO:teuthology.orchestra.run.smithi098.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-11-28T17:56:14.290 INFO:teuthology.orchestra.run.smithi098.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.0.95806.log.gz 2023-11-28T17:56:14.294 INFO:teuthology.orchestra.run.smithi098.stderr: 2023-11-28T17:56:14.294 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m1.794s 2023-11-28T17:56:14.294 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m6.362s 2023-11-28T17:56:14.295 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m1.201s 2023-11-28T17:56:15.795 INFO:teuthology.orchestra.run.smithi113.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-11-28T17:56:15.799 INFO:teuthology.orchestra.run.smithi113.stderr: 2023-11-28T17:56:15.800 INFO:teuthology.orchestra.run.smithi113.stderr:real 0m3.292s 2023-11-28T17:56:15.800 INFO:teuthology.orchestra.run.smithi113.stderr:user 0m7.780s 2023-11-28T17:56:15.800 INFO:teuthology.orchestra.run.smithi113.stderr:sys 0m0.690s 2023-11-28T17:56:15.801 INFO:tasks.ceph:Archiving logs... 2023-11-28T17:56:15.801 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398/remote/smithi098/log 2023-11-28T17:56:15.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-28T17:56:16.939 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/log/ceph to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398/remote/smithi113/log 2023-11-28T17:56:16.940 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-28T17:56:18.080 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mon_thrash.py", line 420, in task manager.wait_for_mon_quorum_size(len(mons)) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3156, in wait_for_mon_quorum_size if len(self.get_mon_quorum()) == size: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3141, in get_mon_quorum out = self.raw_cluster_cmd('quorum_status') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-28T17:56:18.097 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-28T17:56:18.107 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mon_thrash.py", line 420, in task manager.wait_for_mon_quorum_size(len(mons)) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3156, in wait_for_mon_quorum_size if len(self.get_mon_quorum()) == size: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3141, in get_mon_quorum out = self.raw_cluster_cmd('quorum_status') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status' 2023-11-28T17:56:18.122 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-28T17:56:18.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-28T17:56:18.132 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-28T17:56:18.284 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-28T17:56:18.293 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-28T17:56:18.293 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T17:56:18.302 DEBUG:teuthology.orchestra.run.smithi113:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T17:56:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2023-11-28T17:56:18.359 INFO:teuthology.orchestra.run.smithi113.stderr:bash: line 1: ntpq: command not found 2023-11-28T17:56:18.360 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T17:56:18.360 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2023-11-28T17:56:18.360 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 51 +523us[ +527us] +/- 82ms 2023-11-28T17:56:18.360 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -296us[ -292us] +/- 54ms 2023-11-28T17:56:18.361 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 -38us[ -34us] +/- 75ms 2023-11-28T17:56:18.361 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:56:18.368 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T17:56:18.368 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2023-11-28T17:56:18.368 INFO:teuthology.orchestra.run.smithi113.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +492us[ +506us] +/- 82ms 2023-11-28T17:56:18.368 INFO:teuthology.orchestra.run.smithi113.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -293us[ -279us] +/- 54ms 2023-11-28T17:56:18.368 INFO:teuthology.orchestra.run.smithi113.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 -36us[ -36us] +/- 75ms 2023-11-28T17:56:18.368 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:56:18.369 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-28T17:56:18.378 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-28T17:56:18.378 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-28T17:56:18.414 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-28T17:56:18.670 DEBUG:teuthology.orchestra.run.smithi113:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-28T17:56:18.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T17:56:19.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:56:19.040 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T17:56:19.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:56:19.150 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-28T17:56:19.161 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-28T17:56:19.205 INFO:teuthology.task.internal:Duration was 1692.256521 seconds 2023-11-28T17:56:19.206 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-28T17:56:19.246 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-28T17:56:19.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-28T17:56:19.256 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-28T17:56:19.419 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T17:56:19.433 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T17:56:19.955 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-28T17:56:19.956 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2023-11-28T17:56:19.956 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-28T17:56:20.034 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi113.front.sepia.ceph.com 2023-11-28T17:56:20.035 DEBUG:teuthology.orchestra.run.smithi113:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-28T17:56:20.112 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-28T17:56:20.113 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-28T17:56:20.121 DEBUG:teuthology.orchestra.run.smithi113:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-28T17:56:20.216 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-28T17:56:20.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T17:56:20.222 DEBUG:teuthology.orchestra.run.smithi113:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T17:56:20.404 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-28T17:56:20.415 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-28T17:56:20.416 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-28T17:56:20.511 DEBUG:teuthology.orchestra.run.smithi113:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-28T17:56:20.614 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-28T17:56:20.624 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-28T17:56:20.633 DEBUG:teuthology.orchestra.run.smithi113:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-28T17:56:20.738 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2023-11-28T17:56:20.745 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = core 2023-11-28T17:56:20.822 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-28T17:56:20.872 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi098.front.sepia.ceph.com, flagging run as failed 2023-11-28T17:56:20.873 DEBUG:teuthology.orchestra.run.smithi113:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-28T17:56:20.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:56:20.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-28T17:56:20.941 INFO:teuthology.task.internal:Transferring archived files... 2023-11-28T17:56:20.942 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398/remote/smithi098 2023-11-28T17:56:20.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-28T17:56:29.368 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2023-11-28T17:56:29.379 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398/remote/smithi098/coredump/1701193487.92476.core.gz: gzip compressed data, was "1701193487.92476.core", last modified: Tue Nov 28 17:56:21 2023, max compression, original size modulo 2^32 245948416 2023-11-28T17:56:29.379 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2023-11-28T17:56:30.409 INFO:teuthology.task.internal: core looks like: /tmp/tmp6bijp69m: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i e', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/bin/ceph-mds', platform: 'x86_64' 2023-11-28T17:56:30.436 INFO:teuthology.task.internal: dump_program: ceph-mds -f --cluster ceph -i e 2023-11-28T17:56:30.437 DEBUG:teuthology.orchestra.run.smithi098:> which 'ceph-mds -f --cluster ceph -i e' 2023-11-28T17:56:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/which: invalid option -- 'f' 2023-11-28T17:56:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/which: unrecognized option '--cluster' 2023-11-28T17:56:30.499 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-mds 2023-11-28T17:56:30.499 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph 2023-11-28T17:56:30.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:56:30.500 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/which: no e in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2023-11-28T17:56:30.501 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mon_thrash.py", line 420, in task manager.wait_for_mon_quorum_size(len(mons)) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3156, in wait_for_mon_quorum_size if len(self.get_mon_quorum()) == size: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3141, in get_mon_quorum out = self.raw_cluster_cmd('quorum_status') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 1: "which 'ceph-mds -f --cluster ceph -i e'" 2023-11-28T17:56:30.517 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-28T17:56:30.529 INFO:teuthology.task.internal:Not uploading archives. 2023-11-28T17:56:30.529 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-28T17:56:30.610 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-28T17:56:30.610 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-28T17:56:30.627 DEBUG:teuthology.orchestra.run.smithi113:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-28T17:56:30.677 INFO:teuthology.orchestra.run.smithi098.stdout: 395372 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 28 17:56 /home/ubuntu/cephtest 2023-11-28T17:56:30.677 INFO:teuthology.orchestra.run.smithi098.stdout: 395388 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Nov 28 17:56 /home/ubuntu/cephtest/archive 2023-11-28T17:56:30.677 INFO:teuthology.orchestra.run.smithi098.stdout: 395423 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 17:56 /home/ubuntu/cephtest/archive/syslog 2023-11-28T17:56:30.677 INFO:teuthology.orchestra.run.smithi098.stdout: 395178 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Nov 28 17:28 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-11-28T17:56:30.677 INFO:teuthology.orchestra.run.smithi098.stdout: 395185 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Nov 28 17:28 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-11-28T17:56:30.677 INFO:teuthology.orchestra.run.smithi098.stdout: 395186 112 -rw-r--r-- 1 ubuntu ubuntu 111930 Nov 28 17:56 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T17:56:30.678 INFO:teuthology.orchestra.run.smithi098.stdout: 525759 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 17:56 /home/ubuntu/cephtest/archive/audit 2023-11-28T17:56:30.678 INFO:teuthology.orchestra.run.smithi098.stdout: 526038 264 -rw------- 1 ubuntu ubuntu 269061 Nov 28 17:56 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-11-28T17:56:30.678 INFO:teuthology.orchestra.run.smithi098.stdout: 395398 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 17:44 /home/ubuntu/cephtest/archive/coredump 2023-11-28T17:56:30.678 INFO:teuthology.orchestra.run.smithi098.stdout: 404366 36360 -rw------- 1 root root 245948416 Nov 28 17:44 /home/ubuntu/cephtest/archive/coredump/1701193487.92476.core 2023-11-28T17:56:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:56:30.680 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-28T17:56:30.680 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/mon_thrash.py", line 420, in task manager.wait_for_mon_quorum_size(len(mons)) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3156, in wait_for_mon_quorum_size if len(self.get_mon_quorum()) == size: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 3141, in get_mon_quorum out = self.raw_cluster_cmd('quorum_status') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-11-28T17:56:30.692 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-28T17:56:30.705 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-28T17:56:30.731 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi098 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status' 2023-11-28T17:56:30.731 INFO:teuthology.nuke:Checking targets against current locks 2023-11-28T17:56:30.740 INFO:teuthology.orchestra.run.smithi113.stdout: 393746 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 28 17:56 /home/ubuntu/cephtest 2023-11-28T17:56:30.740 INFO:teuthology.orchestra.run.smithi113.stdout: 393747 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Nov 28 17:56 /home/ubuntu/cephtest/archive 2023-11-28T17:56:30.740 INFO:teuthology.orchestra.run.smithi113.stdout: 393759 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 17:56 /home/ubuntu/cephtest/archive/syslog 2023-11-28T17:56:30.741 INFO:teuthology.orchestra.run.smithi113.stdout: 393341 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Nov 28 17:28 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-11-28T17:56:30.741 INFO:teuthology.orchestra.run.smithi113.stdout: 393346 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Nov 28 17:28 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-11-28T17:56:30.741 INFO:teuthology.orchestra.run.smithi113.stdout: 393347 104 -rw-r--r-- 1 ubuntu ubuntu 104631 Nov 28 17:56 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T17:56:30.741 INFO:teuthology.orchestra.run.smithi113.stdout: 525747 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 17:56 /home/ubuntu/cephtest/archive/audit 2023-11-28T17:56:30.741 INFO:teuthology.orchestra.run.smithi113.stdout: 526026 240 -rw------- 1 ubuntu ubuntu 241881 Nov 28 17:56 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-11-28T17:56:30.742 INFO:teuthology.orchestra.run.smithi113.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-28T17:56:30.763 DEBUG:teuthology.nuke:shortname: smithi098 2023-11-28T17:56:30.764 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T17:56:30.787 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 17:18:35.879422', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+iCcxfSf82GEbV3qrLN2zk3B72kftTGkorTEp9DLx11Cnbjrg3wMKm4Bv4ZWsqV9EvbYu0IFCdKCD0aBeVm94='} 2023-11-28T17:56:30.792 DEBUG:teuthology.nuke:shortname: smithi113 2023-11-28T17:56:30.792 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T17:56:30.814 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 17:18:35.877664', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdQNw65LjyaXDZutgOR30XOHDfvOBDbuZ+sVTLSEfsOXxgHHCpJpOhUpLvdBhD8YpfEX13hi6Gj6cLSu9FU5IE='} 2023-11-28T17:56:30.835 INFO:teuthology.orchestra.console.smithi098:Power off 2023-11-28T17:56:30.835 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-28T17:56:30.857 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2023-11-28T17:56:30.857 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T17:56:30.876 INFO:teuthology.orchestra.console.smithi113:Power off 2023-11-28T17:56:30.877 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-28T17:56:30.885 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2023-11-28T17:56:30.902 DEBUG:teuthology.orchestra.console.smithi113:power off output: Chassis Power Control: Down/Off 2023-11-28T17:56:30.902 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T17:56:30.937 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2023-11-28T17:56:34.887 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T17:56:34.939 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T17:56:34.997 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2023-11-28T17:56:35.050 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2023-11-28T17:56:39.002 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T17:56:39.052 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T17:56:39.115 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2023-11-28T17:56:39.164 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is off 2023-11-28T17:56:39.265 INFO:teuthology.orchestra.console.smithi113:Power off completed 2023-11-28T17:56:43.118 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T17:56:43.228 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2023-11-28T17:56:43.329 INFO:teuthology.orchestra.console.smithi098:Power off completed 2023-11-28T17:56:43.436 INFO:teuthology.run:Summary data: description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} duration: 1692.2565214633942 failure_reason: 'Command failed (workunit test suites/ffsb.sh) on smithi098 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47c1c547372ed5bff7984647dc200bac0e019caf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh''' flavor: default mon.a-kernel-sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 mon.b-kernel-sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=001eb7dd2d2c4a7db44673ab0344338c status: fail success: false 2023-11-28T17:56:43.436 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-28T17:56:43.515 INFO:teuthology.run:FAIL